Skip to content

Let a single active node run clustered without pausing jobs or failing health checks - #2050

Merged
binwiederhier merged 19 commits into
mainfrom
cluster-single-active
Oct 15, 2026
Merged

binwiederhier merged 19 commits into
mainfrom
cluster-single-active

Conversation

@binwiederhier

@binwiederhier binwiederhier commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

Phil's Robot: Makes it safe to turn clustering on in production with the production server as the only active node and any other nodes passive (#2049): nothing changes for clients, and the cluster code can be watched under real load.

Builds on #2040 (the message cache tail) and #2049 (node roles), both merged.

Why

With clustering on, a single active node behaved worse than with clustering off, in two ways:

  1. Scheduled jobs paused after every restart. Only the leader runs delayed messages and Firebase keepalives, and a node becomes leader only after a hold-off of two node TTLs (60s). With clustering off, a node is always leader.
  2. /v1/health returned 503 during a database blip. It reported unhealthy whenever the registry heartbeat was stale, so a load balancer check would pull the only active node and leave its clients nowhere to go.

Changes

Lead right away when no other active node was live recently. The hold-off exists so a previous leader's belief cannot overlap with its successor's. If no other active node heartbeated within the lease plus the hold-off (90s with the default TTL), nobody can still believe it leads, so the node leads at once. Any recently live active node still means the full hold-off, so failover and a returning lower id behave as before.

Report unhealthy only when another active node can take the clients. New Cluster.Serving(): true when the registry heartbeat is fresh, or when the last peer snapshot has no active peer. /v1/health uses it. Peer probes between nodes still use Healthy().

Readiness, separate from health. A passive node is healthy, so /v1/health answers 200 for it, and a load balancer checking only that would send it clients. New /v1/ready: 200 only for an active node that is serving; 503 with {"ready":false,"role":"passive"} otherwise. /v1/health keeps its meaning and now also reports the role, so monitoring can tell a passive (or later draining) node from a broken one. Load balancers should check /v1/ready; documented in docs/config.md.

Tests

Red first, then green:

  • TestRegistry_AloneLeadsRightAway, TestRegistry_OnlyPassivePeersLeadsRightAway, TestRegistry_RecentlyLiveActivePeerMeansHoldOff (the guard); TestRegistry_LowestLiveNodeLeads adjusted: the first node to register alone may lead at once, never two at a time.
  • TestMesh_ServingWithoutAnActivePeer, TestMesh_NotServingWhenAnActivePeerCanTakeOver.
  • TestServer_Cluster_HealthReportsRole, TestServer_Cluster_Ready, TestServer_ReadyWithoutClustering.
  • TestServer_Cluster_ActiveWithPassivePeer: two full servers on one database, active and passive. The active node leads within seconds (fails without the change: the hold-off is a minute), the passive one never, each delivers the other's messages, /v1/health is 200.

go test ./..., race runs of the cluster and server packages, and make lint pass. A lab run with this setup (app1 active, app2 and app3 passive, 350 messages/s, restarts, kill -9 and a Postgres cut-off of the active node) is in progress; results follow.

The hold-off keeps a previous leader's belief from overlapping with its
successor's. A node that sees no other active node live within the lease
plus the hold-off has nobody to overlap with, so it now leads at once. A
single production node with clustering on, or one with only passive
peers, no longer pauses delayed messages and Firebase keepalives for a
minute after every restart or database blip.
/v1/health returned 503 whenever the registry heartbeat was stale, so a
load balancer check would pull the only active node during a database
blip and leave its clients nowhere to go. It now reports unhealthy only
while another active node can take the clients (Cluster.Serving).

Adds an end-to-end test with an active and a passive node on one
database: the active one leads at once, the passive one never, and each
delivers the other's messages.
A passive node is healthy, so /v1/health answers 200 for it, and a load
balancer checking only that would send it clients. /v1/ready answers 200
only for an active node that is serving: what a load balancer should
check. /v1/health keeps its meaning and now also reports the role, so
monitoring can tell a passive (or later draining) node from a broken one.
@binwiederhier
binwiederhier changed the base branch from db-tail to main October 15, 2026 10:18
…gistry prune age from the leadership window

- Cluster.Ready: an active node that is healthy, or has no other active node to take its clients;
  /v1/ready returns it, /v1/health goes back to the raw heartbeat (for monitoring)
- Prune keeps rows as long as a new node holds off for them (lease + hold-off), not a separate 3x TTL
- Document health/ready semantics and the passive-node limitation
@binwiederhier
binwiederhier merged commit 947ea23 into main Oct 15, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant