Repository navigation
Let a single active node run clustered without pausing jobs or failing health checks - #2050
Merged
Merged
Conversation
The hold-off keeps a previous leader's belief from overlapping with its successor's. A node that sees no other active node live within the lease plus the hold-off has nobody to overlap with, so it now leads at once. A single production node with clustering on, or one with only passive peers, no longer pauses delayed messages and Firebase keepalives for a minute after every restart or database blip.
/v1/health returned 503 whenever the registry heartbeat was stale, so a load balancer check would pull the only active node during a database blip and leave its clients nowhere to go. It now reports unhealthy only while another active node can take the clients (Cluster.Serving). Adds an end-to-end test with an active and a passive node on one database: the active one leads at once, the passive one never, and each delivers the other's messages.
A passive node is healthy, so /v1/health answers 200 for it, and a load balancer checking only that would send it clients. /v1/ready answers 200 only for an active node that is serving: what a load balancer should check. /v1/health keeps its meaning and now also reports the role, so monitoring can tell a passive (or later draining) node from a broken one.
…gistry prune age from the leadership window - Cluster.Ready: an active node that is healthy, or has no other active node to take its clients; /v1/ready returns it, /v1/health goes back to the raw heartbeat (for monitoring) - Prune keeps rows as long as a new node holds off for them (lease + hold-off), not a separate 3x TTL - Document health/ready semantics and the passive-node limitation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phil's Robot: Makes it safe to turn clustering on in production with the production server as the only active node and any other nodes passive (#2049): nothing changes for clients, and the cluster code can be watched under real load.
Builds on #2040 (the message cache tail) and #2049 (node roles), both merged.
Why
With clustering on, a single active node behaved worse than with clustering off, in two ways:
/v1/healthreturned 503 during a database blip. It reported unhealthy whenever the registry heartbeat was stale, so a load balancer check would pull the only active node and leave its clients nowhere to go.Changes
Lead right away when no other active node was live recently. The hold-off exists so a previous leader's belief cannot overlap with its successor's. If no other active node heartbeated within the lease plus the hold-off (90s with the default TTL), nobody can still believe it leads, so the node leads at once. Any recently live active node still means the full hold-off, so failover and a returning lower id behave as before.
Report unhealthy only when another active node can take the clients. New
Cluster.Serving(): true when the registry heartbeat is fresh, or when the last peer snapshot has no active peer./v1/healthuses it. Peer probes between nodes still useHealthy().Readiness, separate from health. A passive node is healthy, so
/v1/healthanswers 200 for it, and a load balancer checking only that would send it clients. New/v1/ready: 200 only for an active node that is serving; 503 with{"ready":false,"role":"passive"}otherwise./v1/healthkeeps its meaning and now also reports the role, so monitoring can tell a passive (or later draining) node from a broken one. Load balancers should check/v1/ready; documented indocs/config.md.Tests
Red first, then green:
TestRegistry_AloneLeadsRightAway,TestRegistry_OnlyPassivePeersLeadsRightAway,TestRegistry_RecentlyLiveActivePeerMeansHoldOff(the guard);TestRegistry_LowestLiveNodeLeadsadjusted: the first node to register alone may lead at once, never two at a time.TestMesh_ServingWithoutAnActivePeer,TestMesh_NotServingWhenAnActivePeerCanTakeOver.TestServer_Cluster_HealthReportsRole,TestServer_Cluster_Ready,TestServer_ReadyWithoutClustering.TestServer_Cluster_ActiveWithPassivePeer: two full servers on one database, active and passive. The active node leads within seconds (fails without the change: the hold-off is a minute), the passive one never, each delivers the other's messages,/v1/healthis 200.go test ./..., race runs of the cluster and server packages, andmake lintpass. A lab run with this setup (app1 active, app2 and app3 passive, 350 messages/s, restarts,kill -9and a Postgres cut-off of the active node) is in progress; results follow.