Repository navigation
Shared subscription wedges after consumer churn: initial ~20-message dispatch then silence, negative availablePermits, only topics-unload recovers #414
Description
Summary
A Shared subscription wedges broker-side after consumer churn: dispatch stops for ALL consumers of the subscription and never recovers on its own. Only a superuser topics unload of the partitions revives it. Three occurrences in one day (magnetar 1.4.1 clients, 12-partition topic, Shared subscription, ack-grouping enabled), each with a distinct trigger but the same outcome.
Occurrences (2026-08-12, otelgw-accesslogs-reader on a 12-partition topic)
- After an operator cursor reset (reset issued while 12 consumers were attached): sub left with ONE ghost consumer (empty consumerName),
availablePermits = -177300(negative!), 225k unacked, rateOut 0. All 12 fresh consumers after an app restart received exactly ~20 messages each, then never anything again. - After a scale-down 12 → 1 instance (partition rebalance): the single consumer received exactly 20 messages in 2h19, flow never re-armed.
msgs_received=20 acks_sent=20 acks_failed=0frozen. - Mid-drain on an instance recycle: fleet consuming at 118k/s with 12 consumers dropped instantly to rateOut=0 with 1 consumer; never recovered until unload.
Signatures
- consumer(s) receive a small initial dispatch (~20 messages) then nothing, forever;
- broker-side
availablePermitscan go massively negative (-177300 observed); - no client-side errors:
acks_failed=0, no reconnect storm, consumers look attached; topics unloadof every partition + client restart restores full throughput immediately.
Suspected area
Broker/driver flow accounting desync across consumer detach/reattach on a Shared sub — same family as #307 (Failover permits stuck after re-subscribe, fixed in 1.2.0) but for Shared subs and surviving 1.4.1's reconnect hardening (#400/#402). The negative permit counter suggests the broker kept dispatch debt from the dead consumers and applied it to the survivors.
Impact
Every consumer churn event (deploy, scale change, crash, operator reset) risks freezing the entire subscription silently — the app looks healthy (connected, acking its 20 messages) while the backlog grows. We hit it 3× in one day of normal operations; each time required manual superuser intervention.
Activity
miton18 commentedon Aug 12, 2026
Operational note: the only known remedy for this wedge (topics unload) composes badly with #413 — under a large backlog the post-unload redelivery burst (observed ~600k msg/s dispatch vs ~130k msg/s processing) floods the unbounded connection event queue and OOM-kills the whole fleet within minutes, whose consumer churn can then re-trigger this wedge. The two bugs form a loop; fixing either breaks it, fixing #413's missing backpressure breaks it best.
feat(consumer): bounded automatic stall recovery (#414)
fix(proto): refund the flow permit of a dead-lettered dispatch unit
fix(proto): close the consumer before re-subscribing it in place
test(e2e): assert the broker's own permit ledger under Shared churn
docs(consumer): correct shared-stall permit evidence and guidance
docs(consumer): correct shared-stall permit evidence and guidance
docs(consumer): correct shared-stall permit evidence and guidance
FlorentinDUBOIS commentedon Nov 2, 2026
Evidence update. Keeping this open — no client-side fix has been identified, and none is warranted without a reproduction.
The confirmed cause was upstream, and it has shipped
apache/pulsar#26416 closed upstream on 2026-09-16. Verified on 2026-11-02 by reading each tag's own commit range with
gh api repos/apache/pulsar/compare/<previous-tag>...<tag>, #26289 and #26422 ship in4.0.14,4.2.5and5.0.0, alongside the later permit-leak fixes #26576, #26606 and #26661. The newest published tags carrying neither are4.0.13and4.2.4, and all three occurrences reported here predate the fixed releases.Shipped is not the same as closed
The suite that measures the aggregate found, across the 2026-10-12 and 2026-10-16 campaigns (PR #881, PR #885):
4.0.14green 5/5 then 3/3,4.2.5green 7/10 then 1/2,5.0.0green 7/10 — the latter two still reaching a residualtotalAvailablePermits = -2on images that carry both patches. So upgrading is worth doing and is a mitigation, not a cure.#896 (merged) records exactly that in
docs/consumer-stall-recovery.md, including what the floor does not settle, so a reader landing on this symptom now learns the broker story rather than only the client ladder.Why no client change
availablePermits = -177300signature reported here has never been reproduced — 23 runs on 2026-09-18 returned CANNOT-REPRODUCE, and nothing since has reproduced it either.topics unload.Nonedefault is deliberate — "this knob acts on the broker, and an action nobody asked for is not a default" — so that is a settled decision, not an oversight.Writing a speculative client "fix" for an unreproduced fault would add risk without evidence. What would move this issue: a fresh occurrence with the per-consumer
availablePermitsand the broker version captured at the time, or a reproduction against a broker at or above the floor.FlorentinDUBOIS commentedon Nov 4, 2026
Closing this one. The broker-side part is fixed upstream, and magnetar ships a client-side recovery path for any remaining wedge.
Broker-side aggregate accounting.
totalAvailablePermitsdrifted negative after consumer churn: apache/pulsar#26416, fixed by apache/pulsar#26289.e2e_shared_subscription_permit_accountingnow asserts the exact settled ledger: the replayed final aggregate must equal the sum of the consumers' settledavailablePermits. It runs on every matrix axis (test(e2e): assert the exact settled permit ledger and un-ignore the accounting test #901, merged as 0be98de).Client-side recovery. Stall detection and bounded in-place recovery (close then resubscribe) are in place: ADR-0101, ADR-0103 and ADR-0108, with the operator procedure in
docs/consumer-stall-recovery.md.What was not reproduced. The original per-consumer signature (
availablePermitsaround −177300 with silent dispatch) never reproduced under the e2e churn harness, on any image. If it shows up again on a broker that carries apache/pulsar#26289, please reopen with the broker version, the topic stats and the dispatcher DEBUG logs.