Repository navigation
Fix race in TenantBatchQueue composite-member test (holder not yet dequeued) - #1144
Merged
Merged
Conversation
….Waiting a_composite_member_runs_inside_its_parents_turn_without_queueing read queue.Waiting right after the member ran, but the holder batch posted at the top of the test counts as Waiting until the queue's single reader (a thread-pool task) dequeues it. Under load the reader had not run yet, so Waiting read 1 from the holder, not from the member. The holder now signals when it holds the turn and the test waits for it, so the assertion only sees what the composite member did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hardens the intermittently failing
EventTests.Daemon.TenantBatchQueueTests.a_composite_member_runs_inside_its_parents_turn_without_queueing. It failed onmainCI (net9.0), ongenerator-runtime-version-1138(net9.0 + net10.0), and on #1142 (net10.0), always atqueue.Waiting.ShouldBe(0)reading 1.Cause: a race in the test, not a product bug
TenantBatchQueue.RunAsyncincrements_waitingand posts to the block._waitingis only decremented when the block's single reader dequeues the item. That reader is aTask.Runworker, resumed asynchronously because the channel hasAllowSynchronousContinuations = false.The test posts a "holder" batch to occupy the queue, then runs the composite member right away. With NSubstitute fakes the member's
ProcessRangeAsyncfinishes almost synchronously. Under load (CI runners, the whole suite in parallel), the reader often hasn't been scheduled yet, so the holder is still counted.Waitingreads 1 from the holder, not from the member.I confirmed this with a temporary diagnostic that recorded
Waitingbefore and after the member ran. All 14 failures it caught showedbefore=1 after=1, and the count drained to 0 on its own once the reader ran. The composite path inProjectionExecution.processRangeAsyncnever touches the queue: it only queues whenrange.BatchBehavior == BatchBehavior.Individual.Fix
The holder now signals when it has actually taken the queue's turn. The test waits for that signal, and checks
Waiting == 0, before running the member. The original assertions are unchanged, so they only measure what the composite member did. No timeouts were widened and no retries were added.The test still catches the regression it exists for. As a mutation check, I temporarily made
ProcessRangeAsyncqueue composite members too. The hardened test failed with aTimeoutException, because the member deadlocks behind the holder. That change was reverted.Evidence
I ran the full EventTests suite (1831 tests) in a loop, using the built test executable, with 36
yes > /dev/nullCPU hogs on an 18-core machine. Running this test alone never reproduced the failure (30/30 passes under the same hogs), because it needs the rest of the suite competing for the thread pool.No other test failed in any of those runs.
🤖 Generated with Claude Code