Skip to content

transfer: wait for inflight batches before unregisterLocalMemory - #5482

Open
quantz8a wants to merge 5 commits into
kvcache-ai:mainfrom
quantz8a:fix/te-unregister-wait-inflight
Open

quantz8a wants to merge 5 commits into
kvcache-ai:mainfrom
quantz8a:fix/te-unregister-wait-inflight

Conversation

@quantz8a

Copy link
Copy Markdown
Contributor

Summary

  • unregisterLocalMemory / batch unregister now wait for live or deferred MultiTransport batches that still reference the buffer before calling transport deregister.
  • If the buffer is still busy after 5s, return ERR_BATCH_BUSY instead of ibv_dereg_mr under outstanding RDMA WRs.
  • Prevents mid-flight dereg → local protection error → context breaker → sticky Device N is not active (ops Type-2).

Fixes #5481

Context

Live repro on mooncake-dev (mlx5_0): flood transfer_submit_write ×16 then unregister_memory while in flight produced LOC_PROT and sticky Device-not-active; new TransferEngine restored. Peer-store SIGKILL alone does not take this latch (#3559 class is complementary).

Test plan

  • Build transfer-engine with the change
  • Re-run TE flood mid-flight unreg harness: expect no Device-not-active latch (unregister waits / returns ERR_BATCH_BUSY)
  • Existing TE/store unregister paths still succeed after transfers complete
  • Related: [Store] Drain Client TE paths before tearDown (#3909) #3969 drain-before-tearDown remains complementary on store path

@github-actions github-actions Bot added run-ci Transfer Engine size/M 100-1000 changed lines (additions + deletions) labels Nov 13, 2026
} // namespace mooncake

void MultiTransport::trackLiveBatch(BatchID batch_id) {
std::lock_guard<std::mutex> guard(live_batches_mutex_);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may incur significant overhead with concurrent requests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the flag — agreed that a process-wide mutex on the live-batch set can contend under high concurrent allocate/free (and briefly with waitMemoryIdle scans).

A few notes on the current shape:

For this PR I’d prefer to keep the simple lock for the correctness fix (block mid-flight ibv_dereg_mr → sticky Device-not-active). Happy to treat a lighter index (e.g. shared_mutex / RCU-style, or piggybacking an always-on batch set) as a follow-up if we measure real allocate/free contention under concurrent load. Does that sound reasonable, or would you rather see a lock-free/shared design in this PR?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alogfans pushed a follow-up for the contention point: 7113f56 shards the live-batch index into 64 stripes.

  • trackLiveBatch / untrackLiveBatch (allocate / free) now take only batch_id % 64.
  • submit / RDMA post / completion still do not touch these locks.
  • waitMemoryIdle (unregister, cold path) scans one stripe at a time, then confirms a clean result while holding every stripe so a batch inserted into an already-scanned shard is not missed.

That keeps the Type-2 quiesce and should bound allocate/free contention under a large in-flight set. Happy to adjust the shard count if you would rather see a different split.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alogfans benchmark of the live-batch index itself (same algorithm as 7113f56: 64 shards, confirm-scan under every shard lock). This is a standalone model of track/untrack, not a full Transfer Engine build, so it is an upper bound on the lock cost. allocateBatchID still does other work (BatchDesc allocation) that this loop does not.

Correctness: 8 threads inserting and erasing, one thread scanning a pinned id the whole time. 12455 scans, 0 misses. The pinned id was still present after the workers stopped.

Single thread, pairs/s (one track + one untrack):

index pairs/s vs no lock
no lock 9.7e6 1x
64 shards 6.3e6 0.65x
one global mutex 5.5e6 0.57x

Absolute cost is about 0.10 us/pair with no lock and 0.16 us/pair with 64 shards (~60 ns). That is negligible next to an RDMA transfer. A multi-thread run with no lock is not a valid comparison (data race on the set).

Contended track/untrack, pairs/s:

threads one mutex 64 shards
8 1.9e6 5.6e6 (~3.0x)
32 1.9e6 7.1e6 (~3.6x)

submitTransfer / RDMA post / completion do not take these locks. The shard split is what removes the process-wide allocate/free convoy the earlier comment called out.

Mid-flight ibv_dereg_mr causes IBV_WC_LOC_PROT_ERR storms that trip the
RDMA context breaker and leave Device-not-active sticky until TE restart
(Type-2 sticky). Enforce the documented contract by waiting for
live/deferred batches that still reference the buffer, returning
ERR_BATCH_BUSY on timeout.

Fixes kvcache-ai#5481

Signed-off-by: zhangyupeng <zhangyupeng@lixiang.com>
@quantz8a

Copy link
Copy Markdown
Contributor Author

Lab evidence (mooncake-dev) — Type-2 / mid-flight unreg

Mechanism (unpatched TE)

  • Harness: sim_ops_te_inflight_unreg.py RDMA mlx5_0, flood=16, mode=midflight
  • Result: primary=T2-D1, fail_completions=16, latch_suspected=true, type2.seen=true, restart TE restores
  • Smoking-gun logs:
    • local protection error (IBV_WC_LOC_PROT_ERR)
    • Context is not active: mlx5_0
    • Device 0 is not active

Negative controls

  • wait_then_unreg: no latch
  • Same-host store SIGKILL: Type-1 only (T1-P1+T1-M1), no Type-2 device latch
  • userspace mprotect/munmap and store sync unregister_buffer: cannot produce LOC_* (refuted)

Fix verification (this PR build installed on mooncake-dev)

Leg unreg key log latch
midflight -4 (ERR_BATCH_BUSY) buffer still in flight, refusing deregister none
wait_then_unreg 0 — none

Note: old A/B schema fault_model_confirmed expects A to latch (bug repro). On fixed TE that flag stays false — expected. Fix-verify criteria: A refuses busy + no latch; B clean.

Docs in kiln

  • FIX_DESIGN_PROD_TYPE1_TYPE2_FAULT_MODEL.md (§11 test plan / §12 code / §13 logs)
  • PR5482_EVIDENCE.md

Follow-ups

  • baseline (.bak) vs fixed midflight contrast (paste ready)
  • production serving-log hunt (lab webterm has no live prefill fleet logs)

@quantz8a
quantz8a force-pushed the fix/te-unregister-wait-inflight branch from af8489d to 60c50f5 Compare November 13, 2026 13:32
@quantz8a

Copy link
Copy Markdown
Contributor Author

Baseline vs fixed (same host) — VERIFIED

On instance-8ekvdasn-23 / mlx5_0, swapped engine.so.bak5482 vs this PR engine:

Leg Result
Baseline midflight unreg=0, fail_cq=16, T2-D1 latch — local protection error, Device 0 is not active
Fixed midflight unreg=-4 (ERR_BATCH_BUSY), no latch — refusing deregister / still in flight
Fixed wait_then unreg=0, probe ok, no latch

fix_verified: true — baseline reproduces sticky Device-not-active; this PR blocks mid-flight dereg and keeps drain-then-unreg clean.

zhangyupeng and others added 4 commits November 16, 2026 10:02
CI builds failed because trackLiveBatch/waitMemoryIdle were appended
after the namespace mooncake closing brace, so MultiTransport/BatchID/
Status were undeclared.

Signed-off-by: zhangyupeng <zhangyupeng@lixiang.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: zhangyupeng <zhangyupeng@lixiang.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Keep waitMemoryIdle live-batch tracking alongside completion-fd APIs.

Signed-off-by: zhangyupeng <zhangyupeng@lixiang.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Concurrent allocate/free now contend on one of 64 stripes instead of a
process-wide mutex. waitMemoryIdle still scans every stripe and confirms
a clean result under all shard locks.

Signed-off-by: zhangyupeng <zhangyupeng@lixiang.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

@alogfans alogfans left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed current head 7113f56. Four inline findings below: three memory-safety/quiescence correctness issues and one issue with the new shard selection. Verification used the added scanner functions extracted unchanged into a minimal fixture: the interior-range check missed a live transfer, ASan detected a dangling request read, and TSan detected concurrent task-publication races. These are isolated reproductions, not a full TE build or RDMA hardware validation.

Comment on lines +847 to +848
Status st = multi_transports_->waitMemoryIdle(addr, /*length=*/0,
/*timeout_ms=*/5000);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Check the full registered region before deregistering

Passing length=0 makes memoryReferencedByBatch() check only [addr, addr+1). If a 4096-byte registration at base has an unfinished transfer at base+1024 with length 128, neither the request nor its slices overlap that one-byte interval. This returns OK and proceeds to deregister the MR while it is still in use, leaving the original LOC_PROT failure possible. The batch-unregister path passes zero as well. Please retrieve the registered region's full length and use it for the overlap check in both paths. With the added scanner functions extracted unchanged, this case returns OK for length 0 and BatchBusy for length 4096.

Comment on lines +1498 to +1501
if (task.request) {
const uintptr_t s =
reinterpret_cast<uintptr_t>(task.request->source);
const size_t n = task.request->length;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Do not dereference a borrowed request after submission

submitTransfer() assigns task.request = &entries[i]; the batch does not own those elements. The existing Python async path already calls engine_->submitTransfer(batch_id, {entry}), so that temporary request vector is destroyed immediately after submission. Scanning an unfinished task here then reads freed memory. Since unregister scans every live batch, unregistering an unrelated buffer can trigger this too. An ASan reproduction using the extracted scanner reports heap-use-after-free with the payload and slices still alive. Please capture the referenced address ranges in batch-owned storage during submission instead of reading task.request later.

auto& batch_desc = *((BatchDesc*)(batch_id));
const uintptr_t a0 = reinterpret_cast<uintptr_t>(addr);
const uintptr_t a1 = length ? (a0 + length) : (a0 + 1);
for (auto& task : batch_desc.task_list) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Synchronize reference scanning with task and slice publication

The live-batch shard mutex protects set membership and descriptor deletion, but submitTransfer() does not acquire it when it appends to task_list, assigns task.request, or lets the transport populate slice_list. This scan therefore races those non-atomic reads/writes. It does not require submitting to the buffer being unregistered: unregistering buffer A scans a batch concurrently being populated for buffer B. Holding every shard during the confirmation scan cannot synchronize writers that never take those locks. TSan reports this race in an isolated reproduction of the added scanner. Please publish synchronized, batch-owned reference snapshots, or otherwise serialize scanning with the corresponding mutations.

};
std::array<LiveBatchShard, kLiveBatchShards> live_batch_shards_{};
static size_t liveBatchShard(BatchID batch_id) {
return static_cast<size_t>(batch_id % kLiveBatchShards);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Mix pointer bits before selecting a shard

BatchID is the address returned by new BatchDesc, so its low bits are constrained by alignment. Taking the raw pointer modulo 64 cannot distribute allocations over all 64 stripes: with 16-byte alignment only buckets 0, 16, 32, and 48 are possible, and stronger alignment can concentrate them further. An ordinary-pointer allocation fixture used only 4/64 buckets on this host; that is an alignment demonstration, not a full-engine benchmark. Please remove the guaranteed alignment bits and mix the remaining address bits before choosing the stripe, and validate the distribution/contention benchmark with actual allocated BatchIDs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci size/M 100-1000 changed lines (additions + deletions) Transfer Engine

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RDMA: mid-flight unregisterLocalMemory can latch Device not active (Type-2 sticky)

2 participants