Repository navigation
Conversation
| } // namespace mooncake | ||
|
|
||
| void MultiTransport::trackLiveBatch(BatchID batch_id) { | ||
| std::lock_guard<std::mutex> guard(live_batches_mutex_); |
There was a problem hiding this comment.
This may incur significant overhead with concurrent requests.
There was a problem hiding this comment.
Thanks for the flag — agreed that a process-wide mutex on the live-batch set can contend under high concurrent allocate/free (and briefly with waitMemoryIdle scans).
A few notes on the current shape:
- trackLiveBatch / untrackLiveBatch only run on allocateBatchID / tryFreeBatchID; they hold the lock for an unordered_set insert/erase.
- submitTransfer / transport post / completion paths do not take live_batches_mutex_.
- waitMemoryIdle (longer hold while scanning live batches) is only on the unregisterLocalMemory cold path for RDMA: mid-flight unregisterLocalMemory can latch Device not active (Type-2 sticky) #5481 Type-2 mid-flight dereg.
For this PR I’d prefer to keep the simple lock for the correctness fix (block mid-flight ibv_dereg_mr → sticky Device-not-active). Happy to treat a lighter index (e.g. shared_mutex / RCU-style, or piggybacking an always-on batch set) as a follow-up if we measure real allocate/free contention under concurrent load. Does that sound reasonable, or would you rather see a lock-free/shared design in this PR?
There was a problem hiding this comment.
@alogfans pushed a follow-up for the contention point: 7113f56 shards the live-batch index into 64 stripes.
trackLiveBatch/untrackLiveBatch(allocate / free) now take onlybatch_id % 64.- submit / RDMA post / completion still do not touch these locks.
waitMemoryIdle(unregister, cold path) scans one stripe at a time, then confirms a clean result while holding every stripe so a batch inserted into an already-scanned shard is not missed.
That keeps the Type-2 quiesce and should bound allocate/free contention under a large in-flight set. Happy to adjust the shard count if you would rather see a different split.
There was a problem hiding this comment.
@alogfans benchmark of the live-batch index itself (same algorithm as 7113f56: 64 shards, confirm-scan under every shard lock). This is a standalone model of track/untrack, not a full Transfer Engine build, so it is an upper bound on the lock cost. allocateBatchID still does other work (BatchDesc allocation) that this loop does not.
Correctness: 8 threads inserting and erasing, one thread scanning a pinned id the whole time. 12455 scans, 0 misses. The pinned id was still present after the workers stopped.
Single thread, pairs/s (one track + one untrack):
| index | pairs/s | vs no lock |
|---|---|---|
| no lock | 9.7e6 | 1x |
| 64 shards | 6.3e6 | 0.65x |
| one global mutex | 5.5e6 | 0.57x |
Absolute cost is about 0.10 us/pair with no lock and 0.16 us/pair with 64 shards (~60 ns). That is negligible next to an RDMA transfer. A multi-thread run with no lock is not a valid comparison (data race on the set).
Contended track/untrack, pairs/s:
| threads | one mutex | 64 shards |
|---|---|---|
| 8 | 1.9e6 | 5.6e6 (~3.0x) |
| 32 | 1.9e6 | 7.1e6 (~3.6x) |
submitTransfer / RDMA post / completion do not take these locks. The shard split is what removes the process-wide allocate/free convoy the earlier comment called out.
Mid-flight ibv_dereg_mr causes IBV_WC_LOC_PROT_ERR storms that trip the RDMA context breaker and leave Device-not-active sticky until TE restart (Type-2 sticky). Enforce the documented contract by waiting for live/deferred batches that still reference the buffer, returning ERR_BATCH_BUSY on timeout. Fixes kvcache-ai#5481 Signed-off-by: zhangyupeng <zhangyupeng@lixiang.com>
Lab evidence (mooncake-dev) — Type-2 / mid-flight unregMechanism (unpatched TE)
Negative controls
Fix verification (this PR build installed on mooncake-dev)
Note: old A/B schema Docs in kiln
Follow-ups
|
af8489d to
60c50f5
Compare
Baseline vs fixed (same host) — VERIFIEDOn
|
CI builds failed because trackLiveBatch/waitMemoryIdle were appended after the namespace mooncake closing brace, so MultiTransport/BatchID/ Status were undeclared. Signed-off-by: zhangyupeng <zhangyupeng@lixiang.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: zhangyupeng <zhangyupeng@lixiang.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Keep waitMemoryIdle live-batch tracking alongside completion-fd APIs. Signed-off-by: zhangyupeng <zhangyupeng@lixiang.com> Co-authored-by: Cursor <cursoragent@cursor.com>
Concurrent allocate/free now contend on one of 64 stripes instead of a process-wide mutex. waitMemoryIdle still scans every stripe and confirms a clean result under all shard locks. Signed-off-by: zhangyupeng <zhangyupeng@lixiang.com> Co-authored-by: Cursor <cursoragent@cursor.com>
alogfans
left a comment
There was a problem hiding this comment.
Reviewed current head 7113f56. Four inline findings below: three memory-safety/quiescence correctness issues and one issue with the new shard selection. Verification used the added scanner functions extracted unchanged into a minimal fixture: the interior-range check missed a live transfer, ASan detected a dangling request read, and TSan detected concurrent task-publication races. These are isolated reproductions, not a full TE build or RDMA hardware validation.
| Status st = multi_transports_->waitMemoryIdle(addr, /*length=*/0, | ||
| /*timeout_ms=*/5000); |
There was a problem hiding this comment.
[P1] Check the full registered region before deregistering
Passing length=0 makes memoryReferencedByBatch() check only [addr, addr+1). If a 4096-byte registration at base has an unfinished transfer at base+1024 with length 128, neither the request nor its slices overlap that one-byte interval. This returns OK and proceeds to deregister the MR while it is still in use, leaving the original LOC_PROT failure possible. The batch-unregister path passes zero as well. Please retrieve the registered region's full length and use it for the overlap check in both paths. With the added scanner functions extracted unchanged, this case returns OK for length 0 and BatchBusy for length 4096.
| if (task.request) { | ||
| const uintptr_t s = | ||
| reinterpret_cast<uintptr_t>(task.request->source); | ||
| const size_t n = task.request->length; |
There was a problem hiding this comment.
[P1] Do not dereference a borrowed request after submission
submitTransfer() assigns task.request = &entries[i]; the batch does not own those elements. The existing Python async path already calls engine_->submitTransfer(batch_id, {entry}), so that temporary request vector is destroyed immediately after submission. Scanning an unfinished task here then reads freed memory. Since unregister scans every live batch, unregistering an unrelated buffer can trigger this too. An ASan reproduction using the extracted scanner reports heap-use-after-free with the payload and slices still alive. Please capture the referenced address ranges in batch-owned storage during submission instead of reading task.request later.
| auto& batch_desc = *((BatchDesc*)(batch_id)); | ||
| const uintptr_t a0 = reinterpret_cast<uintptr_t>(addr); | ||
| const uintptr_t a1 = length ? (a0 + length) : (a0 + 1); | ||
| for (auto& task : batch_desc.task_list) { |
There was a problem hiding this comment.
[P1] Synchronize reference scanning with task and slice publication
The live-batch shard mutex protects set membership and descriptor deletion, but submitTransfer() does not acquire it when it appends to task_list, assigns task.request, or lets the transport populate slice_list. This scan therefore races those non-atomic reads/writes. It does not require submitting to the buffer being unregistered: unregistering buffer A scans a batch concurrently being populated for buffer B. Holding every shard during the confirmation scan cannot synchronize writers that never take those locks. TSan reports this race in an isolated reproduction of the added scanner. Please publish synchronized, batch-owned reference snapshots, or otherwise serialize scanning with the corresponding mutations.
| }; | ||
| std::array<LiveBatchShard, kLiveBatchShards> live_batch_shards_{}; | ||
| static size_t liveBatchShard(BatchID batch_id) { | ||
| return static_cast<size_t>(batch_id % kLiveBatchShards); |
There was a problem hiding this comment.
[P2] Mix pointer bits before selecting a shard
BatchID is the address returned by new BatchDesc, so its low bits are constrained by alignment. Taking the raw pointer modulo 64 cannot distribute allocations over all 64 stripes: with 16-byte alignment only buckets 0, 16, 32, and 48 are possible, and stronger alignment can concentrate them further. An ordinary-pointer allocation fixture used only 4/64 buckets on this host; that is an alignment demonstration, not a full-engine benchmark. Please remove the guaranteed alignment bits and mix the remaining address bits before choosing the stripe, and validate the distribution/contention benchmark with actual allocated BatchIDs.
Summary
unregisterLocalMemory/ batch unregister now wait for live or deferred MultiTransport batches that still reference the buffer before calling transport deregister.ERR_BATCH_BUSYinstead ofibv_dereg_mrunder outstanding RDMA WRs.local protection error→ context breaker → stickyDevice N is not active(ops Type-2).Fixes #5481
Context
Live repro on mooncake-dev (
mlx5_0): floodtransfer_submit_write×16 thenunregister_memorywhile in flight produced LOC_PROT and sticky Device-not-active; new TransferEngine restored. Peer-store SIGKILL alone does not take this latch (#3559 class is complementary).Test plan