Skip to content

[Store] Avoid quadratic deduplication of local-first candidates - #5528

Open
importcpp wants to merge 1 commit into
kvcache-ai:mainfrom
importcpp:codex/perf-local-first-dedup
Open

importcpp wants to merge 1 commit into
kvcache-ai:mainfrom
importcpp:codex/perf-local-first-dedup

Conversation

@importcpp

@importcpp importcpp commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Description

MasterService::AllocateReplicas prepares a host-ordered preference list for single-replica local_first placement and prefer_alloc_in_same_node. It generates the complete host-order list before allocation, then appends each name through a growing std::find search of the output vector. With no explicit preference and N distinct names, the merge performs N(N-1)/2 string comparisons: 499,500 at N=1,000. The preparation cost is paid even when the first local candidate can satisfy the write.

Use the existing vector for preference order and a request-local unordered_set<string> for membership, only while merging a nonempty host-order list. Seed the set with the already resolved explicit preferences and append a nonempty name only on its first insertion. This makes the automatic merge expected O(P+N), with P resolved explicit preferences; the explicit-preference parsing/deduplication itself is unchanged, and worst-case hash collisions can still degrade complexity.

Minimal scope and preserved behavior

  • Production change: one function in one file, 13 added / 3 removed lines. A second file adds three regression tests using the existing test peer and a recording strategy that delegates to the real Random strategy.
  • Preserve singular-preference precedence over the preference array, explicit preferences before automatic ones, nonempty first-occurrence ordering, and name-based deduplication across hosts/registrations. Strategy iteration still uses the vector, never the hash table.
  • Preserve the full host order, unknown-host lower-bound/wrap behavior, remote fallback, candidate budgets, and RNG calls.
  • Keep the existing segment-lock acquisition/release, snapshot construction/destruction, lock-external allocation, and registration/session liveness gates. The owning-string set is destroyed inside the original shared-lock scope, before snapshot construction; no views into a reallocating vector or cross-request state are introduced.
  • No new strategy, dependency, configuration, production test API, snapshot mechanism, or framework cleanup. Ordinary requests without automatic host candidates do not construct the set.

Related work: preserves the local-first behavior introduced in #2638. This patch is based directly on main, independently of the owner-snapshot optimization #5524 and bounded fallback coverage #5527. Shared directory snapshots in #5106 and serving-count/client-index work in #4429 address separate costs and are not included here. Searches for existing host-order/quadratic-dedup work found no direct duplicate.

Real Master measurements and tradeoffs

Base: 504f2ae4a307abe095d3baa75d40ed2661b6481b.

CPU-only Linux aarch64/GCC 13.3, -O3, local shared Colima VM (container: 3 CPU quota / 6 GiB). Direct, single-writer MasterService::PutStart with real Offset allocators: one 64 KiB segment per client/host, 4 KiB incoming objects, fixed seed and occupancy. Mount/fill, PutRevoke, verification and output are outside the timer. No RPC, network/payload transfer, NoF or device I/O is included. These are local diagnostics, not a concurrent-throughput or production-topology acceptance test.

The broad run uses six alternating AB/BA pairs, 20 scenarios, 300 warmups and 5,000 timed calls per scenario/process: 1,200,000 timed calls. Each pair matches success counts and endpoint-sequence fingerprints; full-pool failures are expected and remain failures in both versions. Table values are medians of the individual run quantiles, in microseconds:

Scenario Main p50 Patch p50 Main p99 Patch p99
Local fit, 1,000 short names 713.163 119.632 901.529 179.742
Local fit, 2,000 short names 2,764.405 258.416 3,405.915 376.682
Local fit, 1,000 long names 1,357.901 253.325 1,768.926 307.903
Local segment full, 1,000 short names 737.101 120.183 873.158 156.325
All full, 1,000 short names 775.273 171.831 910.163 216.364

Small-pool p99 regression: an explicit cost

The hash table adds hashing, temporary allocations and lock-held construction/destruction. A separate small-pool follow-up uses 20,000 warmups / 100,000 timed calls, four AB/BA pairs over 12 scenarios, plus same-binary A/A comparisons: 9,600,000 timed calls. It does not replace the original six-pair results.

Scenario Main p99, us Patch p99, us Median paired p99 change Paired change range
Local fit, N=4, short names 1.0830 1.1875 +0.1045 us +0.042 to +0.167 us
Local fit, N=16, short names 2.4375 2.8750 +0.4375 us (+18.12%) +0.250 to +0.542 us
Local fit, N=16, long names 4.0000 4.7510 +0.7505 us (+18.76%) +0.584 to +0.875 us
All full, N=16, short names 3.7505 4.2300 +0.4585 us +0.416 to +3.292 us

All four N=16 fit pairs worsen in both wall and CPU quantiles: CPU p99 paired medians increase by 0.417 us for short names and 0.771 us for long names. This is reported as a real cost, not dismissed as noise. N=16 short-name same-base A/A p99 changes are +0.167 / -0.041 us; fixed A/A changes are 0 / -0.125 us. They provide drift context and are not subtracted from A/B. The all-full +3.292 us pair also has CPU p99 +3.5 us. N=64 results have both positive and negative pairs.

An independent ordinary-C++-operator-new census, excluded from latency analysis, finds per-request increments of 2 calls / 64 requested bytes at N=1; 1,001 / 56,248 bytes at N=1,000 short names; 2,001 / 163,248 bytes for long names. Ordinary placement adds zero counted allocations. These are main-thread allocation-request counts and summed bytes, not all malloc calls or peak retained memory.

The submitter explicitly accepts the disclosed microsecond-scale small-pool cost. This patch does not claim p99 non-regression for every pool size or group. It keeps the minimal implementation without an arbitrary small-pool threshold or allocator tuning.

Separately retained: unresolved ordinary-path p99 bad pair

The original multi-scenario run contains an ordinary-placement N=1,000 pair with p99 +35.712 us, even though this branch constructs no new set. Other retained adverse points include N=16 all-full +3.917 us and N=64 local-fit +4.239 us. None is replaced by follow-up medians.

A fresh-process control running only ordinary N=1,000 placement uses 5,000 warmups / 20,000 timed calls, four pairs and same-binary A/A: 160,000 timed calls. Run-quantile p99 medians are 89.0965 us main / 84.5535 us patch; paired changes have median -7.2925 us and range -11.461 to +2.375 us, with one adverse pair. Same-base A/A changes are -14.836 / +15.044 us; fixed A/A changes are -7.167 / +1.876 us.

The original +35.712 us point was not reproduced by that control. Its cause is not fully explained, and A/A variation does not prove it was entirely noise. Runtime-state carryover between scenarios remains an unproven explanation. The original measurements are retained; no snapshot, allocator or other framework change is folded into this patch to make them disappear. Local observations are not tail-latency bounds.

Suggested Merge Priority

  • P0 — Urgent
  • P1 — High
  • P2 — Normal: validated local performance improvement with disclosed tradeoffs
  • P3 — Low

Module

  • Mooncake Store (mooncake-store)

Type of Change

  • Performance improvement

How Has This Been Tested?

415 passed, 4 strategy-inapplicable skips, 0 failed, deduplicated across six suites and the HCA-enabled rerun (419 unique cases). The four skips are FreeRatioFirst-specific parameterized cases under Random. The three added tests also pass against the original base library, asserting preserved behavior rather than timing thresholds.

Regression coverage includes the full preference sequence for both local-first entry points; explicit single/array precedence, duplicates and empty names; cross-host same-name registrations; unknown-host lower-bound/wrap; draining/unmount; absent automatic candidates and ordinary placement. Existing local-full remote fallback, snapshot invalidation and HA barrier tests pass.

Test commands (binary paths in the CPU-only build):

./master_service_test-fixed
./allocation_strategy_test-fixed
./segment_test-fixed
./master_service_placement_scenario_test-fixed
./master_service_ha_test-fixed
./promotion_on_hit_test-fixed
MC_STORE_HCA_AFFINITY=1 ./master_service_test-fixed --gtest_filter='MasterServiceTest.MixedGpuBatchUsesEachKeysHcaHint:MasterServiceTest.PutAndBatchUpsertHonorReceiverHcaLocality'
./master_service_test-base --gtest_filter='MasterServiceTest.LocalFirstMerge*'

Build provenance: reused the separately preserved main store archive and unchanged dependencies, then recompiled the changed Master source and test sources with the existing CMake flags. All 172 archive members were compared byte-for-byte; only master_service.cpp.o differs. This is an incremental source build, not a clean rebuild of every dependency. USE_NOF is off; no device I/O validation is claimed.

Raw CSV/XML/logs and exact build/diagnostic commands are retained locally, outside the two-file patch. Codex self-review and independent review in a separate chat checked the diff and validation evidence, including recomputing all 384 CSVs / 39 XMLs, 10,960,000 latency samples and 4,000 separate allocation-diagnostic samples. This is AI-assisted review, not an upstream human-review claim.

Test results:

  • Unit tests pass (with the four inapplicable skips above)
  • Integration tests pass (not run)
  • Manual testing done: real in-process Master A/B, A/A and allocation census above

Checklist

  • Every contributed commit includes a matching Signed-off-by trailer
  • I have performed a self-review of the code
  • I have formatted changed C++ lines using ./scripts/code_format.sh (clang-format 20.1.8)
  • PR-scoped pre-commit and git diff --check pass
  • Documentation updated (no interface or placement-policy change)
  • Regression tests added
  • RFC for >500 production LOC (not applicable)

AI Assistance Disclosure

  • No AI tools were used
  • AI tools were used: OpenAI Codex assisted with implementation, regression tests, diagnostic harnesses, performance analysis and this description. A separate Codex chat independently reviewed the frozen diff and raw evidence. The human submitter remains responsible for understanding and defending the contribution.

Signed-off-by: importcpp <57519936+importcpp@users.noreply.github.com>
@github-actions github-actions Bot added run-ci Store size/M 100-1000 changed lines (additions + deletions) labels Nov 15, 2026
if (!host_ordered_segments.empty()) {
// Keep the vector's preference order while avoiding a
// growing linear search for every host-ordered name.
std::unordered_set<std::string> seen;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we use a request-local std::unordered_set<std::string_view> here, while keeping preferred_segments as the owning, insertion-ordered vector?

The current implementation is safe, but the temporary set owns another copy of every unique name. For names beyond SSO, this adds character-buffer allocations and frees on top of the hash-node allocations, all inside the segment shared-lock scope.

This should not be a mechanical type replacement: views seeded from preferred_segments can dangle when subsequent push_back calls reallocate that vector, especially for SSO strings. One bounded-scope alternative is:

const size_t max_size =
    preferred_segments.size() + host_ordered_segments.size();
preferred_segments.reserve(max_size);  // Before creating any views.

std::unordered_set<std::string_view> seen;
seen.reserve(max_size);
for (const auto& name : preferred_segments) {
    seen.insert(std::string_view{name});
}
for (const auto& name : host_ordered_segments) {
    if (!name.empty() && seen.insert(std::string_view{name}).second) {
        preferred_segments.push_back(name);
    }
}

With this scope, the existing preference strings cannot move during insertion, host_ordered_segments remains unmodified, and seen is destroyed before either backing vector. Preference order and name-based deduplication stay unchanged.

This could reduce temporary bytes/copies and long-name allocations without changing placement policy. It still allocates hash nodes and hashes all names, so it should not be assumed to eliminate the small-pool regression. Could the existing A/B harness and allocation census compare this variant for both small pools and large short-/long-name pools?

This is an optimization suggestion, not a correctness finding against the current owning-string implementation.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci size/M 100-1000 changed lines (additions + deletions) Store

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants