Skip to content

fix(p2p): improve HA diagnostics and lifecycle regression coverage - #5499

Open
skygragon wants to merge 32 commits into
kvcache-ai:P2P-Mooncake-Storefrom
skygragon:wl177541/p2p-ha-stepdown-fixes
Open

skygragon wants to merge 32 commits into
kvcache-ai:P2P-Mooncake-Storefrom
skygragon:wl177541/p2p-ha-stepdown-fixes

Conversation

@skygragon

Copy link
Copy Markdown

Description

Improve P2P HA readiness reporting and diagnostics, and expand regression coverage for Redis OpLog persistence and HA lifecycle transitions.

  • Report snapshot readiness from running services and keep the applied OpLog metric consistent with local status. Clear stale readiness during promotion and shutdown, and prevent old standby callbacks from overwriting the new role.
  • Preserve startup and promotion failure diagnostics through cleanup. Distinguish election readiness waits from election failures, retain Redis commit error details, and reduce repetitive recovery and shutdown logs.
  • Extend regression coverage for pipeline failures and retries, promotion boundaries, repeated metadata restore, blocked and active writers during shutdown, Redis watch recovery in Pub/Sub and polling modes, and continuous writes across SIGTERM and leadership loss.
  • Restore missing HTTP diagnostics, metrics, and status-field assertions; fix test cleanup and counter synchronization; standardize OpLog test names.
  • Increase the Redis E2E suite timeout from 180 to 360 seconds. The expanded 16-case suite took approximately 228 seconds in a complete run; individual wait limits are unchanged.

Suggested Merge Priority

  • P0 — Urgent: Critical security vulnerability, data loss/corruption, or widespread production outage requiring immediate action
  • P1 — High: Significant correctness or stability issue in existing functionality, or a regression blocking users without a practical workaround
  • P2 — Normal: Routine bug fix, validated performance/compatibility improvement, or new feature for a confirmed use case
  • P3 — Low: Experimental feature, optional enhancement, refactoring, or cleanup with no urgent user impact

Module

  • Transfer Engine (mooncake-transfer-engine)
  • Mooncake Store (mooncake-store)
  • Mooncake Conductor (mooncake-conductor)
  • Reshard (mooncake-reshard)
  • Mooncake EP (mooncake-ep)
  • Mooncake PG (mooncake-pg)
  • Integration (mooncake-integration)
  • P2P Store (mooncake-p2p-store)
  • Python Wheel (mooncake-wheel)
  • Common (mooncake-common)
  • Mooncake RL (mooncake-rl)
  • CI/CD
  • Docs
  • Other

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Breaking change
  • Documentation update
  • Performance improvement
  • Other

How Has This Been Tested?

Validated in a Linux Docker container using a Debug P2P build with TCP/HTTP and Redis enabled; ASan and etcd were disabled. All 78 selected build targets, including the master and clientctl dependencies, compiled successfully.

Test commands:

build_dir=/tmp/mooncake-build-stepdown-release

# Final E2E target build and CMake regeneration.
cmake --build "$build_dir" \
  --target redis_chaos_test mooncake_master clientctl -j 4

# 75 related CTest entries; the two suites below run separately.
ctest --test-dir "$build_dir" -L 'p2p|ha|redis|e2e' \
  -E '^(redis_election_helper_test|redis_chaos_test)$' \
  -j 4 --timeout 600 --no-tests=error --output-on-failure

# TEST_REDIS_ENDPOINT identifies an isolated local Redis instance.
GTEST_OUTPUT=xml:/tmp/redis-helper.xml \
  "$build_dir/mooncake-store/tests/p2p/redis_election_helper_test" \
  --redis_endpoint="$TEST_REDIS_ENDPOINT" \
  --redis_username= --redis_password= --cluster_id=full_helper_fixed_ab553464

# Uses the updated TIMEOUT=360 test property.
ctest --test-dir "$build_dir" -R '^redis_chaos_test$' \
  --no-tests=error --output-on-failure

For the final E2E run, the generated CTest command temporarily received an isolated Redis endpoint and cluster ID; its timeout property was unchanged. The generated command was restored afterward.

Test results:

  • Unit tests pass

  • Integration tests pass (if applicable)

  • Manual testing done (describe below)

  • All 75 related CTest entries passed: 1,807 individual cases passed and 10 were skipped.

  • All 53 Redis election helper cases passed without skips.

  • A complete E2E run passed all 16 cases in 227.54 seconds using an extended diagnostic budget, confirming that the previous 180-second suite timeout was insufficient.

  • The latest E2E rerun with the committed 360-second timeout completed in 232.04 seconds with 15 passes and one unresolved failure: RedisSnapshotDisabledTest.ReadinessStaysFalseAcrossStepdown. The successor was not observed in the expected primary / snapshot_ready=false state within the test's 15-second wait. This was an assertion failure, not the suite timeout; integration validation is therefore not marked complete.

  • The 10 existing skips comprise five tests requiring an active RDMA device, three disabled/placeholder etcd tests, and two contract scenarios unsupported by the v1 configuration.

Checklist

  • Every commit I contribute includes my Signed-off-by trailer certifying the DCO
  • I have performed a self-review of my own code
  • I have formatted my code using ./scripts/code_format.sh
  • I have run pre-commit on the files changed in this PR and all hooks pass
  • I have updated the documentation (if applicable)
  • I have added tests to prove my changes are effective
  • For changes >500 LOC: I have filed an RFC issue

All 28 commits have Signed-off-by trailers. Changed C/C++ lines were formatted or checked with clang-format-20 in Docker, and git diff --check passed. The repository-wide formatting script and pre-commit were not run. No public documentation changes or RFC issue are included.

AI Assistance Disclosure

  • No AI tools were used
  • AI tools were used (specify below)

OpenAI Codex assisted with code comparison, implementation, self-review, regression tests, validation, and drafting this PR description. The human submitter is responsible for understanding and defending all changes.

wl177541 added 28 commits November 13, 2026 14:48
Update the applied gauge from HAStatusManager::SetOpLogProgress so it
uses the same local watermark as /ha/status. Reading Redis's persisted
latest must only update the latest gauge, not overwrite local applied
progress. Preserve the distinction between assigned and committed IDs.

Replace the test that asserted the incorrect coupling with a real Redis
regression using an allocated but unpersisted entry. Verify writer and
reader latest reads preserve the local applied gauge, and restore HTTP
metric and unknown-latest status assertions.

Signed-off-by: wl177541 <wl177541@antgroup.com>
Require a running snapshot RPC server as well as healthy metadata before
advertising standby snapshot readiness. Publish server availability through
an atomic flag so heartbeat readers do not race server teardown.

Clear old standby providers before promotion and reset readiness across
primary startup, demotion and exit. Report primary HTTP readiness only after
its snapshot server starts, preserving standby-only registry semantics.

Cover disabled ports, failed startup, promotion, restart and shutdown in
unit tests, and validate HTTP and Redis registry readiness during stepdown
with snapshot services enabled and disabled.

Validation: build six related targets; changed-line clang-format-20 and
git diff --check; 188 tests pass across P2P OpLog, RPC, HTTP, Redis store
and four Redis HA end-to-end cases.

Signed-off-by: wl177541 <wl177541@antgroup.com>
Report snapshot source selection and exclusion counts, including transitional
or unknown roles, without changing standby preference or primary fallback.
Aggregate failed sources in a rate-limited warning and cap the summary at
eight sources with an omitted count.

Keep per-source RPC failures and per-entry recovery details in verbose logs,
avoid duplicate resync errors, and rate-limit recovery warnings. Preserve
terminal required-bootstrap errors and successful snapshot completion logs.

Restore source-selection assertions and cover repeated resync failures,
empty source lists, bounded summaries and absence of duplicate errors.

Validation: five related Docker build targets and changed-line formatting;
110 P2P OpLog tests, 40 Redis store tests and two Redis end-to-end tests pass,
including snapshot recovery after trim and promotion of the restored node.

Signed-off-by: wl177541 <wl177541@antgroup.com>
Capture applied progress and optional latest progress from the readiness
decision instead of mixing its lag with a later status sample. Report an
unknown latest sequence as a wait for information, not excessive lag.

Classify ELECTION_NOT_READY as a warning and suppress it when the specific
P2P readiness reason is present. Clear waiting diagnostics when Redis
election exits, preserving actual election errors and all eligibility rules.

Cover unknown/known transitions, consistent decision samples, deduplication,
real error preservation, HTTP output, readiness recovery and cancellation.

Validation: six Docker targets build; changed-line formatting and diff checks;
26 status, 7 readiness, 27 HTTP, 42 Redis helper and 2 Redis E2E cases pass.
The restart E2E initially shared Redis with CLIENT KILL tests and failed;
it passed unchanged when rerun after the fault-injection suite completed.

Signed-off-by: wl177541 <wl177541@antgroup.com>
Record FAILED and the failure reason when standby startup, promotion,
metadata restore, snapshot service, auto-trim, or active RPC startup fails.
Keep common cleanup from replacing RPC startup failures with STOPPED.

Publish RUNNING after RPC listeners start, and distinguish startup
cancellation during shutdown or demotion from bind/listen failures.

Add real startup-failure regressions and cancellation/readiness checks.
Validation: six Docker build targets, changed-line clang-format checks,
and 116 related tests including three Redis graceful-shutdown E2Es pass.

Signed-off-by: wl177541 <wl177541@antgroup.com>
Treat rejected writes and cancelled synchronous waits during writer shutdown as verbose diagnostics, preserving errors for actual timeouts and unfinished writes.

Rate-limit queue overflow warnings per writer to one every five seconds and include the occurrence count. Preserve write results, counters, cancellation and drain ordering.

Validation: related Docker targets build; changed-line formatting and diff checks pass; 41 Redis store, 15 RPC and two Redis shutdown/stepdown end-to-end tests pass.
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Verify that entries allocated after watermark initialization remain readable from the in-memory buffer. Check that rejected exhausted sequence IDs leave the manager reusable with a valid store watermark.

Validation: Docker oplog_manager_test build and full suite (26 passed, 3 existing etcd placeholders skipped); changed-line clang-format-20 and git diff --check pass.
Signed-off-by: wl177541 <wl177541@antgroup.com>
Extend the existing restore test with a second handoff before the client becomes ready. Verify that metadata survives, reads and write candidates remain unavailable before and after a default heartbeat, and ONLINE restores the original route and write eligibility.

Validation: Docker p2p_oplog_test build and all 110 tests pass; changed-line clang-format-20 and git diff --check pass.
Signed-off-by: wl177541 <wl177541@antgroup.com>
Restore doctor assertions in the two-entry pending buffer test. Verify the rejection issue, the first rejected sequence ID and the configured capacity, while preserving the existing replay progress check and resetting shared HA status.

Validation: Docker oplog_applier_test build and all 30 tests pass; changed-line clang-format-20 and git diff --check pass.
Signed-off-by: wl177541 <wl177541@antgroup.com>
Create the configured Redis writer through OpLogStoreFactory and verify real pipeline batches exceed the default while respecting the configured limit. Reuse the pipeline gate to queue the workload before workers finish, drain and join before reading the committed watermark, and release hooks on assertion exits.

Validation: Docker redis_oplog_store_test build; targeted case passes 10 repeats and all 51 Redis store tests pass on isolated Redis. Changed-line clang-format-20 formatting, dry-run/Werror and git diff --check pass.
Signed-off-by: wl177541 <wl177541@antgroup.com>
Extend the existing HA metrics endpoint test to cover the pipeline batch histogram and failure/retry counters. Verify HTTP counter values advance from their current baselines and reject missing samples without defaulting them to zero.

Validation: Docker p2p_master_http_endpoints_test build and all 27 tests pass; changed-line clang-format-20 formatting, dry-run/Werror and git diff --check pass.
Signed-off-by: wl177541 <wl177541@antgroup.com>
Verify that a drained writer stays healthy past the stall timeout and that ClearDoctorIssues ignores a late progress callback. Keep the existing stopped-writer check, and restart diagnostic tracking before testing the separate clear path.

Validation: Docker p2p_master_http_endpoints_test build and all 27 tests pass; changed-line clang-format-20 formatting, dry-run/Werror and git diff --check pass.
Signed-off-by: wl177541 <wl177541@antgroup.com>
Restore coverage for repeated WATCHING updates and re-enabling the catchup stall threshold. Confirm that clearing a separate read error leaves the catchup stall diagnostic active until apply progress resumes.

Validation: Docker target build and all 27 HTTP endpoint tests pass; changed-line clang-format-20 formatting, dry-run/Werror and git diff --check pass.
Signed-off-by: wl177541 <wl177541@antgroup.com>
Use OpLog consistently in test cases, the record fixture, and its test peer declarations. Preserve all test bodies and runtime behavior.

Validation: Docker changed-line clang-format-20, build, registration checks, and all 22 renamed tests pass, including the Redis handoff E2E. XML reports confirm each expected test ran without skips; CTest entries have no stale name filters.
Signed-off-by: wl177541 <wl177541@antgroup.com>
Check the endpoint alias, timestamp alias, and view age in the existing HA status HTTP test. Preserve the current lease-expiry expectation and production behavior.

Validation: Docker changed-line clang-format-20 formatting and dry-run/Werror, target build, all 27 HTTP endpoint tests, XML execution checks, and git diff --check pass.
Signed-off-by: wl177541 <wl177541@antgroup.com>
The expanded 16-case Redis HA suite takes about 228 seconds, exceeding the previous 180-second total budget. Increase the CTest timeout to 360 seconds while retaining the existing per-case wait limits.

Validation: all 16 E2E cases passed under an extended diagnostic budget, and regenerated CTest metadata reports TIMEOUT=360.
Signed-off-by: wl177541 <wl177541@antgroup.com>
@skygragon
skygragon requested review from XucSh and YiXR as code owners November 13, 2026 13:12
Move the Redis E2E listener ports below Linux's default ephemeral port
range. An outgoing Redis connection could otherwise claim a standby's
RPC port and leave it in TIME_WAIT, preventing the elected successor from
starting its listener and causing the readiness assertion to time out.

Share named metrics and heartbeat port constants between configuration
and probes. Preserve the test assertions and all timeout limits.

Validation: changed lines formatted and checked with clang-format-20 in
Docker; redis_chaos_test, mooncake_master and clientctl built; all 16 Redis
E2E cases passed in 227.38 seconds under the CTest TIMEOUT=360 property,
with no skips. git diff --check passed.

Signed-off-by: wl177541 <wl177541@antgroup.com>
@github-actions github-actions Bot added size/L 1001+ changed lines (additions + deletions) run-ci Store labels Nov 13, 2026
Move the heartbeat timeout flag to the client executable and pass it through
P2PClientConfig, its builders and both Python setup entrypoints. Configure
the RPC client instance before connecting and retain the 1000ms default,
dedicated heartbeat pools and existing positional argument order.

Validate timeout values before replacing an initialized Python client.
Cover invalid configuration, independent per-client timeouts and isolation
from business RPCs on both heartbeat port modes.

Validation: Docker formatting and builds passed; 72 related C++ cases and
Python smoke checks passed. The Python instance-preservation check failed
before the ordering fix and passed afterward. Redis E2E passed 15/16 cases;
the SIGTERM drain case failed its pending-write precondition before SIGTERM
and passed three isolated reruns. The initial failure remains unexplained.

Signed-off-by: wl177541 <wl177541@antgroup.com>
wl177541 added 2 commits November 16, 2026 10:50
Preserve heartbeat timeout configuration across native and Python entrypoints while adopting the upstream expected-based configuration API and integer parser. Adapt validation tests and retain GIL protection when initializing the Python wrapper after configuration succeeds.

Validation: native client and Python binding builds; 9 targeted CTest entries; 110 P2P OpLog cases; both Python setup overloads with invalid timeout and local put/get smoke coverage.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Integration run-ci size/L 1001+ changed lines (additions + deletions) Store

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants