Repository navigation
Conversation
added 28 commits
November 13, 2026 14:48
Update the applied gauge from HAStatusManager::SetOpLogProgress so it uses the same local watermark as /ha/status. Reading Redis's persisted latest must only update the latest gauge, not overwrite local applied progress. Preserve the distinction between assigned and committed IDs. Replace the test that asserted the incorrect coupling with a real Redis regression using an allocated but unpersisted entry. Verify writer and reader latest reads preserve the local applied gauge, and restore HTTP metric and unknown-latest status assertions. Signed-off-by: wl177541 <wl177541@antgroup.com>
Require a running snapshot RPC server as well as healthy metadata before advertising standby snapshot readiness. Publish server availability through an atomic flag so heartbeat readers do not race server teardown. Clear old standby providers before promotion and reset readiness across primary startup, demotion and exit. Report primary HTTP readiness only after its snapshot server starts, preserving standby-only registry semantics. Cover disabled ports, failed startup, promotion, restart and shutdown in unit tests, and validate HTTP and Redis registry readiness during stepdown with snapshot services enabled and disabled. Validation: build six related targets; changed-line clang-format-20 and git diff --check; 188 tests pass across P2P OpLog, RPC, HTTP, Redis store and four Redis HA end-to-end cases. Signed-off-by: wl177541 <wl177541@antgroup.com>
Report snapshot source selection and exclusion counts, including transitional or unknown roles, without changing standby preference or primary fallback. Aggregate failed sources in a rate-limited warning and cap the summary at eight sources with an omitted count. Keep per-source RPC failures and per-entry recovery details in verbose logs, avoid duplicate resync errors, and rate-limit recovery warnings. Preserve terminal required-bootstrap errors and successful snapshot completion logs. Restore source-selection assertions and cover repeated resync failures, empty source lists, bounded summaries and absence of duplicate errors. Validation: five related Docker build targets and changed-line formatting; 110 P2P OpLog tests, 40 Redis store tests and two Redis end-to-end tests pass, including snapshot recovery after trim and promotion of the restored node. Signed-off-by: wl177541 <wl177541@antgroup.com>
Capture applied progress and optional latest progress from the readiness decision instead of mixing its lag with a later status sample. Report an unknown latest sequence as a wait for information, not excessive lag. Classify ELECTION_NOT_READY as a warning and suppress it when the specific P2P readiness reason is present. Clear waiting diagnostics when Redis election exits, preserving actual election errors and all eligibility rules. Cover unknown/known transitions, consistent decision samples, deduplication, real error preservation, HTTP output, readiness recovery and cancellation. Validation: six Docker targets build; changed-line formatting and diff checks; 26 status, 7 readiness, 27 HTTP, 42 Redis helper and 2 Redis E2E cases pass. The restart E2E initially shared Redis with CLIENT KILL tests and failed; it passed unchanged when rerun after the fault-injection suite completed. Signed-off-by: wl177541 <wl177541@antgroup.com>
Record FAILED and the failure reason when standby startup, promotion, metadata restore, snapshot service, auto-trim, or active RPC startup fails. Keep common cleanup from replacing RPC startup failures with STOPPED. Publish RUNNING after RPC listeners start, and distinguish startup cancellation during shutdown or demotion from bind/listen failures. Add real startup-failure regressions and cancellation/readiness checks. Validation: six Docker build targets, changed-line clang-format checks, and 116 related tests including three Redis graceful-shutdown E2Es pass. Signed-off-by: wl177541 <wl177541@antgroup.com>
Treat rejected writes and cancelled synchronous waits during writer shutdown as verbose diagnostics, preserving errors for actual timeouts and unfinished writes. Rate-limit queue overflow warnings per writer to one every five seconds and include the occurrence count. Preserve write results, counters, cancellation and drain ordering. Validation: related Docker targets build; changed-line formatting and diff checks pass; 41 Redis store, 15 RPC and two Redis shutdown/stepdown end-to-end tests pass. Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Signed-off-by: wl177541 <wl177541@antgroup.com>
Verify that entries allocated after watermark initialization remain readable from the in-memory buffer. Check that rejected exhausted sequence IDs leave the manager reusable with a valid store watermark. Validation: Docker oplog_manager_test build and full suite (26 passed, 3 existing etcd placeholders skipped); changed-line clang-format-20 and git diff --check pass. Signed-off-by: wl177541 <wl177541@antgroup.com>
Extend the existing restore test with a second handoff before the client becomes ready. Verify that metadata survives, reads and write candidates remain unavailable before and after a default heartbeat, and ONLINE restores the original route and write eligibility. Validation: Docker p2p_oplog_test build and all 110 tests pass; changed-line clang-format-20 and git diff --check pass. Signed-off-by: wl177541 <wl177541@antgroup.com>
Restore doctor assertions in the two-entry pending buffer test. Verify the rejection issue, the first rejected sequence ID and the configured capacity, while preserving the existing replay progress check and resetting shared HA status. Validation: Docker oplog_applier_test build and all 30 tests pass; changed-line clang-format-20 and git diff --check pass. Signed-off-by: wl177541 <wl177541@antgroup.com>
Create the configured Redis writer through OpLogStoreFactory and verify real pipeline batches exceed the default while respecting the configured limit. Reuse the pipeline gate to queue the workload before workers finish, drain and join before reading the committed watermark, and release hooks on assertion exits. Validation: Docker redis_oplog_store_test build; targeted case passes 10 repeats and all 51 Redis store tests pass on isolated Redis. Changed-line clang-format-20 formatting, dry-run/Werror and git diff --check pass. Signed-off-by: wl177541 <wl177541@antgroup.com>
Extend the existing HA metrics endpoint test to cover the pipeline batch histogram and failure/retry counters. Verify HTTP counter values advance from their current baselines and reject missing samples without defaulting them to zero. Validation: Docker p2p_master_http_endpoints_test build and all 27 tests pass; changed-line clang-format-20 formatting, dry-run/Werror and git diff --check pass. Signed-off-by: wl177541 <wl177541@antgroup.com>
Verify that a drained writer stays healthy past the stall timeout and that ClearDoctorIssues ignores a late progress callback. Keep the existing stopped-writer check, and restart diagnostic tracking before testing the separate clear path. Validation: Docker p2p_master_http_endpoints_test build and all 27 tests pass; changed-line clang-format-20 formatting, dry-run/Werror and git diff --check pass. Signed-off-by: wl177541 <wl177541@antgroup.com>
Restore coverage for repeated WATCHING updates and re-enabling the catchup stall threshold. Confirm that clearing a separate read error leaves the catchup stall diagnostic active until apply progress resumes. Validation: Docker target build and all 27 HTTP endpoint tests pass; changed-line clang-format-20 formatting, dry-run/Werror and git diff --check pass. Signed-off-by: wl177541 <wl177541@antgroup.com>
Use OpLog consistently in test cases, the record fixture, and its test peer declarations. Preserve all test bodies and runtime behavior. Validation: Docker changed-line clang-format-20, build, registration checks, and all 22 renamed tests pass, including the Redis handoff E2E. XML reports confirm each expected test ran without skips; CTest entries have no stale name filters. Signed-off-by: wl177541 <wl177541@antgroup.com>
Check the endpoint alias, timestamp alias, and view age in the existing HA status HTTP test. Preserve the current lease-expiry expectation and production behavior. Validation: Docker changed-line clang-format-20 formatting and dry-run/Werror, target build, all 27 HTTP endpoint tests, XML execution checks, and git diff --check pass. Signed-off-by: wl177541 <wl177541@antgroup.com>
The expanded 16-case Redis HA suite takes about 228 seconds, exceeding the previous 180-second total budget. Increase the CTest timeout to 360 seconds while retaining the existing per-case wait limits. Validation: all 16 E2E cases passed under an extended diagnostic budget, and regenerated CTest metadata reports TIMEOUT=360. Signed-off-by: wl177541 <wl177541@antgroup.com>
Move the Redis E2E listener ports below Linux's default ephemeral port range. An outgoing Redis connection could otherwise claim a standby's RPC port and leave it in TIME_WAIT, preventing the elected successor from starting its listener and causing the readiness assertion to time out. Share named metrics and heartbeat port constants between configuration and probes. Preserve the test assertions and all timeout limits. Validation: changed lines formatted and checked with clang-format-20 in Docker; redis_chaos_test, mooncake_master and clientctl built; all 16 Redis E2E cases passed in 227.38 seconds under the CTest TIMEOUT=360 property, with no skips. git diff --check passed. Signed-off-by: wl177541 <wl177541@antgroup.com>
Move the heartbeat timeout flag to the client executable and pass it through P2PClientConfig, its builders and both Python setup entrypoints. Configure the RPC client instance before connecting and retain the 1000ms default, dedicated heartbeat pools and existing positional argument order. Validate timeout values before replacing an initialized Python client. Cover invalid configuration, independent per-client timeouts and isolation from business RPCs on both heartbeat port modes. Validation: Docker formatting and builds passed; 72 related C++ cases and Python smoke checks passed. The Python instance-preservation check failed before the ordering fix and passed afterward. Redis E2E passed 15/16 cases; the SIGTERM drain case failed its pending-write precondition before SIGTERM and passed three isolated reruns. The initial failure remains unexplained. Signed-off-by: wl177541 <wl177541@antgroup.com>
added 2 commits
November 16, 2026 10:50
Preserve heartbeat timeout configuration across native and Python entrypoints while adopting the upstream expected-based configuration API and integer parser. Adapt validation tests and retain GIL protection when initializing the Python wrapper after configuration succeeds. Validation: native client and Python binding builds; 9 targeted CTest entries; 110 P2P OpLog cases; both Python setup overloads with invalid timeout and local put/get smoke coverage.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Improve P2P HA readiness reporting and diagnostics, and expand regression coverage for Redis OpLog persistence and HA lifecycle transitions.
OpLogtest names.Suggested Merge Priority
Module
mooncake-transfer-engine)mooncake-store)mooncake-conductor)mooncake-reshard)mooncake-ep)mooncake-pg)mooncake-integration)mooncake-p2p-store)mooncake-wheel)mooncake-common)mooncake-rl)Type of Change
How Has This Been Tested?
Validated in a Linux Docker container using a Debug P2P build with TCP/HTTP and Redis enabled; ASan and etcd were disabled. All 78 selected build targets, including the master and clientctl dependencies, compiled successfully.
Test commands:
For the final E2E run, the generated CTest command temporarily received an isolated Redis endpoint and cluster ID; its timeout property was unchanged. The generated command was restored afterward.
Test results:
Unit tests pass
Integration tests pass (if applicable)
Manual testing done (describe below)
All 75 related CTest entries passed: 1,807 individual cases passed and 10 were skipped.
All 53 Redis election helper cases passed without skips.
A complete E2E run passed all 16 cases in 227.54 seconds using an extended diagnostic budget, confirming that the previous 180-second suite timeout was insufficient.
The latest E2E rerun with the committed 360-second timeout completed in 232.04 seconds with 15 passes and one unresolved failure:
RedisSnapshotDisabledTest.ReadinessStaysFalseAcrossStepdown. The successor was not observed in the expectedprimary/snapshot_ready=falsestate within the test's 15-second wait. This was an assertion failure, not the suite timeout; integration validation is therefore not marked complete.The 10 existing skips comprise five tests requiring an active RDMA device, three disabled/placeholder etcd tests, and two contract scenarios unsupported by the v1 configuration.
Checklist
Signed-off-bytrailer certifying the DCO./scripts/code_format.shAll 28 commits have
Signed-off-bytrailers. Changed C/C++ lines were formatted or checked withclang-format-20in Docker, andgit diff --checkpassed. The repository-wide formatting script and pre-commit were not run. No public documentation changes or RFC issue are included.AI Assistance Disclosure
OpenAI Codex assisted with code comparison, implementation, self-review, regression tests, validation, and drafting this PR description. The human submitter is responsible for understanding and defending all changes.