Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions mooncake-store/src/real_client.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -4777,8 +4777,11 @@ tl::expected<void, ErrorCode> RealClient::register_buffer_internal(
// segment. ShmHelper::allocate rounds the mapping up to hugepage
// granularity, so the MR bounds must coincide with the mapping bounds;
// otherwise madvise(MADV_DONTFORK) (active after ibv_fork_init) fails
// with EINVAL when the kernel has to split the hugetlb VMA. Sub-range
// registrations are not special-cased and behave exactly as before.
// with EINVAL when the kernel has to split the hugetlb VMA. An interior
// sub-range is passed through as given: the RDMA transport tries it as
// given first and widens it to its enclosing huge pages only if that is
// rejected with EINVAL (see RdmaTransport::registerLocalMemoryInternal),
// which also covers callers that never go through this function.
// A zero-length request is left untouched so it keeps hitting the
// existing length==0 rejection instead of being widened to the segment.
size_t registration_size = size;
Expand Down
43 changes: 24 additions & 19 deletions mooncake-store/tests/real_client_hugepage_register_test.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,12 @@
// active after ibv_fork_init() — fails with EINVAL when the kernel has to
// split the hugetlb VMA at a non-hugepage-aligned boundary.
//
// Sub-range registrations are deliberately NOT special-cased: they register
// the logical range exactly as passed. On RDMA a misaligned range may fail
// with EINVAL depending on the environment's fork-protection behavior; the
// invariant is that the range is never silently widened.
// Sub-range registrations are not special-cased at this layer: the logical
// range is passed through exactly as given. RdmaTransport tries that range
// first and widens a misaligned HugeTLB sub-range to its enclosing huge pages
// only when ibv_reg_mr rejects it with EINVAL (see
// RdmaTransport::registerLocalMemoryInternal), so such a range also registers
// on a fork-protected stack, where it used to fail with EINVAL.
//
// Protocol/device are taken from the PROTOCOL / DEVICE_NAME environment
// variables (tcp|rdma, same convention as pybind_client_test). With
Expand Down Expand Up @@ -308,16 +310,19 @@ TEST_F(RealClientHugepageRegisterTest, SubHugepageMaxMrSizeIsRejected) {
EXPECT_EQ(ShmHelper::getInstance()->free(base), 0);
}

// Sub-range registrations are not special-cased: an interior pointer is
// registered exactly as passed. On RDMA (ibv_fork_init active) a misaligned
// interior sub-range of a hugetlb VMA fails with EINVAL; on TCP there is no
// ibv_reg_mr and the exact registration succeeds.
// Sub-range registrations are not special-cased at the store layer: an
// interior pointer is passed through exactly as given. RdmaTransport tries the
// range as given first and, if ibv_reg_mr rejects it with EINVAL, retries it
// over the enclosing huge pages (see registerLocalMemoryInternal), which covers
// an interior pointer just as well as a segment base. So on RDMA (ibv_fork_init
// active) a misaligned interior sub-range of a hugetlb VMA now registers
// instead of failing with EINVAL; on TCP there is no ibv_reg_mr and the exact
// registration succeeds either way.
//
// This test is intentionally the LAST in the suite: the deliberately failed
// ibv_reg_mr leaves residue in libibverbs' process-global fork-protection
// bookkeeping (rdma-core does not roll back the DONTFORK state on failure),
// which can poison the first registration of the following test. Keeping it
// last isolates that upstream artifact from the other tests.
// The test accepts both outcomes so it still runs meaningfully against a
// transport stack that does not widen (e.g. TCP only, or an environment that
// sets IBV_FORK_UNNEEDED / RDMAV_HUGEPAGES_SAFE=1 where ibv_reg_mr never fails
// on a misaligned hugetlb range).
TEST_F(RealClientHugepageRegisterTest,
MisalignedInteriorSubRangeRegistersExactly) {
const size_t kSegmentSize = 8 * 1024 * 1024;
Expand All @@ -329,12 +334,12 @@ TEST_F(RealClientHugepageRegisterTest,
void* sub = static_cast<char*>(base) + kOffset;

const int rc = client_->register_buffer(sub, kLength);
// The interior pointer is registered exactly as passed: widening only
// applies to the segment base (buffer == shm->base_addr), so it never
// triggers here. Whether ibv_reg_mr then rejects the misaligned range
// depends on the environment's fork-protection behavior (e.g.
// IBV_FORK_UNNEEDED, or RDMAV_HUGEPAGES_SAFE=1), so accept both outcomes
// and only require that a successful registration unregisters cleanly.
// The store passes the interior pointer through untouched; the RDMA
// transport registers it as given or, when that is rejected on a
// fork-protected stack, widens it to the enclosing huge pages. A failure
// here means neither happened.
EXPECT_EQ(rc, 0) << "an interior sub-range of a HugeTLB segment must "
"register, as given or widened to whole huge pages";
if (rc == 0) {
EXPECT_EQ(client_->unregister_buffer(sub), 0);
}
Expand Down
9 changes: 8 additions & 1 deletion mooncake-transfer-engine/include/error.h
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,17 @@
#define ERR_METADATA (-200)
#define ERR_ENDPOINT (-201)
#define ERR_CONTEXT (-202)
// Internal to the RDMA transport: ERR_CONTEXT whose errno was EINVAL. On
// HugeTLB memory that is what ibv_reg_mr() reports when the range cuts
// through a huge page (ibv_fork_init()'s MADV_DONTFORK cannot split the VMA),
// which RdmaTransport::registerLocalMemoryInternal retries with a widened
// range. Must never cross a public API boundary; translate it back to
// ERR_CONTEXT first.
#define ERR_CONTEXT_EINVAL (-203)

#define ERR_NUMA (-300)
#define ERR_CLOCK (-301)
#define ERR_MEMORY (-302)
#define ERR_NOT_IMPLEMENTED (-303)

#endif // ERROR_H
#endif // ERROR_H
17 changes: 17 additions & 0 deletions mooncake-transfer-engine/include/transfer_engine.h
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,23 @@ class TransferEngine {

int removeLocalSegment(const std::string& segment_name);

// Register [addr, addr + length) for local and (when remote_accessible)
// remote RDMA access.
//
// HugeTLB sub-views: RDMA transports that madvise the registered range
// (RDMA transports call ibv_fork_init, whose MADV_DONTFORK the kernel
// refuses to apply to a hugetlb VMA split inside a huge page) first try
// the range as given. Only if that is rejected with EINVAL is the
// registration retried over the enclosing huge pages. The resulting MR —
// and the rkey/lkey handed to peers — therefore covers whole huge pages,
// including padding outside [addr, addr + length). Access rights are
// per-MR, not per-range: a peer that holds the key can address that
// padding, which may contain application data that was never registered.
// The published description advertises only the requested range, but that
// is a cooperative boundary, not an enforced one. A caller unwilling to
// expose neighbouring bytes must register the whole hugepage-aligned
// mapping itself (whose sub-ranges it then owns exclusively) rather than
// passing a sub-view.
int registerLocalMemory(void* addr, size_t length,
const std::string& location = kWildcardLocation,
bool remote_accessible = true,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
#include <map>
#include <memory>
#include <mutex>
#include <optional>
#include <string>
#include <thread>
#include <string_view>
Expand Down Expand Up @@ -73,6 +74,11 @@ class RdmaTransport : public Transport {
int sendNativeNotify(const std::string &peer_server_name,
const TransferMetadata::NotifyDesc &notify);

// HugeTLB host memory is tried over the caller's own range first; only an
// EINVAL from ibv_reg_mr (a hugetlb VMA cannot be split for the
// MADV_DONTFORK ibv_fork_init installs) retries it over the enclosing huge
// pages. The widened MR covers that padding and so do its keys; see the
// caller contract on TransferEngine::registerLocalMemory.
int registerLocalMemory(void *addr, size_t length,
const std::string &location, bool remote_accessible,
bool update_metadata) override;
Expand All @@ -93,14 +99,27 @@ class RdmaTransport : public Transport {
bool update_metadata,
bool force_sequential);

// defer_publish_to_caller: used by unregisterLocalMemoryBatch, whose
// members must not each publish the local segment desc. A member that has
// finished everything but the publish records that in pending_republish_
// and returns, leaving the batch to publish once and then clearPending().
int unregisterLocalMemoryInternal(void *addr, bool update_metadata,
bool force_sequential);
bool force_sequential,
bool defer_publish_to_caller = false);
#if defined(USE_HYLINK) && defined(ENABLE_MULTI_PROTOCOL)
// Called by the engine when it releases the registration, after all
// transports finish unregistering or a registration is rolled back.
void clearSkipRegistration(void *addr);
#endif

// Pending-cleanup bookkeeping (see the members below). Each helper takes
// pending_cleanup_mutex_ exactly once.
bool hasPendingDereg(uint64_t addr);
std::optional<std::vector<uint64_t>> lookupPendingRepublish(uint64_t addr);
void markPendingDereg(uint64_t addr);
void markPendingRepublish(uint64_t addr, std::vector<uint64_t> keys);
void clearPending(uint64_t addr);

// amdgpu before 6.19.14 charges a NIC's dma-buf pin of KFD-allocated VRAM
// against the KFD free pool a second time; only pins taken through its
// peer-memory client are offset. Holding one such pin per allocation keeps
Expand Down Expand Up @@ -229,14 +248,59 @@ class RdmaTransport : public Transport {
std::mutex chunk_map_mutex_;
std::unordered_map<uint64_t, std::vector<uint64_t>> chunk_map_;

// Single-buffer counterpart of chunk_map_'s pending bookkeeping: base
// addresses whose metadata was removed but whose MR deregistration has
// not completed. Gates the metadata-missing fall-through so only a known
// pending cleanup retries the per-context deregistration — an address
// that was never a registration base must not reach the containing-range
// MR lookup.
std::mutex pending_dereg_mutex_;
// A registration of HugeTLB host memory is attempted over the caller's own
// range first; only when ibv_reg_mr() rejects it with EINVAL — the kernel
// refuses to split a hugetlb VMA when ibv_fork_init() makes the
// registration madvise() it — is it retried over the enclosing huge pages
// (see registerLocalMemoryInternal). Such an MR is registered under a base
// the caller never sees, while the published BufferDesc keeps the caller's
// range. Two things are tracked:
//
// hugetlb_registered_regions_ holds the range of every live hugetlb-backed
// registration, keyed by MR base (the narrow range where that worked, the
// widened one otherwise). Overlap is what has to be refused, not just a
// shared base: RdmaContext's MR map would silently drop an MR registered at
// a duplicate base, and unregisterMemoryRegion() resolves an address to the
// MR *containing* it, so a deregistration that lands in a second, still
// live MR over part of the range would destroy that one. Overlapping the
// interior of a larger registration has exactly that effect, so every
// hugetlb-backed registration is claimed and checked against the live
// ranges rather than their bases. The check, the claim, both attempts and
// the release are serialized by hugetlb_registry_mutex_ so two workers of a
// batch cannot both pass the check before either records its range.
//
// hugepage_widen_aliases_ maps a widened registration's caller-visible
// address to the base its MR actually lives at, so unregisterLocalMemory
// (which only receives the caller's address) can deregister it. A
// registration that succeeded over the caller's own range has no alias.
std::mutex hugetlb_registry_mutex_;
std::map<uint64_t, size_t> hugetlb_registered_regions_;
std::unordered_map<uint64_t, uint64_t> hugepage_widen_aliases_;

// A failed unregister is resumed in one of two states, never both:
//
// pending_dereg_: MR deregistration is NOT confirmed (an ibv_dereg_mr
// failed, or the metadata removal/publish failed while the MRs were still
// live). A retry must re-enter the per-context MR cleanup, whose MR lookup
// is safe here because the address still belongs to our registration.
//
// pending_republish_: the MRs are confirmed gone, the claim and widen
// alias are released and chunk_map_ was erased; only the final
// updateLocalSegmentDesc() (and possibly a leftover local metadata
// removal) is outstanding. The value holds the metadata keys the
// registration was added under (the caller's address for a single buffer,
// the chunk addresses for a chunked one) so a retry can finish the
// removal idempotently. A retry in this state MUST NOT perform any MR
// lookup: a disjoint view of the same range may have registered in the
// meantime, and RdmaContext::unregisterMemoryRegion() resolves by
// containing range, so it would deregister that view's live MR.
//
// The batch path uses pending_republish_ to defer its single collective
// publish; it must never seed pending_dereg_ for a member it already
// cleaned.
std::mutex pending_cleanup_mutex_;
std::unordered_set<uint64_t> pending_dereg_;
std::unordered_map<uint64_t, std::vector<uint64_t>> pending_republish_;

struct KfdPinAnchor {
ibv_mr *mr = nullptr;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -901,11 +901,16 @@ int RdmaContext::registerMemoryRegionInternal(void *addr, size_t length,
mrMeta.mr = ibv_reg_mr(pd_, addr, length, access);
#endif
if (!mrMeta.mr) {
// Capture errno before PLOG touches it: the transport needs to tell
// the HugeTLB sub-view EINVAL apart from every other failure so it can
// retry with a widened range (see
// RdmaTransport::registerLocalMemoryInternal).
const int saved_errno = errno;
PLOG(ERROR) << "Failed to register memory " << addr << " length "
<< length << " dmabuf_offset " << exp.offset << " on "
<< device_name_ << " MC_RDMA_DATA_DIRECT="
<< globalConfig().rdma_data_direct;
return ERR_CONTEXT;
return saved_errno == EINVAL ? ERR_CONTEXT_EINVAL : ERR_CONTEXT;
}
return 0;
}
Expand Down Expand Up @@ -1000,6 +1005,8 @@ int RdmaContext::preTouchMemory(void *addr, size_t length) {
// safe and does not affect the subsequent dereg.
closeDmabufExport(exp);
if (ret != 0) {
// Propagates ERR_CONTEXT_EINVAL unchanged: the fault-in attempt runs
// before any MR is recorded, so the transport can retry it widened.
return ret;
}
if (!mrMeta.mr) return 0; // skipped on a non-paired Data Direct NIC
Expand Down
Loading
Loading