Skip to content

[Bug]: mooncake memory registration fails with EINVAL #5220

Open

Description

@NUABO

Bug Report

File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py", line 2099, in build
host_pool_group, cache_controller = build_kv_only_stack(
File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py", line 377, in build_kv_only_stack
cache_controller = HybridCacheController(
File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hybrid_cache/hybrid_cache_controller.py", line 225, in init
self.attach_storage_backend(
File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/hybrid_cache/hybrid_cache_controller.py", line 283, in attach_storage_backend
super().attach_storage_backend(
File "/sgl-workspace/sglang/python/sglang/srt/managers/cache_controller.py", line 567, in attach_storage_backend
self.storage_backend.register_mem_pool_host(self.storage_host_pool)
File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/mooncake_store/mooncake_store.py", line 740, in register_mem_pool_host
super().register_buffer(buffer)
File "/sgl-workspace/sglang/python/sglang/srt/mem_cache/storage/mooncake_store/mooncake_store.py", line 338, in register_buffer
raise RuntimeError(
RuntimeError: Failed to register buffer to Mooncake Store, error code: -600
I1103 07:27:11.919000 14591 spdk_initiator.cpp:704] Attached to 10.127.155.11 nqn.2016-06.io.spdk:cnode1
I1103 07:27:11.927270 14591 spdk_initiator.cpp:682] traddr:10.127.155.11trsvcid:4420ns:1subnqn:nqn.2016-06.io.spdk:cnode1trtype:1
[2026-11-03 07:27:11 TP3] Mooncake store warmup successfully.
E1103 07:27:11.972442 14590 rdma_context.cpp:882] Failed to register memory 0x12c0c0c00000 length 296484864 dmabuf_offset 0 on roce0p61s0f0 MC_RDMA_DATA_DIRECT=0: Invalid argument [22]
E1103 07:27:11.972484 14590 rdma_transport.cpp:680] Failed to register memory region (chunk 0) with context 0
[2026-11-03 07:27:11 TP2] Failed to register buffer, error code: -600
[2026-11-03 07:27:11 TP2] attach_hybrid_pool_to_unified_cache failed

Before submitting...

  • Ensure you searched for relevant issues and read the [documentation]

Activity

  1. added
    bugSomething isn't working
    on Nov 4, 2026
  2. NUABO commented on Nov 4, 2026

    @NUABO
    CollaboratorAuthor

    fixed in #5215

  3. github-actions commented on Nov 4, 2026

    @github-actions

    Thanks for opening this issue, @NUABO!

    Field Value
    Issue #5220
    GitHub user ID 29769310
    Reporter @NUABO

    A maintainer will triage this when possible. To help us respond faster, please include:

    • Mooncake version or commit SHA
    • Environment (OS, CUDA/driver, RDMA stack if relevant)
    • Steps to reproduce and expected vs. actual behavior

    Useful links: Documentation · Contributing guide

    This message was posted automatically by the issue bot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions