Skip to content

Linux NVENC fails when CUDA device has no matching DRM node (/dev/dxg / Hyper-V GPU-P) #5598

Open
Open
Linux NVENC fails when CUDA device has no matching DRM node (/dev/dxg / Hyper-V GPU-P)#5598
Labels
aiPR has signs of heavy ai usage (either indicated by user or assumed)

Description

@RoYaL69
RoYaL69
opened on Sep 2, 2026

Description

Sunshine cannot use NVENC on Linux when the CUDA/NVENC device is available but has no matching Linux DRM node.

This is reproducible with an NVIDIA GPU exposed to a Linux Hyper-V guest through GPU-P. The GPU is fully usable through /dev/dxg: NVML, CUDA, NVDEC, and direct NVENC all work, but Sunshine fails while trying to map the CUDA device to a DRM device.

The underlying limitation appears generic rather than Hyper-V-specific: Sunshine assumes that a CUDA device used for NVENC also has a corresponding DRM node under /sys/bus/pci/devices/<bus-id>/drm.

Environment

  • Host: Windows 11, Hyper-V
  • Guest: Gen2 Linux VM, Omarchy / Arch Linux
  • Kernel: 7.1.9.arch1-2
  • GPU: NVIDIA GeForce RTX 4050 via GPU-P
  • NVIDIA host driver: 596.08
  • GPU interface in guest: /dev/dxg
  • Display device in guest: hyperv_drm
  • /dev/dri: only card1 from hyperv_drm; no NVIDIA render node
  • Sunshine installed version: 2026.516.143833-1

Also tested unchanged official Sunshine AppImages:

  • 2025.924.154138 (86188d47)
  • 2026.516.143833 (14ffa6fd)
  • 2026.902.143621 (b91470ff)

All show the same behavior.

GPU functionality outside Sunshine

The GPU-P path itself works correctly:

  • /dev/dxg: PASS
  • NVML / nvidia-smi: PASS
  • CUDA: PASS
  • NVDEC: PASS
  • Direct NVENC API: PASS
  • Direct NVENC stress test: 12,000 frames encoded successfully at 1080p, approximately 251-349 fps

So the NVIDIA hardware, driver bridge, and NVENC API are functional in the guest.

Reproduction

Configure Sunshine with NVENC and use either KMS or wlroots capture.

Example relevant setting:

encoder = nvenc

With KMS capture, Sunshine reaches the CUDA/NVENC path but fails with:

Unable to find DRM device with PCI bus ID: 0000:01:00.0

The same failure occurs with wlroots capture.

CUDA reports 0000:01:00.0, but there is no corresponding NVIDIA DRM device in the guest. GPU-P exposes the GPU through /dev/dxg, not through a normal Linux DRM render node.

The only DRM card is the Hyper-V display device:

/dev/dri/card1  -> hyperv_drm

There is no matching:

/sys/bus/pci/devices/0000:01:00.0/drm
/dev/dri/renderD*

Observed behavior

Sunshine calls the DRM lookup for the CUDA device and aborts NVENC initialization when no matching DRM node exists.

This happens even though direct NVENC encoding on the same CUDA device works correctly.

Both KMS and wlroots capture hit the same limitation.

Expected behavior

If CUDA/NVENC is available but the CUDA device has no matching DRM node, Sunshine should ideally be able to fall back to a system-memory capture/upload path instead of rejecting the encoder.

Conceptually:

capture
  -> system memory
  -> CUDA upload
  -> NVENC

This would avoid requiring DRM/DMABUF interop for devices that expose CUDA/NVENC through a non-DRM interface such as /dev/dxg.

Code-path observations

There already appears to be a DRM-free CUDA RAM path through cuda_ram_t / cuda::make_avcodec_encode_device(..., false).

The current failure appears to happen later when the encode device is initialized:

  • KMS can fall back from display_vram_t to display_ram_t, but NVENC initialization subsequently attempts the CUDA-to-DRM lookup.
  • wlroots currently selects a VRAM-oriented CUDA path and does not appear to fall back to a RAM capture path for this case.
  • open_drm_fd_for_cuda_device() requires a matching DRM node and therefore cannot succeed with /dev/dxg.

PR #4946 may be relevant because it introduces a SHM/RAM capture fallback and wl::force_ram_capture. I am not claiming that it fixes this case as-is, but its RAM-capture routing looks conceptually aligned with what is needed here.

Not a regression between recent releases

I tested the same setup against Sunshine versions spanning roughly a year:

Version Capture Result
2025.924.154138 KMS fails at CUDA -> DRM lookup
2026.516.143833 KMS same failure
2026.902.143621 KMS same failure
2026.902.143621 wlroots same failure

The relevant CUDA-to-DRM assumption therefore appears older than these releases rather than being a recent regression.

Why this matters beyond this exact setup

GPU-P is only one way to reach this state. The more general case is:

A valid CUDA/NVENC device exists, but there is no corresponding Linux DRM node.

Sunshine currently cannot use NVENC in that situation even when the encoder itself works.

A generic RAM-capture/CUDA-upload fallback would make the NVENC path more resilient and avoid coupling encoder availability to the presence of a DRM peer when zero-copy DRM interop is unavailable.

I can provide additional logs or test patches if useful.

Activity

  1. Ashutosh0x commented on Sep 10, 2026

    @Ashutosh0x
    Ashutosh0x
    on Sep 10, 2026

    The DRM node isn't required by NVENC or by CUDA here — it's required by the zero-copy EGL path only, and the interesting part is that Sunshine already has a fallback for this and it fires too early to catch it.

    open_drm_fd_for_cuda_device() is called from exactly one place, gl_cuda_vram_t::init() (src/platform/linux/cuda.cpp L423), and the fd is used to build a GBM device and an EGL display for GL→CUDA interop:

    file = std::move(open_drm_fd_for_cuda_device(0));
...
gbm.reset(gbm::create_device(file.el));
display = egl::make_display(gbm.get());

    There are two CUDA encode-device factories, and only one of them needs that:

    • cuda::make_avcodec_gl_encode_device() -> gl_cuda_vram_t — needs the NVIDIA DRM node (kmsgrab L1746, display_vram_t)
    • cuda::make_avcodec_encode_device() -> cuda_ram_t/cuda_vram_t — no DRM node at all (kmsgrab L1568, display_ram_t; also x11grab L660)

    kms_display() already falls back from vram to ram (L1911-1928):

    if (hwdevice_type == vaapi || cuda || vulkan) {
  auto disp = std::make_shared<kms::display_vram_t>(hwdevice_type);
  if (!disp->init(display_name, config)) {
    return disp;
  }
}
auto disp = std::make_shared<kms::display_ram_t>(hwdevice_type);

    But that fallback is keyed on display_vram_t::init(). On GPU-P that init succeeds — the guest does have a DRM card (hyperv_drm card1) for capture. The lookup that fails is for the CUDA device's own node, and it happens later, in make_avcodec_encode_device(pix_fmt), after the fallback decision is already made. So a case the fallback was written for surfaces as a hard failure instead.

    That also explains why the PCI bus ID in the message is the NVIDIA card while capture itself was fine.

    Two directions, depending on what maintainers prefer: let an encode-device failure on the vram display fall back to display_ram_t the same way an init failure does, or probe the CUDA device's DRM node during display_vram_t::init() so the existing fallback sees it.

    Read from source rather than reproduced — I don't have a GPU-P setup. If you can run it, display_ram_t on this same host should already work, so forcing the RAM path is worth trying as a stopgap.

  2. ReenigneArcher
    added
    aiPR has signs of heavy ai usage (either indicated by user or assumed)
    on Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    aiPR has signs of heavy ai usage (either indicated by user or assumed)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions