[atassis]

Description

Two improvements for Wayland capture on headless NVIDIA setups (no physical display connected):

1. SHM capture fallback

On headless NVIDIA, GBM buffer allocation can fail because the driver cannot create GBM buffers without an active DRM output. The existing code logged "SHM capture not implemented" and gave up.

This PR implements the SHM fallback:

• Binds wl_shm interface from the Wayland registry
• When DMA-BUF fails, allocates a memfd-backed wl_shm_pool and copies frames via SHM
• Handles 4bpp (XRGB8888) and 3bpp (BGR888) pixel formats with correct channel conversion
• Caches GBM failure with sticky gbm_failed flag to avoid retrying every frame
• Makes EGL initialization non-fatal so wlr_ram_t can operate in SHM-only mode

2. Resilient GBM allocation for headless NVIDIA

Even on headless NVIDIA, GBM can work if we relax the usage flags. This change:

• Tries progressively relaxed GBM flags: RENDERING|LINEAR → RENDERING → LINEAR → none
• Prefers DRM render nodes (renderD*) over primary nodes (card*) — primary nodes require DRM master which is unavailable headless
• Logs which flag combination succeeded (once, at init)

With both changes, the capture priority is:

1. DMA-BUF with relaxed GBM flags (zero-copy GPU path) — works on most headless NVIDIA
2. SHM fallback (CPU memcpy path) — works everywhere, slower

The DMA-BUF path is completely unchanged when GBM succeeds with default flags.

Tested on

• RTX 4060 Ti + RTX 5060 Ti headless (TrueNAS, no monitor connected)
• labwc compositor with WLR_BACKENDS=headless
• GBM succeeds with RENDERING flag, VRAM capture path works end-to-end
• SHM fallback verified by disabling GBM

Screenshot

Issues Fixed or Closed

Roadmap Issues

Type of Change

• feat: New feature (non-breaking change which adds functionality)
• fix: Bug fix (non-breaking change which fixes an issue)
• docs: Documentation only changes
• style: Changes that do not affect the meaning of the code (white-space, formatting, missing semicolons, etc.)
• refactor: Code change that neither fixes a bug nor adds a feature
• perf: Code change that improves performance
• test: Adding missing tests or correcting existing tests
• build: Changes that affect the build system or external dependencies
• ci: Changes to CI configuration files and scripts
• chore: Other changes that don't modify src or test files
• revert: Reverts a previous commit
• BREAKING CHANGE: Introduces a breaking change (can be combined with any type above)

Checklist

• Code follows the style guidelines of this project
• Code has been self-reviewed
• Code has been commented, particularly in hard-to-understand areas
• Code docstring/documentation-blocks for new or existing methods/components have been added or updated
• Unit tests have been added or updated for any new or modified functionality

AI Usage

• None: No AI tools were used in creating this PR
• Light: AI provided minor assistance (formatting, simple suggestions)
• Moderate: AI helped with code generation or debugging specific parts
• Heavy: AI generated most or all of the code changes

[atassis]

I also have a follow-up fix that makes GBM allocation work on headless NVIDIA by trying relaxed usage flags (removing GBM_BO_USE_LINEAR). With this fix, zero-copy DMA-BUF capture works on headless setups — the SHM fallback is only needed if all GBM flag combinations fail.

The change is small (two files):

• wayland.cpp: try GBM_BO_USE_RENDERING alone when RENDERING|LINEAR fails
• cuda.cpp: prefer DRM render nodes (renderD*) over primary nodes (card*) — primary nodes require DRM master which is unavailable headless

Tested on RTX 4060 Ti headless (TrueNAS, no monitor) — GBM succeeds with RENDERING flag, VRAM capture path works end-to-end.

Should I add this to this PR or open a separate one?

[ReenigneArcher]

Should I add this to this PR or open a separate one?

I think it can be included here if we make the PR title more generic.

[atassis]

Added the GBM resilience fix in e54a35d. Updated the PR title and description to cover both changes.
With both patches, DMA-BUF capture now works on headless NVIDIA (tested on RTX 4060 Ti + RTX 5060 Ti, no monitors) - GBM succeeds with RENDERING flag alone. SHM fallback is only needed if all flag combinations fail.

[Clutchnp]

What if the user gives a different adapter name in the config, there will be a mismatch, and this will fail

[atassis]

@Clutchnp could you clarify? The code path changed here (cuda.cpp) finds the DRM node by pci bus ID from the CUDA device - it doesn't read config.adapter_name (that's used by VAAPI/Vulkan encoders in vaapi.cpp / vulkan_encode.cpp, which are unchanged). So there shouldn't be a mismatch with any config value.
If you have a specific scenario in mind, could you describe which config field and which code path you're referring to?

[Clutchnp]

Suppose you have 2 gpus in the system one being AMD and other being NVIDIA, the display is running on AMD , here while you have selected one, in wayland.cpp, selection maybe sth else entirely. This would as far as I understand result in a conflict

Sunshine/src/platform/linux/wayland.cpp

Lines 224 to 261 in 3527320

	bool dmabuf_t::init_gbm() {
	if (gbm_device) {
	return true;
	}
	// Find render node
	drmDevice *devices[16];
	int n = drmGetDevices2(0, devices, 16);
	if (n <= 0) {
	BOOST_LOG(error) << "[wayland] No DRM devices found"sv;
	return false;
	}
	int drm_fd = -1;
	for (int i = 0; i < n; i++) {
	if (devices[i]->available_nodes & (1 << DRM_NODE_RENDER)) {
	drm_fd = open(devices[i]->nodes[DRM_NODE_RENDER], O_RDWR);
	if (drm_fd >= 0) {
	break;
	}
	}
	}
	drmFreeDevices(devices, n);
	if (drm_fd < 0) {
	BOOST_LOG(error) << "[wayland] Failed to open DRM render node"sv;
	return false;
	}
	gbm_device = gbm_create_device(drm_fd);
	if (!gbm_device) {
	close(drm_fd);
	BOOST_LOG(error) << "[wayland] Failed to create GBM device"sv;
	return false;
	}
	return true;
	}

[atassis]

Thanks for pushing back on this - it made me re-audit the change and I found a real (separate) bug in my own patch that I just fixed: the renderD* loop was returning open()'s result directly, so if open() failed (e.g. EACCES, device busy) the function returned -1 without trying the card* fallback. The fix logs a warning and continues to the next node.
On your original concern (multi-GPU mismatch in init_gbm()): the code you linked is unchanged by this PR - it's identical on master. init_gbm() picks the first DRM device with a render node and doesn't read config.adapter_name; that's a real pre-existing limitation on multi-GPU setups (rel. #4940), but introducing GPU selection there would expand this PR's scope significantly beyond "SHM fallback + headless GBM resilience".

[sonarqubecloud]

Quality Gate failed

Failed conditions
4 New issues
2 Duplicated Blocks on New Code (required ≤ 0)
4 New Code Smells (required ≤ 0)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

[Clutchnp]

Maybe due to lack of my understanding, can the drm device that is opened in cuda.cpp be different from the one that is opened in wayland.cpp?

[ReenigneArcher]

Can you fix the conflicts?

[mikefaille]

Hi — heads-up from related work: I'm authoring #5873 (wlgrab RAM bridge + LINEAR path for cross-GPU capture, currently in review), which touches the same regions (listen() signature, create_and_copy_dmabuf) with PCI-identity-gated selection up front.

I read the two designs as complementary rungs of one ladder rather than competing approaches: gated selection first, your SHM fallback as the last resort after allocation failures. Flagging it here mainly to coordinate ordering and avoid clobbering each other's fallback logic — #5873 is further along in review, so one natural path is landing it first and rebasing the SHM rung on top, but I'm open to thoughts.

(Also noting both PRs show the same lizardbyte-gh-pages-sunshine RTD failure, which appears environmental — it fails identically on unrelated PRs.)

[ReenigneArcher]

Please fix conflicts.

[atassis]

Hi there
I'll try to rebase in next few days

[atassis]

Hello again
I'll admit I remember about the PR, just busy with some life related accidents

[atassis]

Rebased onto current master and resolved the conflicts.

I have found some issues during hardware testing. I also caught and fixed a fallback regression in 96aea0f: DMA-BUF creation could succeed but the subsequent screencopy fail, leaving the stream black in a reinitialization loop.
Added regression coverage and verified the previously failing AMD/NVIDIA case through Moonlight, including three reconnect sessions and static-screen capture. Healthy same-GPU DMA-BUF capture still works. The non-hardware suites pass with and without CUDA.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud