Skip to content

Feat/nemotron h lm head logprob cuda - #555

Open
effintell wants to merge 26 commits into
RL-Align:test-nemotronfrom
effintell:feat/nemotron-h-lm-head-logprob-cuda
Open

effintell wants to merge 26 commits into
RL-Align:test-nemotronfrom
effintell:feat/nemotron-h-lm-head-logprob-cuda

Conversation

@effintell

Copy link
Copy Markdown

Implements the CUDA column of the RFC #434 row "Embedding lookup (untied), LM head 2688→131072 fused selected-token logprob (no softcap)" for Nemotron-3-Nano (extends #243, #204, #336).

  • NemotronHLMHeadLogprobSM90Op: strict, batch-invariant fused LM-head selected-token logprob (forward never materialises the [T, 131072] logits) + optional per-row temperature.
  • Strict backward: deterministic dlogits kernel + det-GEMM for grad_hidden / grad_weight, chunked by a fixed element budget (chunking is bit-neutral).
  • Untied embedding lookup with a sorted-segment deterministic grad_weight.
  • Fail closed: geometry gate (hidden 2688 / vocab 131072 / BF16 / SM90) and strict det-GEMM backend requirement raise instead of silently falling back; provenance (contract / checkpoint / kernel ids / det-GEMM backend id / fallback flag) is recorded and checked.
  • Fix: the SM90 det-GEMM tree reduces at most K = 32768 (TREE_DEPTH=10). Backward reductions with K > 32768 (grad_hidden over vocab = 131072; grad_weight over tokens > 32768) previously crashed with an illegal memory access. They are now split into ≤ 32768 segments whose FP32 partials are summed in a fixed ascending order (SM90 backend only; cuBLASLt no-split-K needs no split).
  • grad_weight is judged against FP64 with the WS1 reduction gradient_accuracy tolerance (atol 0.1, rtol 0.02) instead of the logprob cell (atol 0.05, rtol 0) — see "Tolerance" below.

Out of scope: ROCm, TP / CP / EP.

Validation (H100 PCIe, torch 2.8.0+cu128, nvcc 12.8)

Build: KERNEL_ALIGN_FORCE_SM90=1 KERNEL_ALIGN_DET_GEMM_SM90=1 pip install -e . --no-build-isolation

# Command Result
2 python -m pytest tests/models/nemotron_h/test_lm_head_logprob_cuda.py::TestGeometryGate tests/runtime/test_dispatch.py -q (CPU) 24 passed
3a python -m pytest tests/models/nemotron_h -q -rs — SM90 det-GEMM (auto, no cuBLAS env) 34 passed, 1 xfailed, 0 skipped
3b same, cuBLASLt (CUBLAS_WORKSPACE_CONFIG=:16:8 CUBLASLT_WORKSPACE_SIZE=1) 31 passed, 3 skipped (SM90-only split tests, see below), 1 xfailed
4 `python tools/validation/operators/check_operator.py --op nemotron_h_lm_head_logprob --candidate cuda-sm90 --device cuda --dtype bf16 --batch 2 --seq 16 --vocab 131072 --normalized-dim 2688
--check-grad` fails — checker issue, not this op: forward max_abs 0.67 vs atol 0.05; the existing linear_logp suite fails with the identical error on the same shape for both cuda-sm90 and triton
candidates
5 benchmarks/operators/logprob/benchmark_nemotron_h_lm_head_logp.py --json ... see Benchmark
6 python -m pytest tests -q -k "linear_logp or fused_logp" -rs --ignore=tests/backends/rocm/test_rocm_aiter_api_contract.py SM90: 101 passed, 30 skipped; cuBLASLt: 101 passed, 30 skipped (skips:
Ascend / MUSA / 2-GPU only)

The xfail is test_embedding_grad_weight_matches_fp64 (strict xfail): SM90EmbeddingOp accumulates repeated-id gradients in BF16; bitwise reproducibility of that gradient is tested and passes.
test_rocm_aiter_api_contract.py fails to collect on the base branch as well.

Acceptance checks

  • Batch invariance (torch.equal): forward logprob / lse rows across batch sizes 1 / 2 / 7 / 64 / 255 / 257 (random row subsets), leading shape [3, 100], repeat runs, and train vs rollout entry;
    grad_hidden rows across batch sizes 1 / 7 / 64 / 257.
  • FP64 parity: forward and both gradients allclose to an independent FP64 reference / VJP.
  • Fail closed: geometry gate + det-GEMM backend gate raise; no silent fallback (provenance fallback: false).
  • Provenance: recorded per run (det_gemm_backend_id, kernel ids, checkpoint nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16@bf77c31).

det-GEMM split verification (SM90 backend, profiler-confirmed sm90 kernels, rlkernel.det_gemm.sm90.v1)

All cases bitwise reproducible across two runs. "budget" = max_abs / allowed tolerance.

case segments K max_abs vs FP64 BF16 floor reduction budget logprob budget
dW rows=32768 [32768] 0.055 0.022 54% 109%
dW rows=32769 [32768, 1] 0.064 0.016 45% 128%
dW rows=32864 [32768, 96] 0.064 0.021 44% 129%
dW rows=65536 [32768, 32768] 0.090 0.031 65% 181%
dW rows=65600 [32768, 32768, 64] 0.089 0.031 58% 178%
dX V=32832 [32768, 64] 0.0013 0.0005 1% 3%
dX V=131072 4 × 32768 0.0022 0.0010 2% 4%
  • Negative control: forcing the segment limit to 2^40 (no split) reproduces CUDA error: an illegal memory access for dW 32864, dW 65536 and dX 131072.
  • test_grad_weight_token_reduction_beyond_det_gemm_tree is parametrised at exactly 32768 (1 segment), 32769 (K=1 tail) and 65600 (3 segments), asserts the segment count, and skips on non-SM90 backends.
    Disabling the split makes the 2- and 3-segment cases fail; dropping the last partial makes the 3-segment case fail (max diff 0.38 > 0.1).

Tolerance (grad_weight)

grad_weight is a reduction over tokens with entries up to |ref| ≈ 18. At that magnitude BF16 output rounding alone is 0.049 (half-ulp in [16, 32)), which exhausts the logprob cell's absolute-only budget
(atol 0.05). Measured on the 64-token model test: kernel max_err 0.051 vs BF16-rounding floor 0.049 (2 / 352M elements > 0.05). So it is judged with the reduction cell (atol 0.1, rtol 0.02); grad_hidden
stays on the logprob cell (max_err 0.0025).

Note: the SM90 det-GEMM tree accumulates in BF16 per level, so grad_weight error grows with token count — 42–54% of the reduction budget at ~32k tokens, 58–65% at ~65k tokens (≈ 2.5–4× the BF16 floor). The
split itself adds no measurable error.

Benchmark (Nemotron-3-Nano shape, H100 PCIe)

Train fwd+bwd, strict vs native PyTorch:

tokens cuBLASLt strict / native (ms) SM90 det-GEMM strict / native (ms) peak MB native / strict (cuBLASLt)
2048 33.6 / 16.0 (2.1×) 109.7 / 16.0 (6.8×) 3072 / 1195
4096 85.7 / 32.3 (2.7×) 238.1 / 32.5 (7.3×) 6144 / 1717
8192 169.9 / 66.0 (2.6×) 480.5 / 66.3 (7.2×) 12288 / 1738
16384 351.2 / 140.0 (2.5×) 996.9 / 139.6 (7.1×) 24576 / 1780

Forward only (prefill): 2048 tokens 19.1 vs 6.4 ms (3.0×), 8192 tokens 64.7 vs 24.8 ms (2.6×), peak memory halved. cuBLASLt no-split-K is the recommended det-GEMM backend for training.

Known issues / follow-ups

  • csrc/cuda/gemm/det_gemm_kernel.cu: when K is not a multiple of 32 or exceeds 32768, the SM90 det-GEMM silently falls back to the naive kernel instead of failing closed. This PR avoids the K > 32768 case
    from Python; the C++ behaviour will be tracked in a separate issue.
  • The split tests only run under the SM90 det-GEMM build; a cuBLASLt-only CI skips them.
  • Embedding grad_weight FP64 parity (BF16 accumulation) — strict xfail.

@coderabbitai

coderabbitai Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 71705187-f164-44c8-88aa-30a804f0c546

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

…y gate

Signed-off-by: effintell <dominicxhy@gmail.com>
… logprob

Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
… logprob

Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
…ead_logprob forward

Signed-off-by: effintell <dominicxhy@gmail.com>
… parity

Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
…ntraction of temperature scaling

Signed-off-by: effintell <dominicxhy@gmail.com>
…fault-budget chunking and embedding grad_weight

Signed-off-by: effintell <dominicxhy@gmail.com>
…_head_logprob backward

Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
…n gradient tolerance

Signed-off-by: effintell <dominicxhy@gmail.com>
…SM90 det-GEMM 32768 tree limit

Signed-off-by: effintell <dominicxhy@gmail.com>
…2768 boundary

Signed-off-by: effintell <dominicxhy@gmail.com>
@effintell
effintell force-pushed the feat/nemotron-h-lm-head-logprob-cuda branch from 141d9c7 to 21d0bca Compare October 16, 2026 06:18
@effintell
effintell marked this pull request as ready for review October 17, 2026 15:16
@Flink-ddd Flink-ddd added platform: cuda Specific optimizations or bugs in NVIDIA graphics cards (such as FlashInfer, TMA optimizations) new-model Nemotron labels Oct 19, 2026
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

🔍 OpenCodeReview found 5 issue(s) in this PR.

  • ✅ Successfully posted inline: 0 comment(s)
  • 📋 Routed to summary by policy: 5 comment(s)

⚠️ 5 warning(s) occurred during review.


performance · medium

📄 rl_engine/models/nemotron_h/lm_head_contract.py (L79-L79)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity medium · category performance)

_require_id_range forces a device→host sync (.item()) on every call. It is called by validate_lm_head_logprob_geometry, which the backend's training entry logprob() → _prepare() executes on every forward/backward microbatch, not only at validation time. Besides stalling the CUDA stream and preventing CPU run-ahead, .item() raises under CUDA graph capture (where the strict op would otherwise be graph-safe), and the check duplicates the same range guard already performed in sm90_deterministic_linear_logp. Consider gating value checks behind an explicit validation mode (or caching per-static-id-buffer) so the steady-state training path does not pay a per-step sync.


performance · low

📄 rl_engine/models/nemotron_h/reference.py (L91-L91)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity low · category performance)

The oracle materializes full-vocab FP64 tensors that are never chunked along the vocab dimension: weight = lm_head_weight.double() (~2.8 GB) plus grad_weight (~2.8 GB) for the pinned 131072×2688 checkpoint, on top of the BF16 weights and the gradient outputs under test. That is ~5.6 GB of extra FP64 allocation even though the row dimension is streamed. Since the oracle is meant to run alongside the real tensors for the allclose check, chunk the vocab dimension too (or accumulate grad_weight in bounded slices) so the reference cannot OOM independently of the operator under test.


bug · low

📄 rl_engine/models/nemotron_h/lm_head_contract.py (L94-L95)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity low · category bug)

_byte_span assumes data_ptr() is the lowest address and strides are non-negative. For a negative-stride tensor (e.g. a flip/reversed view) data_ptr() points at the highest physical element, so start = data_ptr() plus a positive abs(stride)-derived span yields a byte range that extends past the real allocation instead of covering it. embed_weight reaching _shares_storage is not passed through _require_weight, so a negative-stride view of lm_head_weight's storage is possible; the overlap test can then misjudge the tie verdict this fail-closed gate relies on. Compute the span symmetrically from the addressed min/max addresses (or reject negative strides explicitly).


other · medium

📄 benchmarks/operators/logprob/benchmark_nemotron_h_lm_head_logp.py (L96-L98)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity medium · category other)

The module docstring and strict_over_native assume strict_ms measures the raw kernel ("the geometry contract is validated once per shape outside the timed region"), with validation overhead reported only via strict_api_ms. But sm90_deterministic_linear_logp re-validates on every call: it runs _require_id_range (a (...).any().item() host sync), reshapes/contiguous-copies hidden/weight/target, and (when temperature is set) another .any().item() sync. So strict_ms already includes a per-iteration device->host sync, and strict_api_ms (op.logprob_with_logits -> _prepare + the same function) adds only a second validation pass rather than being the sole source of validation cost. The two timings therefore do not separate kernel cost from validation, and strict_over_native/strict_tokens_per_s are measured against a native closure that performs none of this work, biasing the comparison. Time the raw _C.fused_linear_logp_sm90 entry (as _train_rows does with _NemotronHStrictLMHeadLogpAutograd.apply) for the kernel-only number, or document that strict_ms includes per-call validation.


test · medium

📄 tests/runtime/test_dispatch.py (L252-L254)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity medium · category test)

This subset assertion also passes when the CUDA list is empty, i.e. when the SM90 registration added in registry.py (the new nemotron_h_compiled / cc_major == 9 probe) never fires — a typo in the probed symbol names or a skipped insert would go unnoticed and only surface at call time. Since the intent here is "no fallback", assert the empty case only for non-SM90/non-compiled environments and require the exact singleton when the backend was registered.

💡 Suggested Change

Before:

    assert set(registry._priority_map["cuda"]["nemotron_h_lm_head_logprob"]) <= {
        OpBackend.CUDA_NEMOTRON_H_LM_HEAD_LOGPROB_SM90
    }

After:

    cuda_backends = registry._priority_map["cuda"]["nemotron_h_lm_head_logprob"]
    # Empty is only acceptable when the SM90 op could not be probed (non-SM90 or missing symbols);
    # once registered it must be the sole backend (no silent fallback).
    assert cuda_backends in ([], [OpBackend.CUDA_NEMOTRON_H_LM_HEAD_LOGPROB_SM90])

⚠️ Warnings:

  • rl_engine/models/nemotron_h/__init__.py,rl_engine/models/nemotron_h/lm_head_contract.py,rl_engine/models/nemotron_h/reference.py (review_limit_reached): reached the aggregate token budget before finishing; the grace round completed the task
  • rl_engine/models/nemotron_h/__init__.py (token_budget_reached): skipped round 2 of group "rl_engine/models/nemotron_h/init.py,rl_engine/models/nemotron_h/lm_head_contract.py,rl_engine/models/nemotron_h/reference.py": used 552972 tokens exceeds budget 500000
  • rl_engine/validation/operators/operator_inputs.py (token_budget_reached): stopped dispatch: used 552972 tokens + group estimate 78180 = projected 631152 exceeds budget 500000
  • benchmarks/operators/logprob/benchmark_nemotron_h_lm_head_logp.py,csrc/bindings/ops.cpp,csrc/cuda/logprob/fused_linear_logp_sm90.cu,rl_engine/_C.pyi,rl_engine/backends/cuda/logprob/nemotron_h_lm_head_logp.py,rl_engine/kernels/ops/cuda/loss/nemotron_h_lm_head_logp.py (review_limit_reached): reached the aggregate token budget before finishing; the grace round did not complete the task
  • rl_engine/runtime/provenance/module_names.json,rl_engine/runtime/registry.py,tests/runtime/test_dispatch.py (review_limit_reached): reached the aggregate token budget before finishing; the grace round did not complete the task

@github-actions

Copy link
Copy Markdown

@effintell The automated OCR review has completed.

Please review and address all automated review bot findings, then push your updates. Maintainers will begin manual review after the bot findings have been addressed.

…ned bf16 geometry

Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
@effintell
effintell force-pushed the feat/nemotron-h-lm-head-logprob-cuda branch from 08f3c31 to 0c881f9 Compare November 16, 2026 16:08

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Nemotron new-model platform: cuda Specific optimizations or bugs in NVIDIA graphics cards (such as FlashInfer, TMA optimizations)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants