Skip to content

[Nemotron][CUDA] Add deterministic moe_experts_combine - #661

Open
luoyueyuguang wants to merge 11 commits into
RL-Align:test-nemotronfrom
luoyueyuguang:feat/nemotron-experts-combine
Open

luoyueyuguang wants to merge 11 commits into
RL-Align:test-nemotronfrom
luoyueyuguang:feat/nemotron-experts-combine

Conversation

@luoyueyuguang

Copy link
Copy Markdown
Contributor

Implements the SM90 CUDA moe_experts_combine work item of #434: grouped up projection with a relu-square epilogue, grouped down projection, the fixed-order weighted combine over the six route slots, the optional shared expert and the operator's single exit cast — forward and backward — at the Nemotron-3-Nano-30B-A3B geometry (hidden 2688, intermediate 1856, 128 routed experts + 1 shared expert of 3712, top-6, BF16 payload/weights, FP32 router weights and combine). Target branch test-nemotron.

This PR fills the CUDA column only (§7 "one row × one platform = one PR"). The ROCm column is a separate PR owned separately: the shared Triton provider is included here because the CUDA column needs it as the fallback, and it is the entry point the ROCm column will share, but nothing in this PR claims gfx9xx qualification. Per §7 both columns implement one versioned contract (nemotron-experts-sm90-v1) and share one test suite (tests/backends/nemotron_experts_cases.py, imported by both the CUDA and the ROCm provider suites).

What changed, and why each layer owns its part

Layer File Responsibility
Kernel csrc/cuda/moe/nemotron_experts.cu Both instruction paths of the grouped GEMM, the fixed-order combine, the FP32 route-weight dot, dY, the permutation inverse, and the operator's single exit cast
Native provider rl_engine/backends/cuda/moe/nemotron_experts.py autograd wiring; raises instead of falling back when the SM90 build flag is absent
Operator entry rl_engine/ops/moe/nemotron_experts.py hardware-independent validation, MAX_TOKENS bound, one provider-route report line
Shared provider rl_engine/backends/shared/triton/moe/nemotron_experts.py CUDA fallback and the only ROCm implementation
Reference rl_engine/reference/moe/nemotron_experts.py independent scalar FP64 oracle (never shares code with a kernel)
Registry rl_engine/runtime/registry.py moe_experts_combine priority: native → Triton on CUDA, Triton only on ROCm, fail closed on cpu/npu/musa/mps
Bindings csrc/bindings/ops.cpp, rl_engine/_C.pyi, build_tools/extensions.py symbol surface and build wiring; the .cu is not hipified
Tests tests/backends/cuda/, tests/backends/rocm/, tests/backends/nemotron_experts_cases.py, tests/reference/ provider suites plus the CPU-only FP64 oracle
Benchmark benchmarks/operators/moe/benchmark_nemotron_experts_training.py eager per-expert FP32 baseline vs the providers, with an A/B flag for the two instruction paths
Docs / CI docs/operators/nemotron-experts.md, docs/.nav.yml, docs/operators/README.md, .github/workflows/ci.yml operator page + the CPU-safe contract tests in CI

The arithmetic is the det_gemm contract, copied deliberately: a 32-wide FP32 leaf rounded once to BF16, leaves merged with the canonical midpoint tree, fixed ascending K order, one CTA per output tile, no split-K, no atomics, no TF32. What differs is the grouping: each CTA resolves its own (expert, row tile) from the 129 prefix sums in shared memory, so the host never reads offsets and CUDA Graph capture stays legal; the stacked weights are one tensor map over [E·F, H] / [E·H, F]; combine slots are summed in ascending order into an FP32 accumulator.

Two instruction paths satisfy that one contract and are bitwise identical:

sm90_v1 sm90_v2 (default)
Tile BM=128, BN=64, BK=32 BM=64, BN=128, KT=64 (two leaves)
MMA 2 × mma.sync.m16n8k16 per leaf 2 × wgmma.mma_async.m64n128k16 per leaf
Shared memory row-major, TMA swizzle none, ldmatrix TMA SWIZZLE_128B, wgmma reads descriptors
Pipeline 2 stages, __syncthreads in the K loop 3 stages, producer warp + 4-warp consumer warpgroup, mbarriers only
SASS HMMA 32, LDSM 12, REG 66-82, STACK 1552 HGMMA 6, LDSM 0, REG 135 (170 for dW), STACK 1568

RL_KERNEL_NEMOTRON_EXPERTS_BACKEND=sm90_v1|sm90_v2|auto selects the path and is read per launch, so both can be measured in one process. The wgmma path is self-contained in nemotron_experts.cu (descriptor encoding, wgmma wrappers, swizzled tensor map, extra barrier primitives); the mma.sync path stays as the shape fallback. The relu-square operand is materialized by the up epilogue as Z = bf16(relu(U)²) — a BF16 product is exact in FP32, so this is bit-identical to transforming every operand tile, and it removes that transform from the down and dW_down stages.

Out of scope: ROCm hardware execution (the shared Triton provider runs there but was not re-qualified on gfx9xx), TP / CP / EP (separate rows of #434), prefill/decode cache state (this operator is stateless), and the router (separate row, already merged).

Environment covered

One NVIDIA H100 NVL (SM90, 95 GiB), driver 580.173.02, CUDA 13.0, PyTorch 2.11.0+cu130, CUDA-13.0 nvcc, host GCC from the CUDA 13.0 image. Build flags: KERNEL_ALIGN_FORCE_SM90=1 KERNEL_ALIGN_DET_GEMM_SM90=1. Single GPU, no distributed topology. BF16 in/out with FP32 accumulate; FP32 router weights. Not exercised: ROCm, Ascend, MUSA, other NVIDIA SKUs, CUDA 12.x, multi-GPU.

Numerical-contract evidence

  1. Raw-bit invariance (the operator's claim). For a fixed token row and its upstream cotangent, out, dPacked, dX, dWeights and dW are torch.equal across batch sizes (1 … 32768 tokens), token positions, packed layouts (random and concentrated routing), repeated runs and CUDA Graph replay. The operator is stateless, so the RFC's sequence-split clause reduces to this layout invariance: no cache, no cross-CTA merge.
  2. Path parity. test_instruction_paths_agree_bitwise runs both native paths in one process and compares every output and gradient with torch.equal, on the core geometry at 17 and 129 tokens (per-expert segment lengths that are not multiples of the 32-wide leaf, i.e. the weight-gradient tail padding) and on the Nemotron geometry at 64 tokens (truncated 128-wide column tiles, N=1856). 3 cases, all pass.
  3. Contract parity with det_gemm. The up and down projections, dPacked and both weight gradients are pinned bitwise against _C.det_gemm_fwd_rhs_transposed / det_gemm_fwd / det_gemm_db_transposed on real shapes; 4 cases.
  4. Accuracy against a mathematical reference. Relative L2 against an independent FP64 pipeline at 128 tokens, shared expert on, per gradient (worst per provider):
Quantity native Triton
forward 8.83e-3 1.00e-2
dPacked 8.61e-3 8.61e-3
dX 8.66e-3 9.05e-3
dWeights 8.73e-3 9.35e-3
worst weight gradient 7.77e-3 8.78e-3

The floor is set by the contract itself (leaf rounding at BF16 eps 7.8e-3 plus six signed terms before the single exit cast), so these numbers are the pipeline's own quantization, not a kernel defect. The det_gemm parity rows above are the exactness evidence; these L2 figures are the accuracy evidence against mathematics.
5. Not claimed. Bitwise equality between providers (native and Triton associate leaf sums differently; measured 8.3e-3 core / 1.3e-2 Nemotron relative L2, near one BF16 ulp at the output). dW bitwise reproducibility for the same ordered token set across microbatch splits — adding independently computed microbatch dW is not bitwise full-batch equivalence, and neither is claimed for any provider. Cross-compiler equivalence. TP/CP/EP.

Tests and benchmarks actually run

# build
KERNEL_ALIGN_FORCE_SM90=1 KERNEL_ALIGN_DET_GEMM_SM90=1 python setup.py build_ext --inplace

# this operator (SM90 H100 NVL, no skips)
python -m pytest tests/backends/cuda/test_nemotron_experts_cuda.py -q

# neighbours: the det_gemm/router/wrapper contracts this operator reuses
python -m pytest tests/ops/gemm/test_det_gemm.py \
  tests/backends/cuda/test_sm90_linear_wrappers.py \
  tests/backends/cuda/test_nemotron_router_cuda.py \
  tests/runtime/test_kernel_registry.py \
  tests/reference/test_nemotron_experts_reference.py \
  tests/reference/test_nemotron_router_reference.py \
  tests/backends/rocm/test_nemotron_experts_rocm.py -q

# benchmark: eager + production grouped GEMM + both native paths + Triton, one paired run
PYTHONPATH=. python benchmarks/operators/moe/benchmark_nemotron_experts_training.py \
  --tokens 1 16 128 513 1024 4096 8192 --dtype bf16 --providers native triton \
  --baselines eager grouped_mm --phases forward --distributions random concentrated \
  --native-backend sm90_v2 --output /tmp/experts-fwd-v2.json
#   ... --native-backend sm90_v1 --output /tmp/experts-fwd-v1.json   (the mma.sync column)
PYTHONPATH=. python benchmarks/operators/moe/benchmark_nemotron_experts_training.py \
  --tokens 1 16 128 513 1024 4096 8192 --dtype bf16 --providers native triton \
  --baselines eager --phases forward_backward --distributions random concentrated \
  --native-backend sm90_v2 --qualify-max-tokens 128 --output /tmp/experts-bwd-v2.json
# 32768 tokens is a separate invocation (four candidates at that size exceed the memory budget)

# docs / formatting / types
python -m black --check --line-length 100 <the three python files touched>
python -m mypy --ignore-missing-imports rl_engine/backends/cuda/moe/nemotron_experts.py
python -m mkdocs build --strict
Suite Result
tests/backends/cuda/test_nemotron_experts_cuda.py 28 passed, 0 skipped (native + Triton, FP64 qualification, raw-bit batch/relocation invariance, graph replay, autocast, provider selection/rejection, 4 det_gemm bitwise cases, 3 path-parity cases)
neighbouring suites above (7 files) 150 passed, 54 skipped (49 = det_gemm's "Triton is unavailable" and ROCm gfx942 rows in this build, 5 = the ROCm provider suite without a HIP device)
black --check / mypy / mkdocs --strict clean; mkdocs exits 0
torch.profiler + cuobjdump -res-usage/-sass per-stage attribution and the SASS/occupancy figures quoted in the docs

Not exercised: ROCm hardware, Ascend/MUSA (fail closed), TP/CP/EP, multi-GPU, CUDA 12.x, non-Hopper NVIDIA parts.

Performance

Workload: the real Nemotron geometry above, BF16, CUDA Graph replay, five untimed warmup executions per candidate before capture, 20 alternating paired samples of 5 executions, median reported, cotangents prepared outside timing, all columns measured in one sequential session on an idle device at its boost clock (1785/1785 MHz). Reported as measured differences, no automatic regression threshold (§7 leaves the target ratio open; see "Where the remaining gap is").

Candidates: the eager formulation (torch.matmul expert by expert in FP32 + slot-ascending combine), grouped_mm = torch._grouped_mm, the cuBLAS/CUTLASS grouped GEMM a serving stack calls (forward only; the stacked weights are pre-transposed outside the timed region), the two native instruction paths, and the shared Triton provider.

Forward, random routing, median ms:

T baseline grouped_mm native v1 native v2 Triton v1/v2 grouped_mm/v2
1 0.477 0.209 0.782 0.358 0.473 2.18x 0.58x
16 4.277 0.737 3.558 1.416 1.818 2.51x 0.52x
128 10.088 1.147 5.925 2.315 3.106 2.56x 0.50x
513 11.823 1.328 6.268 2.480 3.390 2.53x 0.54x
1024 14.210 1.535 6.608 2.914 3.670 2.27x 0.53x
4096 29.604 3.587 15.200 9.850 8.967 1.54x 0.36x
8192 49.075 6.903 28.293 19.081 16.470 1.48x 0.36x
32768 164.884 27.742 107.390 72.175 65.414 1.49x 0.38x

Forward + backward, random routing, median ms:

T baseline native v1 native v2 Triton v1/v2 baseline/v2
1 16.655 4.365 3.595 1.773 1.21x 4.63x
16 217.770 10.688 7.127 5.124 1.50x 30.56x
128 403.635 16.789 10.803 8.323 1.55x 37.36x
513 414.544 18.510 11.237 8.983 1.65x 36.89x
1024 425.932 21.360 12.624 10.647 1.69x 33.74x
4096 503.735 51.599 32.914 26.341 1.57x 15.30x
8192 602.836 95.971 58.188 49.231 1.65x 10.36x
32768 1201.012 357.091 215.282 184.260 1.66x 5.58x

Concentrated routing (v2 / v1 / Triton forward+backward): 3.564 / 4.314 / 1.775 ms at T=1, 28.405 / 44.880 / 22.144 ms at 4096, 218.603 / 356.957 / 183.355 ms at 32768 — the wgmma path is 1.2-1.6x over mma.sync and 2.3-4.7x over the baseline there too.

Peak memory (increment over the weight stack, random routing, MiB): forward+backward native 2496 at T=1, 2713 at 1024, 4349 at 8192; Triton 2496 / 2684 / 4117; eager baseline 3878 / 8775 / 10613. Forward alone: native 1094 MiB at 8192 against 1010 for grouped_mm and 913 for the baseline. Both native paths allocate identically because both materialize Z (182 MiB of the 8192 row).

The production path is faster, and here is why. grouped_mm is 1.7-2.8x faster than this operator in the forward (1.6-2.9x under concentrated routing) and the gap widens with size. It is not addressed by any schedule change: the grouped GEMM holds an FP32 accumulator and casts once per output element, while the contract rounds and tree-merges every 32-wide leaf, i.e. K/32 extra SIMT passes per element. At T=4096 the forward+backward performs 30.7e9 element-leaves in 29.5 ms, which needs 0.98e12 warp-instructions/s against the 0.94e12 this part can issue — the operator is at the contract's SIMT instruction-issue ceiling. Closing the 2.7x means a wider leaf or a single cast, i.e. an arithmetic decision for the contract owner, not a kernel decision.

What is still not measured: a production training baseline (Transformer Engine GroupedLinear performs grouped dgrad + wgrad; it is pinned at 2.19.0 in this repository and neither installs nor builds against this container's CUDA 13.0 / torch 2.11.0+cu130, and vLLM's fused MoE is not installed). torch._grouped_mm groups over token rows only, so dW_up[e] = dU_eᵀ·packed_e — whose reduction length differs per expert — is not expressible with it, which is why the forward+backward tables still compare against the eager formulation.

Documentation and compatibility

  • New operator page docs/operators/nemotron-experts.md (contract, interface, fixed arithmetic, ordering policy, instruction paths, qualification boundary, tests, performance reproduction, verification record), added to docs/.nav.yml and docs/operators/README.md.
  • No public API change: the entry point is rl_engine.ops.moe.nemotron_experts.nemotron_experts(...); provider ids stay rlkernel.nemotron_experts.sm90.v1 / .triton.v1 (the contract version does not change with the instruction path, which is bit-neutral). New optional environment variable RL_KERNEL_NEMOTRON_EXPERTS_BACKEND (defaults to the wgmma path).
  • Registry: moe_experts_combine resolves native → Triton on CUDA and Triton only on ROCm; unsupported platforms raise.
  • CI: .github/workflows/ci.yml now runs the CPU-safe reference/contract tests for this operator (the CUDA/ROCm provider suites skip without a device).

§7 contribution-note compliance

§7 requirement Status Evidence
One row × one platform = one PR met This PR is the CUDA column of moe_experts_combine, with its validation in the same PR. The row is not marked done: the ROCm column is a separate PR.
CUDA and ROCm PRs implement the same versioned contract and share the test suite met Contract nemotron-experts-sm90-v1; one shared suite tests/backends/nemotron_experts_cases.py imported by tests/backends/cuda/test_nemotron_experts_cuda.py and tests/backends/rocm/test_nemotron_experts_rocm.py; only the kernel and the native baseline differ per platform.
A single Triton PR may fill both columns only if it passes full acceptance on each platform met by not claiming it The Triton provider is here as the CUDA fallback; the ROCm column's acceptance is not claimed (no gfx9xx host was available to me).
Operator scope completeness (mamba2_ssd_mixer bullet) N/A, analogue met That bullet governs the Mamba row. The analogous requirement for this row — forward and backward, every operand mode and every routing density — is covered: both directions, all four GEMM modes, random and concentrated routing, empty experts.
Performance against the native path, decode + training met for forward, declared gap for training Decode (T=1/16/128) and training (T=1024…32768) shapes, forward and forward+backward, against three references: the eager formulation, torch._grouped_mm (the cuBLAS/CUTLASS grouped GEMM production path, forward only) and the shared Triton provider, with warmup, 20 paired samples, medians and peak memory. The production path is 1.7-2.8x faster in the forward and that gap is the contract's leaf rounding, quantified at the SIMT issue ceiling. No production training baseline exists in this container (TE GroupedLinear / vLLM absent; _grouped_mm cannot express grouped wgrad), which is declared rather than substituted.
Oracles: torch.equal between two strict paths, allclose vs an independent FP64 reference, never against shared code met Parity: both native paths vs each other and vs _C.det_gemm_* (3 + 4 cases, torch.equal). Correctness: rl_engine/reference/moe/nemotron_experts.py, a hand-written FP64 oracle that shares no code with any kernel (CPU-only suite + FP64 L2 report). The cross-provider number is reported as a comparison, never as the correctness oracle.
MoE ordering policy stated explicitly met Stated in the operator page §"Ordering policy": experts ascend (the CTA walks the 129 prefix sums), combine slots ascend (six slots into one FP32 accumulator), dW visits grouped rows ascending and empty experts write exact zeros, and the router's permutation/offsets (a bijection and its exclusive prefix sums) are the ordering input. No claim rests on "deterministic on my GPU": invariance is asserted with torch.equal across batch, position, packing, repeat and graph replay. Policy is marked proposed for §4.4 review, consistent with #504.
Fail closed met, with a declared trust boundary Raises instead of silently changing arithmetic: non-SM90 device, missing SM90 build flag, dtype/shape/rank mismatch, widths not 64-aligned, tokens > 65536, offsets.numel() != experts+1, unpaired shared weights, non-contiguous or wrong-device inputs. permutation/offsets values are trusted as the router's contract rather than validated on device (a device-side check would force a host sync and break graph capture); that boundary is documented in the operator page and stated here rather than hidden. The registry may fall through to the Triton provider when the native build is absent — that is a switch between two strict, deterministic implementations of one contract, not a fallback from strict to non-strict arithmetic.
§8 open questions touched answered by this PR Performance targets: "report only" (§7 bullet 5 imposes no automatic ratio) — this PR reports, and additionally shows the throughput is at the contract's SIMT instruction-issue ceiling. Tolerance calibration: no 1.1× native slack is claimed; the operator reports measured FP64 L2 and the BF16 floor instead.
§9 out of scope respected met No LatentMoE, no MTP/speculative decoding, no NVFP4: Nano BF16 only.

Pending

  • Production training baseline: forward+backward against Transformer Engine GroupedLinear (pinned 2.19.0) or vLLM fused MoE at the same shapes. Needs a host where that stack builds against CUDA 13.0 / torch 2.11; the forward comparison already uses torch._grouped_mm.
  • ROCm column: gfx9xx re-qualification of the shared Triton provider (authored from the CUDA run, same source) — separate PR, separate column.
  • Maintainer review of the expert-ordering and combine-ordering policy against §4.4.
  • TP / CP / EP invariance belong to the separate rows of [RFC][Nemotron-3-Nano-30B-A3B][CUDA/ROCm] ROADMAP #434.

…Triton)

The `moe_experts_combine` row of RFC RL-Align#434: grouped up projection with a
relu-square epilogue, grouped down projection, the fixed-order weighted combine
over the six route slots, the optional shared expert and the operator's single
exit cast, forward and backward, at the Nemotron-3-Nano-30B-A3B shapes
(2688/1856, 128 routed experts + 1 shared, top-6).

The arithmetic is the det_gemm contract, copied deliberately: a 32-wide FP32
leaf rounded once to BF16, leaves merged with the canonical midpoint tree, fixed
ascending K order, one CTA per output tile, no split-K, no atomics.  What
differs is the mapping.  Each CTA resolves its own (expert, row tile) from the
129 prefix sums in shared memory, so the host never reads `offsets` and CUDA
Graph capture stays legal; the stacked weights are one tensor map over `[E*F,H]`
and `[E*H,F]`; one grid serves forward, backward and both weight gradients.

Two providers satisfy the contract: `csrc/cuda/moe/nemotron_experts.cu` for
NVIDIA SM90 (mma.sync + TMA) and the shared Triton kernel for CUDA and ROCm.
The registry lists native first and Triton second on CUDA, only Triton on ROCm,
and fails closed on cpu/npu/musa/mps.  `tests/backends/cuda/` pins the up and
down projections, `dPacked` and both weight gradients bitwise against
`_C.det_gemm_*`; the reference suite is an independent FP64 oracle.

Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
A second instruction path for the same contract: TMA with a 128-byte swizzle,
`wgmma.mma_async` reading both operands from shared-memory descriptors, one
producer warp plus a four-warp consumer warpgroup, three stages, 64x128 tiles and
two static accumulators so one leaf is rounded and merged while the next is still
on the tensor pipe.  It is self-contained in `csrc/cuda/moe/nemotron_experts.cu`
- the descriptor encoding, the wgmma wrappers, the swizzled tensor map and the
extra barrier primitives are that file's own - and the mma.sync path stays as the
shape fallback.  `RL_KERNEL_NEMOTRON_EXPERTS_BACKEND=sm90_v1|sm90_v2|auto`
selects between them and is read per launch, so one process can measure both.

Grouped additions over a plain GEMM: either operand may be K-major or MN-major,
which fixes the descriptor start offsets and the instruction's imm-trans flag; a
weight gradient's last reduction tile is zero-padded in shared memory, because a
segment length is not a multiple of the 32-wide leaf.  The relu-square operand is
now materialized by the up epilogue as `Z = bf16(relu(U)**2)` instead of being
recomputed on every down and dW_down operand tile.

Both paths are bitwise identical on every output and gradient
(`test_instruction_paths_agree_bitwise`, 17/129/64 tokens, the small ones with
per-expert segment lengths that are not leaf-aligned) and still match
`_C.det_gemm_*` bitwise.  The benchmark gains `--baselines grouped_mm|eager` and
`--phases`, so the production grouped-GEMM path (`torch._grouped_mm`, cuBLAS /
CUTLASS) is measured next to the eager formulation in the same paired run.

Measured on one H100 NVL at T=4096, forward+backward: 47.6 -> 29.5 ms
(41 -> 66.5 TFLOP/s), per stage 1.31-1.83x.  Benchmark medians over T=1..32768
give 1.4-2.6x for forward and 1.2-1.7x for forward+backward, with the
eager-baseline gap at 2.3-37x; the T=1 forward regression of the mma.sync path
(0.782 ms against a 0.477 ms eager baseline) is gone at 0.358 ms.  The production
grouped GEMM is 1.7-2.8x faster in the forward and that is the contract's price,
not the schedule's: the kernel sits at the SIMT instruction-issue ceiling -
30.7e9 element-leaves in 29.5 ms needs 0.98e12 warp-instructions/s against 0.94e12
issuable - so anything further is a leaf-width arithmetic decision.

Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
@coderabbitai

coderabbitai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 6f35f846-75c0-4bfd-8da6-07b632c11909

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

🔍 OpenCodeReview found 3 issue(s) in this PR.

  • ✅ Successfully posted inline: 0 comment(s)
  • 📋 Routed to summary by policy: 3 comment(s)

⚠️ 5 warning(s) occurred during review.


bug · medium

📄 rl_engine/runtime/registry.py (L550-L553)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity medium · category bug)

CUDA_NEMOTRON_EXPERTS is listed unconditionally as the first candidate for every CUDA device, unlike the other SM90-only kernels in this file (logp, linear_logp, batch_invariant_logp, embedding, lm_head), which are only inserted into the priority list by _adjust_priority_for_hardware when cc_major == 9.

Concrete trigger: a non-SM90 CUDA GPU (A100/Ada, cc_major 8) with the extension built with KERNEL_ALIGN_DET_GEMM_SM90=1 (the common CI/build configuration). get_op("moe_experts_combine", device=...) selects NemotronExpertsOp and caches it, because NemotronExpertsOp.__init__ only checks the build flag (_C.nemotron_experts_compiled()) and never probes device capability. The first __call__ then hits _require_sm90 and raises RuntimeError, so the registered TRITON_NEMOTRON_EXPERTS fallback is never reached and the op is unusable on that device even though the Triton provider supports it.

Fix: gate eligibility on capability before selection — e.g. in _adjust_priority_for_hardware, only insert/prepend CUDA_NEMOTRON_EXPERTS when cc_major == 9 (mirroring the existing SM90 insertions) so non-SM90 devices fall through to the Triton provider.


maintainability · medium

📄 benchmarks/operators/moe/benchmark_nemotron_experts_training.py (L441-L441)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity medium · category maintainability)

provider_ids records the selected provider class id only. For the native provider that id is a fixed string (rlkernel.nemotron_experts.sm90.v1), even though --native-backend (default auto) may select and execute sm90_v2. Because the run also exports RL_KERNEL_NEMOTRON_EXPERTS_BACKEND=auto, the saved result cannot tell which instruction path actually produced the timings — the very thing the CUDA tests deliberately pin with _C.nemotron_experts_last_path(). Record the executed path (e.g. _C.nemotron_experts_last_path() after the first call, falling back to provider_id) so a reader can attribute the numbers to a specific backend.


other · medium

📄 benchmarks/operators/moe/benchmark_nemotron_experts_training.py (L518-L520)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity medium · category other)

The default --baselines is ["eager"], and the eager baseline widens every operand to FP32 (see baseline_calls.body), so with the default arguments speedup_*_vs_baseline compares a BF16 operator (FP32 accumulation) against an FP32 reference that does twice the memory traffic. That ratio measures precision choice plus kernel efficiency, not two equivalent-precision GEMMs, and it is emitted as an unqualified speedup_..._vs_baseline field in every row. The BF16 grouped_mm row that would be the apples-to-apples comparison is opt-in. Either make the BF16 grouped_mm baseline part of the default set, or label/keep the FP32 comparison from being reported as a plain speedup so downstream conclusions are not inflated.


⚠️ Warnings:

  • rl_engine/backends/cuda/moe/nemotron_experts.py,rl_engine/backends/shared/triton/moe/__init__.py,rl_engine/backends/shared/triton/moe/nemotron_experts.py,rl_engine/ops/moe/nemotron_experts.py,rl_engine/reference/moe/nemotron_experts.py,rl_engine/runtime/registry.py (review_limit_reached): reached the aggregate token budget before finishing; the grace round did not complete the task
  • rl_engine/backends/cuda/moe/nemotron_experts.py (token_budget_reached): stopped group "rl_engine/backends/cuda/moe/nemotron_experts.py,rl_engine/backends/shared/triton/moe/init.py,rl_engine/backends/shared/triton/moe/nemotron_experts.py,rl_engine/ops/moe/nemotron_experts.py,rl_engine/reference/moe/nemotron_experts.py,rl_engine/runtime/registry.py" mid-review: used 661328 tokens exceeds budget 500000
  • .github/workflows/ci.yml (token_budget_reached): stopped dispatch: used 661328 tokens + group estimate 45480 = projected 706808 exceeds budget 500000
  • build_tools/extensions.py,csrc/bindings/ops.cpp,csrc/cuda/moe/nemotron_experts.cu,rl_engine/_C.pyi (review_limit_reached): reached the aggregate token budget before finishing; the grace round did not complete the task
  • benchmarks/operators/moe/benchmark_nemotron_experts_training.py,tests/backends/cuda/test_nemotron_experts_cuda.py,tests/backends/nemotron_experts_cases.py,tests/backends/rocm/test_nemotron_experts_rocm.py,tests/reference/test_nemotron_experts_reference.py (review_limit_reached): reached the aggregate token budget before finishing; the grace round did not complete the task

Comment thread csrc/cuda/moe/nemotron_experts.cu Resolved
Comment thread csrc/cuda/moe/nemotron_experts.cu Resolved
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

@luoyueyuguang The automated OCR review has completed.

Please review and address all automated review bot findings, then push your updates. Maintainers will begin manual review after the bot findings have been addressed.

OCR review of RL-Align#661 raised two inline defects and five test/reference issues.

`tree_stk` bounds: the midpoint merge keeps one slot per level, so `TREE_DEPTH
= 12` only covered `kd <= 4096`, while a weight gradient's reduction depth is the
expert segment length and one segment may hold up to 6 * MAX_TOKENS = 393216
packed rows (`kd = 12288`). Widen the stack to 14 and fail closed in
`run_grouped` above `1 << TREE_DEPTH` leaves. The guard bounds the row count, not
`offsets`, so the host still never reads the routing table and graph capture
stays legal.

Shared-expert weight gradients were allocated `[1,F,H]`/`[1,H,F]` for rank-two
`w_sh1`/`w_sh2`. Autograd accepted them (it passed the suite) and callers got a
mis-shaped `.grad`; the allocations are rank two now, and `cases.run` asserts
every gradient's shape against its input so the class stays covered.

Reference dtype guard: it demanded one dtype for payload, route weights and
stacked weights, which excluded the operator's own pairing (BF16 payload with
FP32 route weights). It now accepts that pairing and rejects the rest, and the
cast-point test covers three pairings instead of two, with a companion test
pinning that BF16 and FP32 payloads share one rounding band while an FP64 payload
lands at 1e-13 - the bound now discriminates the cast points rather than the
payload dtype.

Test defects: the empty-batch case kept the 17-token `offsets` (now zeroed) and
went through the registry instead of the parametrized provider (now the provider
object, so the Triton column is really exercised); `test_reject_bad_geometry` was
parametrized over providers although validation is hardware-independent (now a
single test); the instruction-path parity test reused one provider instance
across both env scopes, so it could degenerate into a self-comparison - it now
resolves a provider per scope and asserts the new
`nemotron_experts_preferred_path()` reports the path it thinks it is testing.

Re-validated on an H100 80GB HBM3 (CUDA 13.0, PyTorch 2.12.0+cu130): 180 passed,
54 skipped across the operator, reference and neighbour suites; `black --check`,
`mypy` and `mkdocs --strict` clean; the instruction-path ratio is unchanged
(T=4096 forward+backward 27.5 ms against 43.5 ms, 1.58x), so the wider merge
stack costs nothing measurable.

Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
@luoyueyuguang

Copy link
Copy Markdown
Contributor Author

All seven findings addressed in 5a087d7a. Two were real defects, the rest were test/reference gaps; one I pushed back on after measuring.

Inline 1 — tree_stk out of bounds for a long weight-gradient segment (security/high): confirmed, fixed. TREE_DEPTH = 12 covered kd <= 4096, but a weight gradient's reduction depth is the expert segment length, and one segment may hold up to 6 * MAX_TOKENS = 393216 packed rows (kd = 12288). The stack is now 14 slots (covers kd <= 16384) and run_grouped raises above 1 << TREE_DEPTH leaves. The guard bounds the row count rather than offsets, so the host still never reads the routing table and CUDA Graph capture stays legal. Documented in the operator page.

Inline 2 — shared-expert weight gradients returned rank three (bug/high): confirmed, fixed. d_w_sh1/d_w_sh2 were allocated [1,F,H]/[1,H,F] for rank-two parameters. It did not raise — the suite passed before this change because autograd broadcast the leading dimension away and the shared assertions compare relative L2 — so callers silently got a mis-shaped .grad. The allocations are rank two now (the WGRAD kernel writes [M,N] from offset zero, so the storage is unchanged), and cases.run asserts every gradient's shape against its input, which covers the class for all providers.

Reference dtype guard (medium): confirmed, fixed. rl_engine/reference/moe/nemotron_experts.py now accepts the operator's canonical pairing (BF16 payload + FP32 route weights) and rejects everything else, so experts_reference can be called with the dtype combination moe_experts_combine actually consumes. The docstring states the rule.

Empty-batch case kept a stale routing table (medium): fixed. empty["offsets"] is zeroed for the zero-token case, and the test drives the parametrized provider object instead of nemotron_experts(...), so the triton row now exercises the Triton op rather than the registry's native pick.

Parity test could degenerate into a self-comparison (medium): fixed. Each env scope resolves its own provider, and the test asserts a new _C.nemotron_experts_preferred_path() reports sm90_v1 / sm90_v2 respectively — so if the selection were ever cached on the instance, the test fails instead of comparing a path with itself. The accessor is also usable as run provenance.

Unused name parametrization (low): fixed. test_reject_bad_geometry is a single test (input validation is hardware-independent, so there is nothing to parametrize); test_empty_batch keeps its native/triton rows and now really drives them.

FP32 cast-point bound (low): investigated, partially pushed back. The premise that an FP32 payload means "exit cast only" does not hold for this reference: _leaf_round rounds the up result and the gate to BF16 regardless of the payload dtype, so both payload dtypes sit in the same ~1e-3 band (measured: FP32 3.1e-3, BF16 2.4e-3) and the error there is not dominated by the exit cast alone. Instead of per-dtype thresholds I parametrized the test over three pairings (including the canonical BF16-payload/FP32-routes one) with the two-sided bound, and added test_payload_dtype_does_not_move_the_rounding_band: BF16 and FP32 payloads must agree within 3x while an FP64 payload lands below 1e-12 — that is what discriminates "the cast points exist" from "the path is unrounded", without a tuned threshold.

The five token_budget_reached warnings mean those groups were not fully read by the bot; I re-read them myself and found nothing beyond the above — csrc/bindings/ops.cpp, _C.pyi, registry.py and build_tools/extensions.py are wiring only, and the benchmark's --baselines/--phases additions are measurement-only.

Re-validated on an idle H100 80GB HBM3 (CUDA 13.0, PyTorch 2.12.0+cu130, KERNEL_ALIGN_FORCE_SM90=1 KERNEL_ALIGN_DET_GEMM_SM90=1):

  • tests/backends/cuda/test_nemotron_experts_cuda.py + tests/reference/test_nemotron_experts_reference.py: 45 passed, 0 skipped;
  • det_gemm, router, wrapper, registry, reference and ROCm neighbours: 135 passed, 54 skipped (the skips are Triton-unavailable/ROCm-only rows of other operators);
  • black --check --line-length 100, mypy --ignore-missing-imports, mkdocs build --strict: clean;
  • the wider merge stack costs nothing measurable: at T=4096 forward+backward the wgmma path is 27.5 ms against the mma.sync path's 43.5 ms (1.58x, matching the 1.57-1.61x in the tables, which stay H100 NVL figures).

…iring)

The re-review of RL-Align#661 found the same two defects in the shared Triton provider
plus two unreachable-looking tail bugs and a CI false green.

Triton shared-expert weight gradients were allocated `[1,F,H]`/`[1,H,F]` for
rank-two parameters (the same defect the native path had), and the zero-token
early return handed back seven flat `(0,)` buffers.  The allocations are rank two
now, and a zero-token backward returns properly shaped results per `need_*`: empty
row-shaped gradients, and exact zeros of the weight's own shape for every
requested weight gradient - an expert-parallel rank that holds no tokens must not
leave the optimizer with an absent gradient.  The native path is fixed the same
way, and `test_empty_batch` now asserts, for both providers, that every weight
gradient is present, correctly shaped and all-zero.

The Triton leaf loops step the reduction in whole 32-wide leaves without a tail
mask (`_rows_gemm`, `_dweights_kernel`); the masked `_wgrad_kernel` is the one
place whose reduction length is arbitrary.  Through the public entry the reduction
width is always 64-aligned, so this was latent, but silently dropping a tail leaf
would be a wrong answer rather than a slow one, so the provider now fails closed
on an unaligned width instead of truncating.

CI: the CPU-safe step listed the CUDA and ROCm provider modules, which are
hardware-marked and skip wholesale on `ubuntu-latest`, so the job reported a green
run that executed nothing.  Those two modules are dropped from that step (the
CPU-only reference suites stay); the provider modules run on the GPU jobs and in
this PR's verification record.

Verified on the same H100 80GB HBM3 (CUDA 13.0, PyTorch 2.12.0+cu130): 180 passed,
54 skipped across the operator, reference and neighbour suites; `black --check`,
`mypy` and `mkdocs --strict` clean.

Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
@luoyueyuguang

Copy link
Copy Markdown
Contributor Author

Second round addressed in 3d306b0d. Four of the five were real; the two tail-truncation findings were latent rather than reachable, and I fixed them by failing closed instead of masking the hot loop.

Triton shared-expert weight gradients, [1,F,H] for a rank-2 parameter (inline, bug/high): confirmed, fixed. Same defect the native path had in the first round, in the Triton backward's allocation. Now torch.empty((shared_width, width)) / (width, shared_width); _wgrad_kernel writes [M,N] from the group offset, so the storage is byte-identical.

Triton zero-token backward returned seven flat (0,) buffers (inline, bug/high): confirmed, fixed. Reachable whenever an expert-parallel rank holds no tokens. The early return is now shaped per need_*: empty row-shaped gradients for packed/x/weights, and exact zeros of the weight's own shape for every requested weight gradient — an absent gradient is not the same as a zero one for an optimizer step or an all-reduce. The native path had the same gap (it returned undefined tensors), so both are fixed, and test_empty_batch now asserts for both providers that every weight gradient is present, correctly shaped and all-zero. (Row-shaped gradients legitimately collapse to "absent" through the wrapper's _optional, since they have zero elements; the test allows that and still rejects a flat (0,).)

K // BK / width // BK leaf loops without a tail mask (summary, bug/medium ×2): latent, fixed by failing closed. Through the public entry the reduction width is always 64-aligned (the operator validates that), and the one place where the reduction length is arbitrary — the weight-gradient kernel — already uses tl.cdiv with masks. So neither loop was reachable with a truncated tail. I did not add a per-element mask to the hot loop: the provider now checks the leaf alignment on entry and raises, which matches the "fail closed on unsupported strict geometry" rule and keeps the mask off the inner loop.

CI false green (summary, test/medium): confirmed, fixed. The CPU-safe step listed tests/backends/cuda/test_nemotron_experts_cuda.py and tests/backends/rocm/test_nemotron_experts_rocm.py, both hardware-marked, so on ubuntu-latest the job skipped everything and still reported green. Those two modules are dropped from that step; the CPU-only reference suites remain, and a comment records why. The provider suites run on the GPU jobs and in this PR's verification record.

The four token_budget_reached warnings again mean those file groups were not fully read; I re-read them (bindings, _C.pyi, registry wiring, rl_engine/ops/...) and found nothing further.

Re-verified on the same idle H100 80GB HBM3 (CUDA 13.0, PyTorch 2.12.0+cu130, KERNEL_ALIGN_FORCE_SM90=1 KERNEL_ALIGN_DET_GEMM_SM90=1): 180 passed, 54 skipped across the operator, reference, det_gemm, router, wrapper, registry and ROCm suites; black --check --line-length 100, mypy --ignore-missing-imports and mkdocs build --strict clean.

@Flink-ddd Flink-ddd added platform: cuda Specific optimizations or bugs in NVIDIA graphics cards (such as FlashInfer, TMA optimizations) new-model Nemotron labels Nov 5, 2026

@Flink-ddd Flink-ddd left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix the outstanding per-device shared-memory setup in launch_grouped_v2: the process-wide static bool attributed skips configuration on the second GPU and can cause launch failures.

- `launch_grouped_v2` cached the `cudaFuncSetAttribute` call behind a
  function-local `static bool`, but the attribute is scoped to the
  (kernel, device) pair: a process that ran the operator on a second GPU
  never configured it there and the first v2 launch failed with
  `invalid argument`. Issue the request on every launch, which is the
  pattern the v1 path and `det_gemm` already use, and check the result
  instead of dropping it. The v1 path now checks its call too.

- `_C.pyi`: `nemotron_experts_backward` returns undefined entries as
  `None` for gradients that were not requested; annotate
  `list[torch.Tensor | None]`.

- The provider test derived the token index with a literal `// 6`; use
  the case's own slot count so a case with another top-k cannot compare
  the wrong row.

Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
- Registry: the native provider now checks the device compute capability
  at construction (major >= 9) instead of only the build flag. On a
  pre-Hopper GPU the SM90 kernel image does not exist, so the provider
  declined only at the first launch; it now declines up front and
  `get_op` falls through to the strict Triton provider. New regression
  test pins the fallback, and the operator page documents it.

- C++ forward: `weights` must be rank two `[tokens, 6]`. The old check
  accepted a flat `[rows]` tensor while backward returns `[tokens, 6]`,
  which autograd rejects at accumulation time.

- Benchmark `provider_op` no longer treats every name but `native` as
  Triton: `--providers` is restricted to the supported names and an
  unknown name raises instead of silently attributing Triton results to
  a mistyped provider.

- Benchmark robustness: a provider whose module import fails is reported
  unavailable instead of aborting the run; `--dtype` is threaded into
  `build` and the eager/grouped_mm casts follow the payload dtype rather
  than hard-coding BF16, so the recorded dtype cannot diverge from what
  ran; the FP64 qualification skips with a stated reason when its
  estimated footprint (token-independent FP64 weights plus gradients)
  exceeds free device memory.

- Benchmark graph mode: peak memory is measured around capture, not
  around an eager call before it, so the graph's private pool is
  counted; the subtraction isolates each provider's increment.

- Benchmark output records the precision mismatch: the eager baseline
  runs FP32 while providers run BF16, so `speedup_*_vs_baseline` is
  qualified at both the result and row level.

- CI: the CPU-safe Nemotron step no longer lists the CUDA-marked router
  suite, which skipped wholesale on `ubuntu-latest` and contradicted the
  step's comment.

Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
- The SM90 gate now follows `packed.device` on every call instead of
  `torch.cuda.get_device_capability()` once at construction. The old
  check judged the process-wide current device, while the registry caches
  one instance and picks it for every CUDA device: a Hopper current device
  could select the native kernel for an Ampere payload, and an Ampere
  current device disabled the provider for a Hopper payload. A clear
  fail-closed error now names the payload device; the regression test
  patches the capability and calls the provider with real tensors.

- Gradient wrappers no longer coerce zero-element tensors to `None`.
  The zero-token backward deliberately returns row-shaped empty gradients
  and exact-zero weight gradients; `_optional` collapsed the former, so
  the documented empty-payload contract did not hold. Both providers now
  return the extension/kernel result unchanged (undefined results already
  arrive as `None`), and `test_empty_batch` requires the shaped empty row
  gradient instead of accepting absence. The native zero-token branch also
  gates `d_weights` on `need_weights` like the other entries.

- `tests/backends/nemotron_experts_cases.py`: the isolated-token dPacked
  comparison indexes the one-token payload by sorted-expert position, but
  used `slot_order[slot]` (original slot) instead of the inverse map. The
  router sorts ids by expert id, so `argsort(ids)` is the identity and the
  old expression was correct only for that no-op case; the inverse keeps
  the comparison valid for any slot order.

- `tests/reference/test_nemotron_experts_reference.py`: the `tokens == 0`
  guard ran after a `torch.stack` over an empty list, which raises first,
  so the zero-token reference path was unreachable. Build `ids`
  conditionally.

- Operator page: the selection text no longer claims a device-based
  fallthrough; the fail-closed gate states that the check follows the
  payload device.

Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
- Native backward makes `grad_out` contiguous before the extension call.
  Autograd may hand over a strided gradient (narrowed/transposed output)
  while the kernel requires a contiguous `[tokens, width]` buffer, and the
  Triton provider already normalized it.

- The device gate admits exactly SM90 (`major == 9`), not `>= 9`. The build
  emits `sm_90a` code and architecture-specific images are not forward
  compatible, so Blackwell must fail closed instead of selecting a kernel
  with no matching image. The regression test now covers an sm_80 and an
  sm_100 capability.

- Backward's shared-expert `grad_out` goes through `aligned16` before it
  becomes a TMA base for the shared dX/dW projections, matching every other
  TMA operand (an upstream sliced view can be contiguous but not 16-byte
  aligned).

- `run_grouped` bounds the reduction depth the kernel actually uses: a
  weight gradient's segment leaves, and `K / K_TREE_LEAF` for the other
  modes. The old guard checked the row count for every mode, so a wide
  hand-built call with a tiny batch passed and could overrun `tree_stk`.

Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
- The two-path test only asserted the env-derived preference, so a
  dispatch bug that ignored the selector (or resolved it once at
  construction) could turn the bitwise comparison into a kernel compared
  with itself. Expose `nemotron_experts_last_path()`, written by
  `run_grouped` at the branch it actually takes (including the per-shape
  v1 fallback), and assert it after each run.

- `test_provider_selection_prefers_native`: assert the patched
  `native._C` reports the compile flag as False before the second
  lookup, which pins that the provider reads the object the test rebinds.

- Operator page: document both accessors and what the parity test asserts.

Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
…ce cast note

- `inverse` is now zero-initialized and the scatter ignores an
  out-of-range route id. The table is filled from `permutation`, whose
  values are trusted by design; a duplicate left slots uninitialized and
  an out-of-range value wrote past the buffer, both of which the combine
  and dW leaf kernels then dereference. A contract-violating table now
  yields a wrong result instead of an out-of-bounds access, with no host
  sync, so graph capture stays legal. Operator page updated.

- The reference module's cast note claimed the operator's cast table
  leaves the down projection in FP32. That describes the reference's own
  `torch.matmul` baseline, not the kernel contract: the `det_gemm`
  arithmetic rounds every 32-wide leaf to BF16 in the down projection and
  the backward GEMMs as well, and `test_matches_det_gemm_bitwise` pins the
  down projection against `_C.det_gemm_fwd_rhs_transposed`. The docstring
  now states both levels explicitly, so the Triton provider's per-leaf
  rounding is not read as a contract violation.

Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
The inverse buffer protects the combine's `down` lookups; the other
stages still trust `permutation`/`offsets` values, so the operator page
must not read as if a violated table were memory-safe everywhere.

Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Nemotron new-model platform: cuda Specific optimizations or bugs in NVIDIA graphics cards (such as FlashInfer, TMA optimizations)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants