Repository navigation
[Nemotron][CUDA] Add deterministic moe_experts_combine - #661
luoyueyuguang wants to merge 11 commits into
Conversation
…Triton) The `moe_experts_combine` row of RFC RL-Align#434: grouped up projection with a relu-square epilogue, grouped down projection, the fixed-order weighted combine over the six route slots, the optional shared expert and the operator's single exit cast, forward and backward, at the Nemotron-3-Nano-30B-A3B shapes (2688/1856, 128 routed experts + 1 shared, top-6). The arithmetic is the det_gemm contract, copied deliberately: a 32-wide FP32 leaf rounded once to BF16, leaves merged with the canonical midpoint tree, fixed ascending K order, one CTA per output tile, no split-K, no atomics. What differs is the mapping. Each CTA resolves its own (expert, row tile) from the 129 prefix sums in shared memory, so the host never reads `offsets` and CUDA Graph capture stays legal; the stacked weights are one tensor map over `[E*F,H]` and `[E*H,F]`; one grid serves forward, backward and both weight gradients. Two providers satisfy the contract: `csrc/cuda/moe/nemotron_experts.cu` for NVIDIA SM90 (mma.sync + TMA) and the shared Triton kernel for CUDA and ROCm. The registry lists native first and Triton second on CUDA, only Triton on ROCm, and fails closed on cpu/npu/musa/mps. `tests/backends/cuda/` pins the up and down projections, `dPacked` and both weight gradients bitwise against `_C.det_gemm_*`; the reference suite is an independent FP64 oracle. Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
A second instruction path for the same contract: TMA with a 128-byte swizzle, `wgmma.mma_async` reading both operands from shared-memory descriptors, one producer warp plus a four-warp consumer warpgroup, three stages, 64x128 tiles and two static accumulators so one leaf is rounded and merged while the next is still on the tensor pipe. It is self-contained in `csrc/cuda/moe/nemotron_experts.cu` - the descriptor encoding, the wgmma wrappers, the swizzled tensor map and the extra barrier primitives are that file's own - and the mma.sync path stays as the shape fallback. `RL_KERNEL_NEMOTRON_EXPERTS_BACKEND=sm90_v1|sm90_v2|auto` selects between them and is read per launch, so one process can measure both. Grouped additions over a plain GEMM: either operand may be K-major or MN-major, which fixes the descriptor start offsets and the instruction's imm-trans flag; a weight gradient's last reduction tile is zero-padded in shared memory, because a segment length is not a multiple of the 32-wide leaf. The relu-square operand is now materialized by the up epilogue as `Z = bf16(relu(U)**2)` instead of being recomputed on every down and dW_down operand tile. Both paths are bitwise identical on every output and gradient (`test_instruction_paths_agree_bitwise`, 17/129/64 tokens, the small ones with per-expert segment lengths that are not leaf-aligned) and still match `_C.det_gemm_*` bitwise. The benchmark gains `--baselines grouped_mm|eager` and `--phases`, so the production grouped-GEMM path (`torch._grouped_mm`, cuBLAS / CUTLASS) is measured next to the eager formulation in the same paired run. Measured on one H100 NVL at T=4096, forward+backward: 47.6 -> 29.5 ms (41 -> 66.5 TFLOP/s), per stage 1.31-1.83x. Benchmark medians over T=1..32768 give 1.4-2.6x for forward and 1.2-1.7x for forward+backward, with the eager-baseline gap at 2.3-37x; the T=1 forward regression of the mma.sync path (0.782 ms against a 0.477 ms eager baseline) is gone at 0.358 ms. The production grouped GEMM is 1.7-2.8x faster in the forward and that is the contract's price, not the schedule's: the kernel sits at the SIMT instruction-issue ceiling - 30.7e9 element-leaves in 29.5 ms needs 0.98e12 warp-instructions/s against 0.94e12 issuable - so anything further is a leaf-width arithmetic decision. Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
|
🔍 OpenCodeReview found 3 issue(s) in this PR.
📄
|
|
@luoyueyuguang The automated OCR review has completed. Please review and address all automated review bot findings, then push your updates. Maintainers will begin manual review after the bot findings have been addressed. |
OCR review of RL-Align#661 raised two inline defects and five test/reference issues. `tree_stk` bounds: the midpoint merge keeps one slot per level, so `TREE_DEPTH = 12` only covered `kd <= 4096`, while a weight gradient's reduction depth is the expert segment length and one segment may hold up to 6 * MAX_TOKENS = 393216 packed rows (`kd = 12288`). Widen the stack to 14 and fail closed in `run_grouped` above `1 << TREE_DEPTH` leaves. The guard bounds the row count, not `offsets`, so the host still never reads the routing table and graph capture stays legal. Shared-expert weight gradients were allocated `[1,F,H]`/`[1,H,F]` for rank-two `w_sh1`/`w_sh2`. Autograd accepted them (it passed the suite) and callers got a mis-shaped `.grad`; the allocations are rank two now, and `cases.run` asserts every gradient's shape against its input so the class stays covered. Reference dtype guard: it demanded one dtype for payload, route weights and stacked weights, which excluded the operator's own pairing (BF16 payload with FP32 route weights). It now accepts that pairing and rejects the rest, and the cast-point test covers three pairings instead of two, with a companion test pinning that BF16 and FP32 payloads share one rounding band while an FP64 payload lands at 1e-13 - the bound now discriminates the cast points rather than the payload dtype. Test defects: the empty-batch case kept the 17-token `offsets` (now zeroed) and went through the registry instead of the parametrized provider (now the provider object, so the Triton column is really exercised); `test_reject_bad_geometry` was parametrized over providers although validation is hardware-independent (now a single test); the instruction-path parity test reused one provider instance across both env scopes, so it could degenerate into a self-comparison - it now resolves a provider per scope and asserts the new `nemotron_experts_preferred_path()` reports the path it thinks it is testing. Re-validated on an H100 80GB HBM3 (CUDA 13.0, PyTorch 2.12.0+cu130): 180 passed, 54 skipped across the operator, reference and neighbour suites; `black --check`, `mypy` and `mkdocs --strict` clean; the instruction-path ratio is unchanged (T=4096 forward+backward 27.5 ms against 43.5 ms, 1.58x), so the wider merge stack costs nothing measurable. Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
|
All seven findings addressed in Inline 1 — Inline 2 — shared-expert weight gradients returned rank three (bug/high): confirmed, fixed. Reference dtype guard (medium): confirmed, fixed. Empty-batch case kept a stale routing table (medium): fixed. Parity test could degenerate into a self-comparison (medium): fixed. Each env scope resolves its own provider, and the test asserts a new Unused FP32 cast-point bound (low): investigated, partially pushed back. The premise that an FP32 payload means "exit cast only" does not hold for this reference: The five Re-validated on an idle H100 80GB HBM3 (CUDA 13.0, PyTorch 2.12.0+cu130,
|
…iring) The re-review of RL-Align#661 found the same two defects in the shared Triton provider plus two unreachable-looking tail bugs and a CI false green. Triton shared-expert weight gradients were allocated `[1,F,H]`/`[1,H,F]` for rank-two parameters (the same defect the native path had), and the zero-token early return handed back seven flat `(0,)` buffers. The allocations are rank two now, and a zero-token backward returns properly shaped results per `need_*`: empty row-shaped gradients, and exact zeros of the weight's own shape for every requested weight gradient - an expert-parallel rank that holds no tokens must not leave the optimizer with an absent gradient. The native path is fixed the same way, and `test_empty_batch` now asserts, for both providers, that every weight gradient is present, correctly shaped and all-zero. The Triton leaf loops step the reduction in whole 32-wide leaves without a tail mask (`_rows_gemm`, `_dweights_kernel`); the masked `_wgrad_kernel` is the one place whose reduction length is arbitrary. Through the public entry the reduction width is always 64-aligned, so this was latent, but silently dropping a tail leaf would be a wrong answer rather than a slow one, so the provider now fails closed on an unaligned width instead of truncating. CI: the CPU-safe step listed the CUDA and ROCm provider modules, which are hardware-marked and skip wholesale on `ubuntu-latest`, so the job reported a green run that executed nothing. Those two modules are dropped from that step (the CPU-only reference suites stay); the provider modules run on the GPU jobs and in this PR's verification record. Verified on the same H100 80GB HBM3 (CUDA 13.0, PyTorch 2.12.0+cu130): 180 passed, 54 skipped across the operator, reference and neighbour suites; `black --check`, `mypy` and `mkdocs --strict` clean. Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
|
Second round addressed in Triton shared-expert weight gradients, Triton zero-token backward returned seven flat
CI false green (summary, test/medium): confirmed, fixed. The CPU-safe step listed The four Re-verified on the same idle H100 80GB HBM3 (CUDA 13.0, PyTorch 2.12.0+cu130, |
Flink-ddd
left a comment
There was a problem hiding this comment.
Please fix the outstanding per-device shared-memory setup in launch_grouped_v2: the process-wide static bool attributed skips configuration on the second GPU and can cause launch failures.
- `launch_grouped_v2` cached the `cudaFuncSetAttribute` call behind a function-local `static bool`, but the attribute is scoped to the (kernel, device) pair: a process that ran the operator on a second GPU never configured it there and the first v2 launch failed with `invalid argument`. Issue the request on every launch, which is the pattern the v1 path and `det_gemm` already use, and check the result instead of dropping it. The v1 path now checks its call too. - `_C.pyi`: `nemotron_experts_backward` returns undefined entries as `None` for gradients that were not requested; annotate `list[torch.Tensor | None]`. - The provider test derived the token index with a literal `// 6`; use the case's own slot count so a case with another top-k cannot compare the wrong row. Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
- Registry: the native provider now checks the device compute capability at construction (major >= 9) instead of only the build flag. On a pre-Hopper GPU the SM90 kernel image does not exist, so the provider declined only at the first launch; it now declines up front and `get_op` falls through to the strict Triton provider. New regression test pins the fallback, and the operator page documents it. - C++ forward: `weights` must be rank two `[tokens, 6]`. The old check accepted a flat `[rows]` tensor while backward returns `[tokens, 6]`, which autograd rejects at accumulation time. - Benchmark `provider_op` no longer treats every name but `native` as Triton: `--providers` is restricted to the supported names and an unknown name raises instead of silently attributing Triton results to a mistyped provider. - Benchmark robustness: a provider whose module import fails is reported unavailable instead of aborting the run; `--dtype` is threaded into `build` and the eager/grouped_mm casts follow the payload dtype rather than hard-coding BF16, so the recorded dtype cannot diverge from what ran; the FP64 qualification skips with a stated reason when its estimated footprint (token-independent FP64 weights plus gradients) exceeds free device memory. - Benchmark graph mode: peak memory is measured around capture, not around an eager call before it, so the graph's private pool is counted; the subtraction isolates each provider's increment. - Benchmark output records the precision mismatch: the eager baseline runs FP32 while providers run BF16, so `speedup_*_vs_baseline` is qualified at both the result and row level. - CI: the CPU-safe Nemotron step no longer lists the CUDA-marked router suite, which skipped wholesale on `ubuntu-latest` and contradicted the step's comment. Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
- The SM90 gate now follows `packed.device` on every call instead of `torch.cuda.get_device_capability()` once at construction. The old check judged the process-wide current device, while the registry caches one instance and picks it for every CUDA device: a Hopper current device could select the native kernel for an Ampere payload, and an Ampere current device disabled the provider for a Hopper payload. A clear fail-closed error now names the payload device; the regression test patches the capability and calls the provider with real tensors. - Gradient wrappers no longer coerce zero-element tensors to `None`. The zero-token backward deliberately returns row-shaped empty gradients and exact-zero weight gradients; `_optional` collapsed the former, so the documented empty-payload contract did not hold. Both providers now return the extension/kernel result unchanged (undefined results already arrive as `None`), and `test_empty_batch` requires the shaped empty row gradient instead of accepting absence. The native zero-token branch also gates `d_weights` on `need_weights` like the other entries. - `tests/backends/nemotron_experts_cases.py`: the isolated-token dPacked comparison indexes the one-token payload by sorted-expert position, but used `slot_order[slot]` (original slot) instead of the inverse map. The router sorts ids by expert id, so `argsort(ids)` is the identity and the old expression was correct only for that no-op case; the inverse keeps the comparison valid for any slot order. - `tests/reference/test_nemotron_experts_reference.py`: the `tokens == 0` guard ran after a `torch.stack` over an empty list, which raises first, so the zero-token reference path was unreachable. Build `ids` conditionally. - Operator page: the selection text no longer claims a device-based fallthrough; the fail-closed gate states that the check follows the payload device. Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
- Native backward makes `grad_out` contiguous before the extension call. Autograd may hand over a strided gradient (narrowed/transposed output) while the kernel requires a contiguous `[tokens, width]` buffer, and the Triton provider already normalized it. - The device gate admits exactly SM90 (`major == 9`), not `>= 9`. The build emits `sm_90a` code and architecture-specific images are not forward compatible, so Blackwell must fail closed instead of selecting a kernel with no matching image. The regression test now covers an sm_80 and an sm_100 capability. - Backward's shared-expert `grad_out` goes through `aligned16` before it becomes a TMA base for the shared dX/dW projections, matching every other TMA operand (an upstream sliced view can be contiguous but not 16-byte aligned). - `run_grouped` bounds the reduction depth the kernel actually uses: a weight gradient's segment leaves, and `K / K_TREE_LEAF` for the other modes. The old guard checked the row count for every mode, so a wide hand-built call with a tiny batch passed and could overrun `tree_stk`. Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
- The two-path test only asserted the env-derived preference, so a dispatch bug that ignored the selector (or resolved it once at construction) could turn the bitwise comparison into a kernel compared with itself. Expose `nemotron_experts_last_path()`, written by `run_grouped` at the branch it actually takes (including the per-shape v1 fallback), and assert it after each run. - `test_provider_selection_prefers_native`: assert the patched `native._C` reports the compile flag as False before the second lookup, which pins that the provider reads the object the test rebinds. - Operator page: document both accessors and what the parity test asserts. Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
…ce cast note - `inverse` is now zero-initialized and the scatter ignores an out-of-range route id. The table is filled from `permutation`, whose values are trusted by design; a duplicate left slots uninitialized and an out-of-range value wrote past the buffer, both of which the combine and dW leaf kernels then dereference. A contract-violating table now yields a wrong result instead of an out-of-bounds access, with no host sync, so graph capture stays legal. Operator page updated. - The reference module's cast note claimed the operator's cast table leaves the down projection in FP32. That describes the reference's own `torch.matmul` baseline, not the kernel contract: the `det_gemm` arithmetic rounds every 32-wide leaf to BF16 in the down projection and the backward GEMMs as well, and `test_matches_det_gemm_bitwise` pins the down projection against `_C.det_gemm_fwd_rhs_transposed`. The docstring now states both levels explicitly, so the Triton provider's per-leaf rounding is not read as a contract violation. Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
The inverse buffer protects the combine's `down` lookups; the other stages still trust `permutation`/`offsets` values, so the operator page must not read as if a violated table were memory-safe everywhere. Signed-off-by: luoyueyuguang <luoyuelight@outlook.com>
Implements the SM90 CUDA
moe_experts_combinework item of #434: grouped up projection with a relu-square epilogue, grouped down projection, the fixed-order weighted combine over the six route slots, the optional shared expert and the operator's single exit cast — forward and backward — at the Nemotron-3-Nano-30B-A3B geometry (hidden 2688, intermediate 1856, 128 routed experts + 1 shared expert of 3712, top-6, BF16 payload/weights, FP32 router weights and combine). Target branchtest-nemotron.This PR fills the CUDA column only (§7 "one row × one platform = one PR"). The ROCm column is a separate PR owned separately: the shared Triton provider is included here because the CUDA column needs it as the fallback, and it is the entry point the ROCm column will share, but nothing in this PR claims gfx9xx qualification. Per §7 both columns implement one versioned contract (
nemotron-experts-sm90-v1) and share one test suite (tests/backends/nemotron_experts_cases.py, imported by both the CUDA and the ROCm provider suites).What changed, and why each layer owns its part
csrc/cuda/moe/nemotron_experts.cudY, the permutation inverse, and the operator's single exit castrl_engine/backends/cuda/moe/nemotron_experts.pyrl_engine/ops/moe/nemotron_experts.pyMAX_TOKENSbound, one provider-route report linerl_engine/backends/shared/triton/moe/nemotron_experts.pyrl_engine/reference/moe/nemotron_experts.pyrl_engine/runtime/registry.pymoe_experts_combinepriority: native → Triton on CUDA, Triton only on ROCm, fail closed on cpu/npu/musa/mpscsrc/bindings/ops.cpp,rl_engine/_C.pyi,build_tools/extensions.py.cuis not hipifiedtests/backends/cuda/,tests/backends/rocm/,tests/backends/nemotron_experts_cases.py,tests/reference/benchmarks/operators/moe/benchmark_nemotron_experts_training.pydocs/operators/nemotron-experts.md,docs/.nav.yml,docs/operators/README.md,.github/workflows/ci.ymlThe arithmetic is the
det_gemmcontract, copied deliberately: a 32-wide FP32 leaf rounded once to BF16, leaves merged with the canonical midpoint tree, fixed ascending K order, one CTA per output tile, no split-K, no atomics, no TF32. What differs is the grouping: each CTA resolves its own(expert, row tile)from the 129 prefix sums in shared memory, so the host never readsoffsetsand CUDA Graph capture stays legal; the stacked weights are one tensor map over[E·F, H]/[E·H, F]; combine slots are summed in ascending order into an FP32 accumulator.Two instruction paths satisfy that one contract and are bitwise identical:
sm90_v1sm90_v2(default)BM=128, BN=64, BK=32BM=64, BN=128, KT=64(two leaves)mma.sync.m16n8k16per leafwgmma.mma_async.m64n128k16per leafldmatrixSWIZZLE_128B, wgmma reads descriptors__syncthreadsin the K loopHMMA 32,LDSM 12,REG 66-82,STACK 1552HGMMA 6,LDSM 0,REG 135(170for dW),STACK 1568RL_KERNEL_NEMOTRON_EXPERTS_BACKEND=sm90_v1|sm90_v2|autoselects the path and is read per launch, so both can be measured in one process. The wgmma path is self-contained innemotron_experts.cu(descriptor encoding, wgmma wrappers, swizzled tensor map, extra barrier primitives); the mma.sync path stays as the shape fallback. The relu-square operand is materialized by the up epilogue asZ = bf16(relu(U)²)— a BF16 product is exact in FP32, so this is bit-identical to transforming every operand tile, and it removes that transform from the down anddW_downstages.Out of scope: ROCm hardware execution (the shared Triton provider runs there but was not re-qualified on gfx9xx), TP / CP / EP (separate rows of #434), prefill/decode cache state (this operator is stateless), and the router (separate row, already merged).
Environment covered
One NVIDIA H100 NVL (SM90, 95 GiB), driver 580.173.02, CUDA 13.0, PyTorch 2.11.0+cu130, CUDA-13.0 nvcc, host GCC from the CUDA 13.0 image. Build flags:
KERNEL_ALIGN_FORCE_SM90=1 KERNEL_ALIGN_DET_GEMM_SM90=1. Single GPU, no distributed topology. BF16 in/out with FP32 accumulate; FP32 router weights. Not exercised: ROCm, Ascend, MUSA, other NVIDIA SKUs, CUDA 12.x, multi-GPU.Numerical-contract evidence
out,dPacked,dX,dWeightsanddWaretorch.equalacross batch sizes (1 … 32768 tokens), token positions, packed layouts (random and concentrated routing), repeated runs and CUDA Graph replay. The operator is stateless, so the RFC's sequence-split clause reduces to this layout invariance: no cache, no cross-CTA merge.test_instruction_paths_agree_bitwiseruns both native paths in one process and compares every output and gradient withtorch.equal, on the core geometry at 17 and 129 tokens (per-expert segment lengths that are not multiples of the 32-wide leaf, i.e. the weight-gradient tail padding) and on the Nemotron geometry at 64 tokens (truncated 128-wide column tiles,N=1856). 3 cases, all pass.det_gemm. The up and down projections,dPackedand both weight gradients are pinned bitwise against_C.det_gemm_fwd_rhs_transposed/det_gemm_fwd/det_gemm_db_transposedon real shapes; 4 cases.dPackeddXdWeightsThe floor is set by the contract itself (leaf rounding at BF16 eps 7.8e-3 plus six signed terms before the single exit cast), so these numbers are the pipeline's own quantization, not a kernel defect. The
det_gemmparity rows above are the exactness evidence; these L2 figures are the accuracy evidence against mathematics.5. Not claimed. Bitwise equality between providers (native and Triton associate leaf sums differently; measured 8.3e-3 core / 1.3e-2 Nemotron relative L2, near one BF16 ulp at the output).
dWbitwise reproducibility for the same ordered token set across microbatch splits — adding independently computed microbatchdWis not bitwise full-batch equivalence, and neither is claimed for any provider. Cross-compiler equivalence. TP/CP/EP.Tests and benchmarks actually run
tests/backends/cuda/test_nemotron_experts_cuda.pydet_gemmbitwise cases, 3 path-parity cases)det_gemm's "Triton is unavailable" and ROCmgfx942rows in this build, 5 = the ROCm provider suite without a HIP device)black --check/mypy/mkdocs --strictmkdocsexits 0torch.profiler+cuobjdump -res-usage/-sassNot exercised: ROCm hardware, Ascend/MUSA (fail closed), TP/CP/EP, multi-GPU, CUDA 12.x, non-Hopper NVIDIA parts.
Performance
Workload: the real Nemotron geometry above, BF16, CUDA Graph replay, five untimed warmup executions per candidate before capture, 20 alternating paired samples of 5 executions, median reported, cotangents prepared outside timing, all columns measured in one sequential session on an idle device at its boost clock (1785/1785 MHz). Reported as measured differences, no automatic regression threshold (§7 leaves the target ratio open; see "Where the remaining gap is").
Candidates: the eager formulation (
torch.matmulexpert by expert in FP32 + slot-ascending combine),grouped_mm=torch._grouped_mm, the cuBLAS/CUTLASS grouped GEMM a serving stack calls (forward only; the stacked weights are pre-transposed outside the timed region), the two native instruction paths, and the shared Triton provider.Forward, random routing, median ms:
Forward + backward, random routing, median ms:
Concentrated routing (v2 / v1 / Triton forward+backward): 3.564 / 4.314 / 1.775 ms at T=1, 28.405 / 44.880 / 22.144 ms at 4096, 218.603 / 356.957 / 183.355 ms at 32768 — the wgmma path is 1.2-1.6x over mma.sync and 2.3-4.7x over the baseline there too.
Peak memory (increment over the weight stack, random routing, MiB): forward+backward native 2496 at T=1, 2713 at 1024, 4349 at 8192; Triton 2496 / 2684 / 4117; eager baseline 3878 / 8775 / 10613. Forward alone: native 1094 MiB at 8192 against 1010 for
grouped_mmand 913 for the baseline. Both native paths allocate identically because both materializeZ(182 MiB of the 8192 row).The production path is faster, and here is why.
grouped_mmis 1.7-2.8x faster than this operator in the forward (1.6-2.9x under concentrated routing) and the gap widens with size. It is not addressed by any schedule change: the grouped GEMM holds an FP32 accumulator and casts once per output element, while the contract rounds and tree-merges every 32-wide leaf, i.e.K/32extra SIMT passes per element. At T=4096 the forward+backward performs 30.7e9 element-leaves in 29.5 ms, which needs 0.98e12 warp-instructions/s against the 0.94e12 this part can issue — the operator is at the contract's SIMT instruction-issue ceiling. Closing the 2.7x means a wider leaf or a single cast, i.e. an arithmetic decision for the contract owner, not a kernel decision.What is still not measured: a production training baseline (Transformer Engine
GroupedLinearperforms grouped dgrad + wgrad; it is pinned at 2.19.0 in this repository and neither installs nor builds against this container's CUDA 13.0 / torch 2.11.0+cu130, and vLLM's fused MoE is not installed).torch._grouped_mmgroups over token rows only, sodW_up[e] = dU_eᵀ·packed_e— whose reduction length differs per expert — is not expressible with it, which is why the forward+backward tables still compare against the eager formulation.Documentation and compatibility
docs/operators/nemotron-experts.md(contract, interface, fixed arithmetic, ordering policy, instruction paths, qualification boundary, tests, performance reproduction, verification record), added todocs/.nav.ymlanddocs/operators/README.md.rl_engine.ops.moe.nemotron_experts.nemotron_experts(...); provider ids stayrlkernel.nemotron_experts.sm90.v1/.triton.v1(the contract version does not change with the instruction path, which is bit-neutral). New optional environment variableRL_KERNEL_NEMOTRON_EXPERTS_BACKEND(defaults to the wgmma path).moe_experts_combineresolves native → Triton on CUDA and Triton only on ROCm; unsupported platforms raise..github/workflows/ci.ymlnow runs the CPU-safe reference/contract tests for this operator (the CUDA/ROCm provider suites skip without a device).§7 contribution-note compliance
moe_experts_combine, with its validation in the same PR. The row is not marked done: the ROCm column is a separate PR.nemotron-experts-sm90-v1; one shared suitetests/backends/nemotron_experts_cases.pyimported bytests/backends/cuda/test_nemotron_experts_cuda.pyandtests/backends/rocm/test_nemotron_experts_rocm.py; only the kernel and the native baseline differ per platform.mamba2_ssd_mixerbullet)T=1/16/128) and training (T=1024…32768) shapes, forward and forward+backward, against three references: the eager formulation,torch._grouped_mm(the cuBLAS/CUTLASS grouped GEMM production path, forward only) and the shared Triton provider, with warmup, 20 paired samples, medians and peak memory. The production path is 1.7-2.8x faster in the forward and that gap is the contract's leaf rounding, quantified at the SIMT issue ceiling. No production training baseline exists in this container (TEGroupedLinear/ vLLM absent;_grouped_mmcannot express grouped wgrad), which is declared rather than substituted.torch.equalbetween two strict paths,allclosevs an independent FP64 reference, never against shared code_C.det_gemm_*(3 + 4 cases,torch.equal). Correctness:rl_engine/reference/moe/nemotron_experts.py, a hand-written FP64 oracle that shares no code with any kernel (CPU-only suite + FP64 L2 report). The cross-provider number is reported as a comparison, never as the correctness oracle.dWvisits grouped rows ascending and empty experts write exact zeros, and the router'spermutation/offsets(a bijection and its exclusive prefix sums) are the ordering input. No claim rests on "deterministic on my GPU": invariance is asserted withtorch.equalacross batch, position, packing, repeat and graph replay. Policy is marked proposed for §4.4 review, consistent with #504.tokens > 65536,offsets.numel() != experts+1, unpaired shared weights, non-contiguous or wrong-device inputs.permutation/offsetsvalues are trusted as the router's contract rather than validated on device (a device-side check would force a host sync and break graph capture); that boundary is documented in the operator page and stated here rather than hidden. The registry may fall through to the Triton provider when the native build is absent — that is a switch between two strict, deterministic implementations of one contract, not a fallback from strict to non-strict arithmetic.Pending
GroupedLinear(pinned 2.19.0) or vLLM fused MoE at the same shapes. Needs a host where that stack builds against CUDA 13.0 / torch 2.11; the forward comparison already usestorch._grouped_mm.