Repository navigation
[DSv4][CUDA] DSv4 testing control threads. - #635
Draft
KJLdefeated wants to merge 161 commits into
Draft
KJLdefeated wants to merge 161 commits into
KJLdefeated wants to merge 161 commits into
Conversation
…/isort config in pyproject Signed-off-by: KJLdefeated <linkai0508@gmail.com>
Freezes the contract, the reference answers and the checker for the P1 work package so all six sub-tasks (P1-D1..P1-D6) can start in parallel against the same frozen math and the same golden bytes. No GPU kernels. - rl_engine/mhc/reduction.py: the two pinned reduction trees (serial ascending left fold for long reductions, (a0+a1)+(a2+a3) for the four streams). Every P1 accumulation goes through here. - rl_engine/mhc/contract.py: LayerContract, ResidualBatch, controller and norm params, GradBoundary, checkpoint fingerprints. - rl_engine/mhc/oracle.py: FP32 reference for hc_split_sinkhorn, fp32_gemm_rms, mhc_post, mhc_pre/h_aggregate, rmsnorm_residual and the fixed-K GEMM, forward and backward, plus the block composition. - rl_engine/mhc/provider.py: MHCProvider protocol, oracle-backed reference, fail-closed stub, capability gate. - rl_engine/mhc/fixtures.py + tests/fixtures/p1/golden_hashes.json: seeded cases and the CI anchor for the golden bytes. - scripts/check_p1.py: the acceptance command. - 53 CPU tests; every hand-written backward is cross-checked against an autograd graph built from the same frozen formulas. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kcu8vXve5YMZ5wqCCKRfvJ
…lars Follow-up after reading the Megatron mHC source (megatron/core/transformer/hyper_connection.py and megatron/core/fusions/fused_mhc_kernels.py, dev). Megatron's native and fused paths disagree with each other: the fused kernel uses TF32 MMA, split_k=16 at K>=16384, and runtime autotuning, and computes sqrt(s/K) where native computes sqrt(s)/sqrt(K). Three of issue #2's explicit bans (Split-K off, TF32 off, fixed reduction tree) are violated by that path at exactly the production shape. So Megatron supplies the formulas and constants for this contract, never the reduction order. Changes: - fusion: 'unfused' is now CANONICAL and documented as such. Train/infer byte-equality needs every boundary hashable on its own, which only the unfused decomposition gives. v1 does not reuse TE's TEFusedResidualRMSNorm -- it refuses to expose the intermediate (raises on any forward hook), collapsing two boundaries into one. Fusion stays legal; changing the reduction layout or a downcast point does not. - alpha: Megatron holds three learnable scalars (alpha_pre/post/res) broadcast over the PRE/POST/COMB segments, not 24 independent gains. Forward was already equivalent; backward was not, so dAlpha is now three scalars with the segment fold pinned like every other reduction. - design doc: new section on what Megatron actually does and why it is not the byte reference; D1/D6/D8 rewritten with the source evidence; open questions updated (Miles must also run unfused -- needs their sign-off). - the fused/unfused test now says what it actually proves (oracle self-consistency, not that a real fused kernel matches). 54 tests pass; check_p1.py PASSes on cpu and cuda, stub still fails closed; ruff/flake8/black/isort/mypy clean. Golden manifest regenerated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kcu8vXve5YMZ5wqCCKRfvJ
Issue #2 splits P1 by function, not by operator: P1-D1 owns the mHC forwards, P1-D2 the RMSNorm forwards, P1-D3 every custom backward across both, P1-D4 the provider adapter, P1-D5 trace/tests/benchmarks and P1-D6 the fixed-K GEMM reference -- eight tasks with P1-S0 and P1-R0. The kit had invented a one-task-per-operator split instead. That was wrong in the design doc and, worse, in StubProvider's messages, which told a developer to "claim it on P1-D1 (hc_split_sinkhorn)" and would have pointed them at the wrong task. - StubProvider now names the real owning task per (operator, direction); every *_bwd points at P1-D3. - design doc: the eight tasks with owners, the dependency line, and a note that operators and tasks are different axes. - provider docstring and tests updated to match. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kcu8vXve5YMZ5wqCCKRfvJ
#2 has eight GitHub sub-issues, and they are split one per operator -- not by function as the older 任务分工 section in the issue body reads. The previous commit followed that superseded section and mislabelled everything; this restores the operator split and uses the authoritative issue numbers, matching how P5 labels its sub-tasks. - P1-1 (#14) hc_split_sinkhorn, P1-2 (#15) fp32_gemm_rms, P1-3 (#16) mhc_post, P1-4 (#17) mhc_pre/h_aggregate, P1-5 (#18) rmsnorm_residual -- each fwd + bwd together; P1-6..P1-8 (#19-#21) are the WS2 TP/SP/CP/ DP/PP tasks. - #15 explicitly absorbs the fixed-K / batch-invariant GEMM reference and its equivalence harness, so fixed_k_gemm points there rather than at a task of its own. - StubProvider names the owning sub-issue per operator. - D8 reframed against #18's actual wording: it says prefer TE's TEFusedResidualRMSNorm and self-write only if TE fails the deterministic contract. TE fails it -- it refuses to expose the pre-normalization intermediate -- so unfused v1 is the escape hatch #18 provides, not a departure from it. - design doc: the eight sub-issues with stage, issue number and the WS2 attachment points (placement + check_capability). 54 tests pass; check_p1.py reference PASS, stub fails closed; lint clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kcu8vXve5YMZ5wqCCKRfvJ
Implements shared_expert_mlp_fwd/bwd per the P5-S0 contract: every valid token runs fc1 -> one-round SwiGLU -> fc2 on BF16 frozen weights, backward returns dX only (FP32 accumulator), and the shared output stays independent of the routed path. Both backends reproduce the FP32 oracle's numeric profile oracle-fp32-serial-v1 byte-for-byte on the same device: one lane owns one output element and reduces serially in ascending k, multiply and add rounded separately (__fmul_rn/__fadd_rn on CUDA, uncontracted IEEE fp32 in Triton), sigmoid computed as 1/(1+expf(-x)) to match torch.sigmoid on FP32 CUDA tensors. No cross-lane floating-point reduction exists anywhere, so results are batch/padding invariant by construction (fwd(x)[t] == fwd(x[t:t+1]) byte-equal). The one-round SwiGLU core runs in shared mode (p_s=None, no clamp, per S0 decision D6) and is the reuse point for P5-2 (#63). Providers subclass ReferenceProvider and override only the two shared-expert methods, so the full acceptance command runs unchanged; unsupported input (non-CUDA device, missing extension or triton, foreign numeric profile) raises instead of falling back (fail-closed). Provenance records split_k=1 / serial-ascending-k / no-FMA per the P5-5 provenance requirement. Acceptance: python scripts/check_p5.py --provider rl_engine.moe.backends.shared_expert:CudaSharedExpertProvider --device cuda python scripts/check_p5.py --provider rl_engine.moe.backends.shared_expert:TritonSharedExpertProvider --device cuda pytest tests/test_shared_expert_mlp.py python benchmarks/benchmark_shared_expert_mlp.py Signed-off-by: Yizheng Jiao <jyizheng@gmail.com>
tl.exp is the fast exp2-based path and does not bit-match torch.sigmoid; libdevice __nv_expf does (0/4M mismatches on the device probe). Signed-off-by: Yizheng Jiao <jyizheng@gmail.com>
Neither tl.exp (exp2-based) nor libdevice __nv_expf bit-matches the nvcc expf inside torch.sigmoid (~45% / ~10% of fp32 values differ by 1 ulp); the tiny fixtures passed only because the BF16 round absorbed the difference, and the T=256 benchmark cross-check caught the divergence. The Triton path now takes torch.sigmoid(gate) as a kernel input and fuses the remaining SwiGLU math; a (256, 1024, 512) cross-backend byte-equality test locks the regression in. Signed-off-by: Yizheng Jiao <jyizheng@gmail.com>
… _rn ops The compiler may contract a * b + c into an FMA; at T=256 that rounded dsilu = sig * (1 + g * (1 - sig)) differently on 2/262144 dgate elements (1 ulp after the BF16 round). All mul/add/sub in the Triton strict GEMM and SwiGLU kernels now go through libdevice add_rn/mul_rn/sub_rn, the exact Triton spelling of the CUDA kernel's __fadd_rn/__fmul_rn/__fsub_rn. Signed-off-by: Yizheng Jiao <jyizheng@gmail.com>
Signed-off-by: Hsiu-I Liao <hsiu-iliao@Hsiu-Is-MacBook-Pro.local>
Signed-off-by: Hsiu-I Liao <hsiu-iliao@Hsiu-Is-MacBook-Pro.local>
Signed-off-by: Hsiu-I Liao <hsiu-iliao@Hsiu-Is-MacBook-Pro.local>
Signed-off-by: Hsiu-I Liao <hsiu-iliao@Hsiu-Is-MacBook-Pro.local>
Signed-off-by: Hsiu-I Liao <hsiu-iliao@Hsiu-Is-MacBook-Pro.local>
…ct path Adds two explicitly-selected performance providers that keep the P5-5 round positions (fc1 output and dX stay FP32; y/dh round once to BF16) and swap only the GEMM reduction order, per review feedback on #387: - shared-expert-cuda-det (p5-det-gemm-v1): reuses det_gemm_kernel.cu (fixed K-order, no split-K; TMA+mma.sync on SM90+, scalar K-tree fallback elsewhere) through two new FP32-output wrappers that expose the existing gemm_dispatch output_fp32 path. - shared-expert-triton-det (p5-triton-dot-v1): tl.dot with fixed 64x64x32 tiles, ascending-k, no autotune and no split-K. Both are deterministic and batch-invariant but not byte-equal to oracle-fp32-serial-v1; selecting them by provider name is the explicit opt-in, and the strict providers remain the oracle-parity gate. New tests pin repeat-run byte-equality, per-row batch invariance, and closeness to the oracle; the benchmark now reports all four backends. Signed-off-by: Yizheng Jiao <jyizheng@gmail.com>
det_gemm merges K with a BF16 mid-split tree (its TP-equivalence design), so its deviation from the FP32-serial oracle is BF16-tree-sized; the tl.dot path keeps FP32 accumulators and only carries reduction-order noise. Each provider now declares its own closeness tolerance instead of sharing one number. Signed-off-by: Yizheng Jiao <jyizheng@gmail.com>
Signed-off-by: Bignonia7 <Bignonia7@users.noreply.github.com>
[P5-2] Add deterministic clamp_swiglu_weighted CUDA kernel
[DSv4][P5-5] Shared Expert MLP: strict CUDA + Triton kernels
Implements the LoRA delta on the routed expert path (#62): U = X @ A^T FP32 accumulate, round to BF16 dY = (U @ B^T) * a FP32 accumulate, FP32 output Backward follows the same graph and returns (dX, dA, dB); there is no dW, since the base weights are frozen. dY*alpha and dU round to BF16 between the GEMMs (contract D4) and the association order is frozen as written. Three providers, each overriding only this operator so the other eight stay on the oracle and the acceptance command runs end to end: LoRADeltaProvider torch-native benchmark baseline; not invariant LoRADeltaCudaProvider composed from _C.det_gemm_* LoRADeltaTritonProvider composed from TritonDetGemmOp Neither deterministic path adds a GEMM -- both compose the repo's existing fixed-K primitives, per the guidance that GEMMs route through the shared batch-invariant originals. The two differ in how close they stay to the oracle, and neither declares ORACLE_PROFILE. The Triton path holds one FP32 accumulator across the whole K loop and rounds once at the store, which is what _serial_dot does -- but the oracle adds one product at a time while this path adds one BLOCK_K=32 tile at a time, and the two parenthesisations agree only while nothing rounds at all. BF16 carries 8 mantissa bits, so a BF16xBF16 product needs at most 16 and lands in FP32's 24 exactly; while the partial sums stay inside that headroom the accumulation order cannot change a single bit. As K grows the accumulator climbs into the remaining bits and starts dropping low ones, so the agreement degrades rather than breaking at a threshold -- over 12 seeds at M=24 N=2048, byte-equal holds 12/12 up to K=512, then 11/12 at 1024, 9/12 at 2048 and 7/12 at 4096. The P5 fixture geometry is K=128 for fc1 and K=64 for fc2, comfortably inside the headroom, which is why check_p5.py passes. The gap at production width is FP32-epsilon sized, ~1e-6 relative on a handful of elements. The CUDA path reuses det_gemm's mid-split tree, which rounds to BF16 at every 32-wide leaf and so sits ~5e-3 from the oracle at all K. It buys something the Triton path lacks: a contiguous half-K shard is one child of the tree, so TP=2 reproduces TP=1 bitwise, which is likely what P5-8 needs. Batch invariance is a separate property -- it compares a backend against itself -- and holds for both at every probed shape, production width included. Batch invariance did not require avoiding tensor cores, which contradicts a reasonable first guess. tl.dot is deterministic once the tile shape stops depending on M; what matters is pinned BLOCK sizes (autotune would pick per-shape configs), no split-K, and allow_tf32=False. provenance() reports the reduction strategy per P5-6's fail-closed contract. Signed-off-by: Loring Ni <93922846+ScottNi0809@users.noreply.github.com>
Follows benchmark_det_gemm.py: reports overhead against a fair baseline (cuBLAS with TF32 disabled) rather than a speedup, since both deterministic paths disable TF32 by construction. Forward on RTX A2000 (sm_86), median of 5x100 iterations: shape M native fp32 Triton CUDA Triton ovh CUDA ovh decode_m1 1 0.200 0.248 0.205 1.2x 1.0x small_m8 8 0.176 0.233 0.201 1.3x 1.1x batch_m64 64 0.193 0.344 0.352 1.8x 1.8x batch_m256 256 0.214 0.240 0.718 1.1x 3.4x The Triton path stays within 1.1-1.8x of cuBLAS while being byte-equal to the oracle. The CUDA path degrades with M because det_gemm falls back to its naive scalar kernel below SM90; an SM90 runner would take the TMA path instead. Profiling explains the absolute numbers: at M=64 the two BF16 GEMMs cost 0.070 ms but the whole native forward costs 0.222 ms, so 68% of the wall clock is the elementwise .float() / *alpha / BF16-cast traffic rather than the matmuls. Traffic analysis agrees -- at X=[M,4096], N=2048, r=8 the arithmetic intensity is about r = 8 FLOP/byte and the split is ~66% reading X, ~33% writing Y, and only 0.26% for the U round trip. So fusing away the intermediate U is not where the headroom is; folding the scale and the cast into the GEMM2 epilogue is, and that is #58 (Step 10), deliberately out of scope here. It also means small-M rows are launch-bound, which is why the TF32 column can land below the FP32 one -- the table should be read as relative overhead, not GEMM throughput. Signed-off-by: Loring Ni <93922846+ScottNi0809@users.noreply.github.com>
tests/test_p5_lora_delta.py -- 43 cases parametrised over the CUDA and Triton providers, each validated independently. The torch-native provider is excluded on purpose: it is the benchmark baseline and fails batch-invariance by design. Skips cleanly on CPU-only runners. Covers the five acceptance criteria from the P5-3 brief: golden vector base-only / LoRA-only / base+LoRA, via the oracle tamper fixture flipping a packed base byte must reach the output no dW frozen weights untouched, backward returns exactly 3 grads finite/nonzero the adapters actually receive signal fc1 and fc2 both insertion points, whose shapes differ plus return order and dtypes, chunked-prefill and padding invariance, run-to-run stability, and provenance naming the backend that really ran. Two tests bracket the Triton path's agreement with the oracle instead of claiming it outright: one pins the byte-equality at fixture width, the other pins the bounded divergence at production width and fails if that divergence ever disappears -- which would mean the docstring and the profile have gone stale. The second sweeps seeds rather than pinning one, since whether a given input diverges depends both on where its values fall relative to a rounding boundary and on which instruction tl.dot lowers to (MMA on Ampere, WGMMA on Hopper). scripts/check_p5_lora_delta_invariance.py -- 100 probes per backend (5 seeds x 5 sub-batch sizes x 4 shapes), reporting per-shape and per-boundary counts so a real pass can be told apart from a lucky one: backend invariant u y dX rerun torch-native 47/100 100 87 47 ok cuda det_gemm 100/100 100 100 100 ok triton det_gemm 100/100 100 100 100 ok Three findings shaped both files. Mutating four real bugs (swapped return order, dropped alpha, transposed dX, a GEMM replaced by torch.matmul) showed the invariance checks were running at the fixture's 128x64 geometry, where cuBLAS keeps one strategy across all M and a non-deterministic implementation passes unnoticed; production geometry (4096x2048) is now probed too. GEMM1 cannot carry an invariance check at all -- u is [M, 8], too narrow for cuBLAS to switch strategy. And dA/dB reduce over every row, so a sub-batch result is a different quantity rather than a slice; they are covered by the rerun check. The tamper test carries a control assertion that an untampered rerun is bitwise identical, so the not-equal check cannot pass on incidental noise. One flipped byte moves 494/3072 output elements by up to 2.2e-2. Signed-off-by: Loring Ni <93922846+ScottNi0809@users.noreply.github.com>
Signed-off-by: Bignonia7 <326766516+Bignonia7@users.noreply.github.com>
…5 oracle
P5-1: MXFP8 activation quantization (OCP E4M3 elements, block-32 E8M0
shared scales) plus the straight-through backward, in both CUDA and Triton.
Both backends reproduce rl_engine.moe.mx_format.mx_quantize byte for byte, so
they keep the oracle numeric profile (oracle-fp32-serial-v1) instead of
registering a relaxed one:
the block amax is an integer max over |x| bit patterns, floor(log2) is read
off the FP32 exponent field after the FLT_MIN clamp (exact integer
equivalent of the oracle's frexp path), the scale is assembled from its bit
pattern so code 0 lands on the subnormal 2**-127, and the division is an
explicit non-.ftz div.rn.f32 / tl.fdiv(ieee_rounding=True). Nothing on the
subnormal-sensitive path can be flushed by --use_fast_math; a fast-math build
is a regression test. E4M3 encoding is clamp-then-RNE (start-kit decision
D1), which is what cvt.rn.satfinite.e4m3 does in hardware.
Determinism: the amax reduction is row-local over a fixed 32-element window and
every element is then transformed independently, so the bytes do not depend on
grid shape, block size, or how many rows are in flight.
Non-finite input is fail-closed in both backends and in the providers (P5-1 spec s4);
the STE backward accepts any floating dtype (P5-1 spec s2).
Providers live in rl_engine/moe/backends/mxfp8_act_quant.py, following the
backends package P5-5 introduced. scripts/check_p5.py --device cuda passes
byte-equal on every boundary for both {Cuda,Triton}MXFP8ActQuantProvider,
with act_quant1.*/act_quant2.* hashed and act_quant_bwd noted as "ste".
Two outside MX implementations are byte-identical to these kernels and are
asserted as tests when installed: torchao's to_mx (FLOOR) and the Triton repo's
triton_kernels.downcast_to_mxfp (ROUND_DOWN), the quantizer vLLM runs for its
MXFP4 path. The benchmark measures against both, against vLLM's production
per_token_group_quant_fp8 CUDA kernel (a different format: group 128, FP32
scale), and against a plain fp8 cast as the memory-bandwidth lower bound. On
H100 the kernels reach 2.99 TB/s at 16384x7168, at that bound.
The CUDA wrapper prefers the compiled rl_engine._C symbols and otherwise
JIT-builds the single .cu file, so the tests and the benchmark run in a source
tree without a full extension build (RL_KERNEL_P5_DISABLE_JIT=1 to require AOT).
Signed-off-by: Ryan Huang <ryankert01@gmail.com>
…nd past 2**31 elements The benchmark shapes (4096x7168 and 16384x7168, bf16 and fp32, random exponents over 2**+-60) were only timed before; they are now asserted byte-equal against the oracle, at ~0.2 s each. A 2**31-element bf16 input (262144 x 8192) checks that the Triton kernel's int64 element offsets do not wrap: the oracle cannot run at that size, so rows at the start, middle and far end of the tensor are compared with their standalone quantization, which the row-local contract requires to be identical. Skipped when the device has less than ~7 GiB free. Signed-off-by: Ryan Huang <ryankert01@gmail.com>
…indings - Providers resolve their backend once in __init__ and raise NotImplementedError when it is unavailable, matching the other P5 providers and what scripts/check_p5.py renders as a FAIL row; the per-call import and the _Ops bag are gone, and the recorded linkage is the one that resolved. - The CUDA wrapper resolves rl_engine._C or the JIT module once per process and refuses the JIT fallback under RL_KERNEL_REQUIRE_EXT=1 (the repo's existing "extension must be present" switch, set by the GPU CI) instead of a new flag. - Input gates and the fail-closed read-back live in rl_engine/kernels/ops/moe_common.py, shared by both backends; rl_engine/moe/mx_format.py is back to defining only the format. - The non-finite check is derived from the block amax bit pattern (inf/NaN >= 0x7F800000) instead of a second full reduction, in both kernels; bytes are unchanged. - csrc/ops.cpp binds the _cuda entry points directly, as the sibling P5 ops do; the six load_vec specializations collapse to one template with a 16-bit converter; rl_engine.moe no longer re-exports the providers (the backends package is the registry). - Tests: the --use_fast_math build uses torch's build cache instead of a fresh tmp_path per session (~50 s saved per run); one _wide_range generator. - Benchmark: dead _speedup helper removed; one byte-equality check. Signed-off-by: Ryan Huang <ryankert01@gmail.com>
[DSv4][P5-4] MXFP8×MXFP4 Routed Expert grouped GEMM: wgmma CUDA kernel
…rapper Review on #419: `envs.py` is a build-time helper next to setup.py and is not part of the installed rl_engine package, so importing it unconditionally broke a non-editable install. Import it optionally and fall back to reading RL_KERNEL_REQUIRE_EXT directly (the spelling the CI scripts export). Signed-off-by: Ryan Huang <ryankert01@gmail.com>
m_grouped_bf16_gemm_nt_contiguous gives the bytes of one bf16_gemm_nt per expert in both the m-indices layout and the psum layout (each expert starts at the previous end aligned up to 128), over 5 repeats and uneven / decode / large row distributions; a row alone in its expert gives the same bytes; k_grouped_bf16_gemm_tn_contiguous wgrad is within 8e-7 of FP32. H200, 32 experts, H 4096, rank 64: 0.06-0.77 ms against 0.22-0.92 ms for the per-expert loop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
rl_engine/models/deepseek_v4/layerwise_gate.py runs the same sequences as a continuous-batching rollout (staggered arrivals, chunked prefill under a token budget, decode next to prefill chunks), as the actor recompute (one packed prefill, reversed request order) and through forward_train, then compares every layer's streams per token and reports the first layer that differs. forward_train gains return_streams. scripts/check_dsv4_layerwise_gate.py runs it on a random checkpoint or the first --layers layers of a real one. H200, real dimensions, nonzero LoRA, window / C4 / C128 with hash and scored routing, 2108 tokens: every layer, the final hidden state and the logprobs are byte-identical for 4 layers (with training) and 8 layers. Negative control: a one-element change in layer 2 is reported at layer 2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
Per-expert MoE adapters are deferred until verl needs them; when built they use DeepGEMM's grouped BF16 GEMM (probed byte-equal to per-expert GEMMs) under a new profile with a BF16-rounded delta. Chunked training across sequence chunks is not planned; packed multi-sequence batches already work. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
… removal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
The Megatron cross-config training provider (adapters, readback, gradient ranks), tests/p7_doubles and the test_megatron_training_* suite are not on the DSv4 path and reference APIs that main no longer has (mypy). The DSv4 logprob head replaces P7; verl integration will be redone on its own. Drop the CI step and the conftest sys.path hook that only these helpers needed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
rl_engine/p2, rl_engine/p6, their tests, CI workflows, smoke script, design notes and validation reports came from the start-kit phase; the DSv4 attention and MoE combine replace them and nothing imports them. The start-kit design notes for P1, P5 and P7 go too. Duplicate validation reports (the same probe on the old toolchain) are dropped; the torch 2.13 / CUDA 13 reports stay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
… use The P1 (unfused mHC, fp32_gemm_rms, rmsnorm_residual, h_aggregate) and P5 (providers, LoRA delta, clamp SwiGLU, Triton lanes, BF16 shared expert) kernels stay in the tree but are not on the DSv4 path; their tests, check scripts and benchmarks go, as does the P6 moe_merge test harness. Tests of modules DSv4 does use move to tests/ops/moe (mx_format, MXFP8 act quant, MoE contract, oracle), and the golden-hash fixtures move from tests/fixtures/p1, p5 to mhc, moe. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
- DSv4: Any for the MoE block's object-typed fields; overloads for silu_and_mul_clamp_quant_cuda's return_h; narrow Optional cache / weight fields with asserts; type the layerwise gate's locals; rename a reused variable in random_checkpoint. - Import _C from rl_engine.backends.extension (where main moved it) instead of the kernels.ops.base shim mypy cannot see through. - mxfp8_act_quant parses RL_KERNEL_REQUIRE_EXT like build_tools.envs.env_flag instead of importing the repo-root envs shim. - Legacy P1 / P5 code: TritonDetGemmOp from its current module, Any for the extension handle, and targeted type: ignore where the old code cannot be typed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
docs/architecture/dsv4-design.md replaces the plan / todo log (dsv4-cuda-design.md), the batch-invariant profile and the attention-backward proposal, and folds in the performance table. It covers the goal and guarantees, environment, build, the regression / benchmark / probe commands, the module map (code, kernels, operator page, tests), decisions, pinning rules and the reduction order of every module, numeric profiles, training, validation including the layerwise gate, what the rollout must patch, legacy code that DSv4 does not use, open work and numerics lessons. - Every DSv4 operator page links to it and drops the P1-P7 / T0x labels. - The MoE page describes the DeepGEMM shared expert DSv4 runs (it still described the legacy strict kernels) and lists the legacy backends instead of their ROCm results. - Legacy operator pages (fp32_gemm_rms, h_aggregate, rmsnorm_residual, moe_merge) say they are not used by DSv4. - p7-t02-m10-sampling-contract.md becomes dsv4-sampling-contract.md. - CLAUDE.md points to the design doc and drops the account-specific environment notes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
… count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
Every CTA of the routed-expert forward and backward GEMMs scanned all E expert offsets to find its token block, and the grid was sized from ceil(M / BM) + E. At small M most CTAs only scanned and exited: E = 256, T = 1 launched about 8k CTAs for 6 rows and took 1.9 ms on H200. A one-block kernel now builds tile -> (expert, first row) and the tile count once per GEMM (moe_tile_map.cuh); a CTA decodes its block with two loads. The grid bound is ceil(M / BM) + min(E, M). Scheduling only: tile bytes do not depend on which CTA computes them, so outputs are unchanged. H200, routed experts (BF16 in -> combined out), CUDA-graph replay: E = 256, T = 1: 1.88 -> 0.131 ms E = 256, T = 64: 3.47 -> 2.27 ms E = 256, T = 16384: 17.2 -> 12.9 ms E = 32, T = 1..256 unchanged within 10% DSv4 regression: 1217 passed, 72 skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
check_dsv4_moe_ep.py --lora ran on 2 H200s. Forward is byte equal to EP = 1 for scored and hash layers in every split. dX differed (relative L2 up to 8.3e-5), deterministic run to run. Cause: lora_backward runs FP32 torch.mm, and cuBLAS picks its kernel from M, so a row's result depends on the row count of the call (rows of [M, 1024] @ [1024, 16] at M = 1, 2, 12 differ from M = 97). On one GPU the routed fc2 adapter already shows it (block dX alone vs in batch, 5e-5); EP changes each rank's row count, so it shows up against EP = 1. The design already does not require the LoRA dX path to be deterministic. The check now requires, with --lora: forward byte equal, dX run-to-run equal and within 1e-3 relative of EP = 1, adapter gradients within 1e-3 (seen 2.2e-4). Without LoRA, dX must still be byte equal (passes). The design doc records the result, the cause and the lesson. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
… 8 GPUs scripts/verl_e2e/ lets an engineer or a coding agent on an 8 x H100 / H200 machine check whether verl + vLLM batch-invariant mode (verl #7688, image iseekyan/verl:ds4_vllm_align.preview-amd64 by digest, pinned verl and Megatron-LM commits) gives bitwise train/rollout log-probs for DeepSeek-V4-Flash, and what it costs: - setup.sh / launch_container.sh: preflight, pull, pinned clones, model and DAPO data, a long-lived container. - patch_verl_bitwise_metric.py: verl's rollout_probs_diff compares exp(log_prob) and reads 0.0 for log-probs 1 ulp apart (checked on the real metrics.py); the patch adds the count of response tokens whose FP32 bits differ. - run_e2e.sh: T1 (the recipe's 4-GPU quick test), T2 (8 GPUs, EP8, 4 layers), T3 (more layers, longer responses); summarize_metrics.py gives the per-step verdict and timing. - bench_rollout.sh / probe_server_bi.py: vLLM DP8+EP8 throughput with BI on, off and native experts, plus a black-box alone-vs-batch probe. Checked here: shell syntax, the metric patch on verl's file (1-ulp case caught), the summarizer on a synthetic log. Nothing has run on 8 GPUs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
T4 trains 200 steps on the first 8 layers of DeepSeek-V4-Flash (32 prompts x 8 samples, one update per step, LR 1e-5) and checks the FP32 log-prob bits every step. A truncated model scores 0 on the math reward, so GRPO advantages and gradients would vanish and the weights would never move; custom_reward.py (fraction of digits) is a dense synthetic reward that keeps real updates flowing to the rollout. The summarizer counts steps with a nonzero grad norm and shortens long per-step tables. The README now targets 8 x H100 (memory and OOM fallbacks, T3 as the memory probe for T4, tmux, no checkpoints at ~700 GB each). Checked here: shell syntax, the reward function, the summarizer on a synthetic 200-step log. Nothing has run on 8 GPUs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
The guide now runs setup -> T1 (environment check) -> rollout speed (BI on / off, full model) -> T4 (50 steps, 8 layers, early check at step 5, OOM fallbacks inline), about 6-10 h of machine time instead of ~1.5 days. T2 / T3 stay in run_e2e.sh as optional variants. The baseline recipe trains all parameters on Hopper, so T4 keeps the truncated model; the full-model LoRA run belongs to the RL-Kernel stack once it is in verl. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
The guide has two training tests now: T1 (the recipe's quick check) and T2 (8 GPUs, 8 layers, 50 steps). run_e2e.sh drops the old T2 / T3 variants so the names match the README. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
rl_engine/integrations/dsv4/vllm_model.py registers RlkDeepseekV4ForCausalLM under vLLM's DeepseekV4ForCausalLM name (RL_KERNEL_DSV4_ROLLOUT=1 general plugin; no vLLM source changes). It keeps vLLM's scheduler, paged KV cache, FlashMLA layer, indexer and FP8 projections, and runs RL-Kernel's kernels where the vLLM fork rounds differently: - mHC pre / post / head (four identical first-layer streams), whole MoE block (Dsv4MoE, loaded from the checkpoint; vLLM allocates no experts), plain RoPE on window layers - q / kv RMSNorm, q-norm / RoPE / KV insert into vLLM's fp8_ds_mla cache, indexer q RoPE + FP8 quant, output projection (BI mode on Hopper swaps fp8_einsum for Triton) - FlashMLA: RL-Kernel's pinned build serves vLLM's interface (same commit, other build flags; both register torch.ops._flashmla_C, so ours loads first) - sparse decode kernel (VLLM_DS4_DECODE_KERNEL=sparse), one stream in attention - FP32 logits head; sampled-token log-prob from the 64-tile LSE (V1 and V2 runners) - unmerged LoRA on wq_a / wkv / wq_b / wo_b and in Dsv4MoE - VLLM_BATCH_INVARIANT starts on the pinned DeepGEMM (vLLM's set_batch_invariant check guards MoE kernels this model does not run) scripts/check_dsv4_vllm_gate.py: vLLM with chunked prefill (decode next to prefill chunks) against one packed Dsv4Model prefill, every layer, sub-layer and sampled-token log-prob in bytes. Toy checkpoint (4 layers: window, window, C4, C128): PASS without LoRA (2102 tokens, 54 log-probs at T = 0.8). With LoRA the C4 layer still differs. Also: build_lora_adapters (attach_lora's adapters, reusable by the rollout), FP32 in LoraAdapter.init (vLLM loads under a BF16 default dtype), load_output_projection, and scripts/probe_dsv4_vllm_attention_ops.py (q-norm / RoPE / KV insert vs vLLM's kernel). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
… with LoRA vLLM's compress -> RMSNorm -> RoPE -> FP8 quant -> store step (cutedsl for the main C4 / C128 KV, Triton for the indexer K) rounds a rare compressed entry differently; with LoRA the toy data hit one in the C4 layer (positions 35 and 303), and every later query attends to it. The plugin routes that step to compress_main_cuda / compress_indexer_cuda (same state-cache and block-table addressing); vLLM's save_partial_states stays. scripts/check_dsv4_vllm_gate.py, toy checkpoint (window, window, C4, C128), chunked prefill, T = 0.8: PASS without LoRA and with rank-16 adapters on wq_a / wkv / wq_b / wo_b and the MoE (2102 tokens at every layer and sub-layer, 54 sampled-token log-probs). The report also lists the (request, position) of every differing attention output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
…tatus Section 10 now records the rollout plugin: each vLLM fork kernel the gate caught and what replaces it (the earlier "already equal to vLLM" list was wrong for the q / kv norm, the q-norm / RoPE / KV insert, the compressor store and the FlashMLA build). Section 12: the plugin is done on one GPU with LoRA; the actor is a verl engine backend on Dsv4Model (2026-11-15 decision). Section 13: two lessons from the gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
persist_w8 (default on) expands every expert's folded FP8 weights once, on first use, and caches the scratch with the prepared weight. Same bytes as the per-call expansion; costs E x N x K bytes of E4M3 per weight. Job 181224 (H200): 1237 passed / 52 skipped, layerwise gate PASS (2108 tokens), EP2 checks pass. E=256 MoE vs vLLM BI: T=4096 1.70x -> 0.88x, T=16384 1.62x -> 1.35x; E=32 T=1024 2.39x -> 1.32x. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
A verl BaseEngine around Dsv4Model, registered in EngineRegistry on import (model.external_lib) and selected through hydra config groups shipped in rl_engine (model_engine=rlkernel, hydra.searchpath=pkg://...verl_config). verl's source is unchanged. - forward_backward_batch on verl's micro-batching: one packed full-prefill forward per micro-batch, jagged next-token log-probs (dsv4-logprob-v1), per-sample temperatures grouped per row; entropy as an FP32 metric only. - LoRA only: FP32 master adapters with verl's build_optimizer and LR schedules, FP32 grad accumulation, one flat DP all-reduce, clip, step. - get_per_tensor_param exports the BF16 adapters with a PEFT-style config; disable_adapter gives the base model; checkpoint save / load. - LoRA rank lives in the engine config: verl's model.lora_rank would turn on vLLM's own LoRA in the rollout. scripts/check_dsv4_verl_engine.py (torchrun, 2 x H200, toy checkpoint): log-probs equal byte for byte across micro-batch layouts, against each sequence alone and with per-sample temperatures, before and after training; adapters identical on both ranks; disable_adapter equals the base model; export and checkpoint round trip. VERL ENGINE PASS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
…to the actor The rlkernel engine (actor) and the vLLM rollout plugin now run together under verl with no change to verl or vLLM: - vllm_worker.py: RlkWorkerExtension installs the adapters the engine exports (BF16 lora_A / lora_B) on the plugin's layers instead of vLLM's LoRA manager; RlkWorker keeps level-1 sleep, since the base weights load once from the checkpoint and are never re-sent. Selected through rollout.engine_kwargs.vllm.worker_cls / worker_extension_cls. RL_KERNEL_DSV4_SKIP_ADAPTER_SYNC=1 is a negative control. - vllm_model.py: vLLM's spawned processes unpickle the DeepSeek-V4 quant config, which imports vLLM's DeepSeek-V4 package and its FlashMLA build before any plugin runs; register a subclass defined in this module so the plugin (and RL-Kernel's FlashMLA build) loads first. Refuse load_format=dummy. - verl_engine.py: accept verl's trainer-side rollout quantizer (offered for quantized checkpoints) and leave it unused; only adapters are synced. - scripts/verl_e2e/run_dsv4_rlkernel_grpo.sh: the launcher. Toy GRPO on 2 x H200 (toy checkpoint, rank-16 LoRA, 8 x 4 samples x 128 tokens, synthetic reward): training/rollout_logprobs_mismatch_count 0 at all 4 steps. Negative control: 0 at step 1, 4096 / 4096 from step 2. The vLLM gate still passes with LoRA (2102 tokens). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
How the rlkernel engine, the vLLM plugin and the worker extension plug into verl and vLLM, the toy GRPO result and its negative control, and the open items: the 8-GPU guide against verl #7483, EP > 1 in the engine, memory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
Batch, lengths, token budget, rollout memory, LR, logger and experiment name are environment variables; the defaults remain the 2-GPU toy run. Checked on the toy checkpoint (1 x H200, file logger): rollout_logprobs_mismatch_count 0 at all 4 steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
scripts/verl_e2e_lora/: setup of both stacks on one verl commit (#7483's head), the real checkpoint's first LAYERS layers, RL-Kernel's gate on that checkpoint, the RL-Kernel run, the negative control, #7483's own launcher on the same workload, and compare.py (bitwise verdict per step, timing breakdown, side by side). The RL-Kernel env pins the tested 2-GPU env's packages and the local DeepGEMM fix. Not run here (2 GPUs per job); the BI-off ablation needs a plugin switch first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
CLAUDE.md now carries the owner's decisions, the non-obvious verl integration facts, status and next steps, and this cluster's environment (limits, envs, job scripts). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
Formatting only. The gate's plugin-before-vLLM import order is kept with isort: off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
RL_KERNEL_DSV4_NON_BI=1 with VLLM_BATCH_INVARIANT=0 (the launcher's BI=0): the rollout plugin keeps vLLM's own attention paths (non-invariant planning, paged FP8 decode, fused q / kv norm, q norm / RoPE / KV insert, indexer query, compressor store, aux streams) and RL-Kernel's mHC, MoE, output projection, LoRA, RoPE tables and FP32 head, so the model stays the one the actor trains. Mixing the two env settings is refused. check_dsv4_vllm_gate.py --non-bi reports the difference. Toy checkpoint, 1 x H200: - BI gate with LoRA still byte-identical. - --non-bi with LoRA: 302 of 2102 tokens differ after layer 0 (decode tokens and some prefill chunks), the rest are byte-identical; differences then grow through the random-weight layers. - GRPO BI=0: every response token mismatches (as intended), timing_s/gen 0.79x and timing_s/step 0.85x of the BI run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
run.sh ablation (BI=0) and a report-only --non-bi check in run.sh gate; section 10 describes the switch and the toy numbers; section 12 and CLAUDE.md status. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019tkttC4AW5Y1FLX1EvSsYj Signed-off-by: KJLdefeated <linkai0508@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a complete DeepSeek-V4 forward and LoRA-training path on Hopper (H100 / H200) whose forward is bitwise batch-invariant. Rollout and actor recompute can therefore produce identical logprobs on the same kernel set.
Start here:
docs/architecture/dsv4-design.md. It is the single design doc: guarantees, environment and build, how to run tests / benchmarks / probes, the module map, decisions, the reduction order of every module, performance, what the vLLM rollout must patch, legacy code, and open work. Each operator has a detail page underdocs/operators/linked from it.Size and how to review
About 44.6k added lines over 256 files, but most of it is tests (7.4k), goldens (2.0k), benchmarks (1.9k), probe / check scripts (2.7k) and docs (4.0k). Reviewers only need the modules below; the arithmetic of each kernel is specified on its operator page and pinned by a byte-equality test against its golden, so the goldens, benchmarks and scripts do not need line-by-line review.
Suggested order:
docs/architecture/dsv4-design.md, sections 1, 3, 4, 5.rl_engine/models/deepseek_v4/model.py: layer / model forward,forward_train, finite-check modes.rl_engine/models/deepseek_v4/attention.py:Dsv4Batch(host-side sizes, no readback), compressor / indexer / FlashMLA wiring, canonical index order.rl_engine/models/deepseek_v4/moe.py: route plan, dispatch, EP all-to-all, combine.rl_engine/kernels/ops/cuda/deepgemm.py,flashmla.py,dsa_backward.py;pyproject.tomlextradsv4.rl_engine/kernels/ops/cuda/fp8_linear.py(all dense FP8 projections),dsv4_attention.py(FlashMLA wrapper, output projection).csrc/cuda/mhc_dsv4/dsv4_mhc.cu(dsv4-mhc)csrc/cuda/moe/sm90_fused_moe_mlp.cu,sm90_moe_mlp_backward.cu(dsv4-moe); largest kernel, FP8 WGMMA + TMA.csrc/cuda/moe/moe_route_plan.cu,moe_slot_sum.cu,csrc/cuda/router/dsv4_router.cucsrc/cuda/compressor/compress_fp8.cu,csrc/cuda/attention/dsv4_qo.cu,csrc/cuda/topk/stable_topk.cu,csrc/cuda/indexer/indexer_q_rope_fp8.cucsrc/cuda/logprob_dsv4/dsv4_logprob.curl_engine/models/deepseek_v4/lora.py,attention_backward.py,csrc/cuda/attention/dsv4_attn_bwd.cu.rl_engine/models/deepseek_v4/layerwise_gate.pyandtests/models/deepseek_v4/.build_tools/extensions.pyandcsrc/bindings/ops.cpp(SM90 MoE kernels behindKERNEL_ALIGN_MOE_SM90=1,sm_90aonly),rl_engine/_C.pyi,.github/workflows/ci.yml.Can be skimmed or skipped:
rl_engine/kernels/ops/pytorch/**(goldens),benchmarks/,scripts/probe_dsv4_*/scripts/check_*,docs/validation/*.json, and the legacy modules listed in design doc section 11.Dependencies
vllm-project/DeepGEMM@e21c821f(optional extradsv4)1.0.0+b70aff3(SM90 sources = vLLM's pina8f794d1)1.31.0All are optional at install time and checked fail-closed when a DSv4 op is constructed.
Testing
H200, torch 2.13 / CUDA 13.0,
KERNEL_ALIGN_MOE_SM90=1build:python -m pytest -q tests/ops/logprob_dsv4 tests/ops/mhc_dsv4 tests/models tests/ops/router \ tests/ops/moe tests/ops/compressor tests/ops/indexer tests/ops/topk tests/ops/attention_dsv4 \ tests/ops/quant tests/ops/activation tests/ops/gemm/test_dsv4_fp8_linear.py \ tests/ops/gemm/test_deepgemm_capability.py tests/test_deepgemm_shared_expert.py \ tests/test_sm90_fused_moe_mlp.py tests/test_sm90_moe_backward.py tests/build # 1217 passed, 72 skipped, 0 failed (80 s) PYTHONPATH=. python scripts/check_dsv4_layerwise_gate.py # GATE PASS, 0 diff at every layertorchrun --nproc-per-node 2 scripts/check_dsv4_moe_ep.py): forward and dX bitwise equal to EP = 1 (verified earlier on this branch). The same check with LoRA adapters (--lora) is written but has not run yet.