Repository navigation
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
…y gate Signed-off-by: effintell <dominicxhy@gmail.com>
… logprob Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
… logprob Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
…ead_logprob forward Signed-off-by: effintell <dominicxhy@gmail.com>
… parity Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
…ntraction of temperature scaling Signed-off-by: effintell <dominicxhy@gmail.com>
…fault-budget chunking and embedding grad_weight Signed-off-by: effintell <dominicxhy@gmail.com>
…_head_logprob backward Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
…n gradient tolerance Signed-off-by: effintell <dominicxhy@gmail.com>
…SM90 det-GEMM 32768 tree limit Signed-off-by: effintell <dominicxhy@gmail.com>
…2768 boundary Signed-off-by: effintell <dominicxhy@gmail.com>
141d9c7 to
21d0bca
Compare
|
🔍 OpenCodeReview found 5 issue(s) in this PR.
📄
|
|
@effintell The automated OCR review has completed. Please review and address all automated review bot findings, then push your updates. Maintainers will begin manual review after the bot findings have been addressed. |
…ned bf16 geometry Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
Signed-off-by: effintell <dominicxhy@gmail.com>
08f3c31 to
0c881f9
Compare
Implements the CUDA column of the RFC #434 row "Embedding lookup (untied), LM head 2688→131072 fused selected-token logprob (no softcap)" for Nemotron-3-Nano (extends #243, #204, #336).
NemotronHLMHeadLogprobSM90Op: strict, batch-invariant fused LM-head selected-token logprob (forward never materialises the[T, 131072]logits) + optional per-row temperature.grad_hidden/grad_weight, chunked by a fixed element budget (chunking is bit-neutral).grad_weight.TREE_DEPTH=10). Backward reductions with K > 32768 (grad_hiddenover vocab = 131072;grad_weightover tokens > 32768) previously crashed with an illegal memory access. They are now split into ≤ 32768 segments whose FP32 partials are summed in a fixed ascending order (SM90 backend only; cuBLASLt no-split-K needs no split).grad_weightis judged against FP64 with the WS1 reductiongradient_accuracytolerance (atol 0.1, rtol 0.02) instead of the logprob cell (atol 0.05, rtol 0) — see "Tolerance" below.Out of scope: ROCm, TP / CP / EP.
Validation (H100 PCIe, torch 2.8.0+cu128, nvcc 12.8)
Build:
KERNEL_ALIGN_FORCE_SM90=1 KERNEL_ALIGN_DET_GEMM_SM90=1 pip install -e . --no-build-isolationpython -m pytest tests/models/nemotron_h/test_lm_head_logprob_cuda.py::TestGeometryGate tests/runtime/test_dispatch.py -q(CPU)python -m pytest tests/models/nemotron_h -q -rs— SM90 det-GEMM (auto, no cuBLAS env)CUBLAS_WORKSPACE_CONFIG=:16:8 CUBLASLT_WORKSPACE_SIZE=1)linear_logpsuite fails with the identical error on the same shape for bothcuda-sm90andtritonbenchmarks/operators/logprob/benchmark_nemotron_h_lm_head_logp.py --json ...python -m pytest tests -q -k "linear_logp or fused_logp" -rs --ignore=tests/backends/rocm/test_rocm_aiter_api_contract.pyThe xfail is
test_embedding_grad_weight_matches_fp64(strict xfail):SM90EmbeddingOpaccumulates repeated-id gradients in BF16; bitwise reproducibility of that gradient is tested and passes.test_rocm_aiter_api_contract.pyfails to collect on the base branch as well.Acceptance checks
torch.equal): forward logprob / lse rows across batch sizes 1 / 2 / 7 / 64 / 255 / 257 (random row subsets), leading shape[3, 100], repeat runs, and train vs rollout entry;grad_hiddenrows across batch sizes 1 / 7 / 64 / 257.allcloseto an independent FP64 reference / VJP.fallback: false).det_gemm_backend_id, kernel ids, checkpointnvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16@bf77c31).det-GEMM split verification (SM90 backend, profiler-confirmed
sm90kernels,rlkernel.det_gemm.sm90.v1)All cases bitwise reproducible across two runs. "budget" = max_abs / allowed tolerance.
CUDA error: an illegal memory accessfor dW 32864, dW 65536 and dX 131072.test_grad_weight_token_reduction_beyond_det_gemm_treeis parametrised at exactly 32768 (1 segment), 32769 (K=1 tail) and 65600 (3 segments), asserts the segment count, and skips on non-SM90 backends.Disabling the split makes the 2- and 3-segment cases fail; dropping the last partial makes the 3-segment case fail (max diff 0.38 > 0.1).
Tolerance (
grad_weight)grad_weightis a reduction over tokens with entries up to |ref| ≈ 18. At that magnitude BF16 output rounding alone is 0.049 (half-ulp in [16, 32)), which exhausts the logprob cell's absolute-only budget(atol 0.05). Measured on the 64-token model test: kernel max_err 0.051 vs BF16-rounding floor 0.049 (2 / 352M elements > 0.05). So it is judged with the reduction cell (atol 0.1, rtol 0.02);
grad_hiddenstays on the logprob cell (max_err 0.0025).
Note: the SM90 det-GEMM tree accumulates in BF16 per level, so
grad_weighterror grows with token count — 42–54% of the reduction budget at ~32k tokens, 58–65% at ~65k tokens (≈ 2.5–4× the BF16 floor). Thesplit itself adds no measurable error.
Benchmark (Nemotron-3-Nano shape, H100 PCIe)
Train fwd+bwd, strict vs native PyTorch:
Forward only (prefill): 2048 tokens 19.1 vs 6.4 ms (3.0×), 8192 tokens 64.7 vs 24.8 ms (2.6×), peak memory halved. cuBLASLt no-split-K is the recommended det-GEMM backend for training.
Known issues / follow-ups
csrc/cuda/gemm/det_gemm_kernel.cu: when K is not a multiple of 32 or exceeds 32768, the SM90 det-GEMM silently falls back to the naive kernel instead of failing closed. This PR avoids the K > 32768 casefrom Python; the C++ behaviour will be tracked in a separate issue.
grad_weightFP64 parity (BF16 accumulation) — strict xfail.