Repository navigation
[perf](kt-kernel): reuse BF16 LoRA recompute buffers - #2272
Merged
yyj6666667 merged 2 commits intoNov 13, 2026
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Long-sequence BF16 CPU-expert LoRA training keeps independent forward/backward GEMM scratch and duplicate BF16 results resident even when CPU activations are recomputed. Reuse the packed-input and FP32 accumulation storage across synchronous phases, and write BF16 outputs directly into shared activation storage from the first no-save forward.
This is automatically selected for BF16 CPU experts with LoRA, shared CPU-recompute activations, the existing authoritative optimizer-gradient lifecycle, no GPU experts, and no full-weight gradients. Retain, Full/Hybrid, SkipLoRA, legacy gradient paths and other kernel formats keep their existing layouts. No new YAML setting/environment variable is required; checkpoint formats, top-k, precision and arithmetic order are unchanged.
The constructor selects the internal layout before base allocations. Per-TP shared pools own the memory; forward/backward reacquire their views after any growth. Gate and up keep disjoint C views because they run concurrently. Later phases reuse the pools after existing worker barriers. Raw gate and post-activation intermediate remain distinct. A no-save forward uses storage without pushing or validating a backward cache; recompute publishes the inputs/routing/cache needed by backward.
For Qwen3.5-35B-A3B with two CPU TP partitions, the logical target, per additional 1,024 tokens, is:
These are payload targets, not measured PSS savings. Allocations use kernel required sizes, expert padding and alignment.
Validation
Independent baseline/candidate builds use the same compiler and dependencies, with separate import paths. Every training run records the loaded extension SHA256. The final revision passed:
rtol=1e-2, atol=1e-5. Active-expert gradients must be nonzero and inactive-expert gradients zero.Hardware: dual AMD EPYC 9355, CPU TP=2 / 32 threads per NUMA (64 total), RTX 5090 GPU0 only. This exercises AVX512 BF16; Intel AMX hardware has not been validated. Model: Qwen3.5-35B-A3B BF16, LoRA r8/alpha16/dropout0, batch1/GA4, seed42, FA2 + FLA + Liger fused loss, CPU/GPU activation recompute. Eight fixed synthetic telemetry examples are reused with the same order and seed.
Memory
Each memory diagnostic runs one optimizer step (GA4). PSS snapshots are taken after microbatch 4 backward, before optimizer step. Units are GiB (2^30 bytes); 1k means 1,024 tokens. The last column is a 5-second sampled process-lifetime peak, including load/save, and can miss brief peaks.
The three changes were verified separately using source-defined allocation regions and
mincore, without reading/touching the payload. These residency figures are distinct from allocation capacity and process PSS:The measured 16k-to-64k endpoint PSS growth slope is 837.12 → 509.88 MiB per additional 1k tokens, a reduction of 327.24 MiB/1k.
The old output/A/C regions are no longer allocated in the eligible path. Differences from the 328 MiB/1k logical target include expert padding/alignment, actual input lengths and the touched-page high-water mark. Process PSS additionally includes other allocations and shared-page accounting; the residual is reported without attributing all of it to the allocator. Probe snapshots reported zero errors and zero dropped allocations. 128k is excluded because of the independent GPU-memory limit on this device.
As a quantitative cross-check, dividing shared-C residency by
2 TP * 4 bytes * H2048 * top-k8gives equivalent touched lengths of approximately 16,968 / 33,448 / 66,168. These are workspace high-water equivalents, not actual input/routing counts. Substituting them in the 176/72 MiB-per-1k formulas and using actual input lengths for the 80 MiB duplicate-output formula explains the three independently measured region deltas to within 25 KiB combined at every length.Throughput
Memory instrumentation and periodic PSS/GPU sampling are disabled. Each length has three paired runs in B/C, C/B, B/C order, four optimizer steps per run. No compilation or correctness tests run concurrently with these measurements.
The first launch attempt for 64k candidate repeat 2 was rejected by the GPU-idle preflight before training started. Its failure log is retained. The same sequence resumed with a three-second stable-idle wait before Trainer timing; no completed measurement was discarded.
Native TPS is the final (step 4) native Trainer log, using non-padding input tokens; this installed Trainer's final
train_losssummary omits TPS. The separate post-warmup value uses the token/time delta betweenon_step_end(1)andon_step_end(4), including inter-step logging. Neither includes model loading. Input-token counts match; losses are finite and total gradient norms are finite/nonzero. Independent full-model runs have nondeterministic variation, including within the baseline build. The strict tensor tolerance above is checked with aligned operator inputs/weights, separately from these aggregate training metrics.Native TPS values by repeat (B/C for each pair):
Full-training metric differences (same-build spread is the maximum per-step
(max-min)/medianacross three repeats; its component-level cause has not been isolated):Same-build maximum loss spread (baseline/candidate): 16k 1.23%/0.43%; 64k 0.97%/1.01%.
Repository CI status
The six Book-CI/Deploy documentation jobs passed. The KT CPU job did not run:
check-changesfailed because this PR lacks therun-cilabel. The repository workflow also explicitly rejects Draft PRs at the next gate (workflow at the tested base). This PR remains Draft as requested; the hardware acceptance results above are separate from repository CI.Before submitting