Skip to content

[perf](kt-kernel): reuse BF16 LoRA recompute buffers - #2272

Merged
yyj6666667 merged 2 commits into
kvcache-ai:mainfrom
yyj6666667:perf/bf16-lora-recompute-buffers
Nov 13, 2026
Merged

yyj6666667 merged 2 commits into
kvcache-ai:mainfrom
yyj6666667:perf/bf16-lora-recompute-buffers

Conversation

@yyj6666667

@yyj6666667 yyj6666667 commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

What does this PR do?

Long-sequence BF16 CPU-expert LoRA training keeps independent forward/backward GEMM scratch and duplicate BF16 results resident even when CPU activations are recomputed. Reuse the packed-input and FP32 accumulation storage across synchronous phases, and write BF16 outputs directly into shared activation storage from the first no-save forward.

This is automatically selected for BF16 CPU experts with LoRA, shared CPU-recompute activations, the existing authoritative optimizer-gradient lifecycle, no GPU experts, and no full-weight gradients. Retain, Full/Hybrid, SkipLoRA, legacy gradient paths and other kernel formats keep their existing layouts. No new YAML setting/environment variable is required; checkpoint formats, top-k, precision and arithmetic order are unchanged.

The constructor selects the internal layout before base allocations. Per-TP shared pools own the memory; forward/backward reacquire their views after any growth. Gate and up keep disjoint C views because they run concurrently. Later phases reuse the pools after existing worker barriers. Raw gate and post-activation intermediate remain distinct. A no-save forward uses storage without pushing or validating a backward cache; recompute publishes the inputs/routing/cache needed by backward.

For Qwen3.5-35B-A3B with two CPU TP partitions, the logical target, per additional 1,024 tokens, is:

Change Logical reduction
FP32 accumulation scratch 176 MiB
Packed input scratch 72 MiB
Duplicate BF16 results 80 MiB
Total 328 MiB

These are payload targets, not measured PSS savings. Allocations use kernel required sizes, expert padding and alignment.

Validation

Independent baseline/candidate builds use the same compiler and dependencies, with separate import paths. Every training run records the loaded extension SHA256. The final revision passed:

  • 96 numerical configurations: TP 1/2, 1/4 threads per TP, top-k 1/8, ranks 8/16/32, GA 1/4, dropout 0/0.1; nonaligned lengths, skewed/uniform routing, inactive experts, alternating layers and pool growth.
  • Forward output, dInput, routing gradients, all six expert LoRA gradients and AdamW updates. Single-thread TP1 is bitwise identical; multithread tolerance is rtol=1e-2, atol=1e-5. Active-expert gradients must be nonzero and inactive-expert gradients zero.
  • ASan with allocation/free poisoning: all 96 configurations plus both allocation-policy checks passed, with no ASan memory errors. Fresh-process logging verifies that legacy gradient callers retain the old allocation policy.
  • 245 activation-policy, router, checkpoint, authoritative-gradient, runtime-ownership, backend-dispatch and numerical regressions passed.
  • Supplementary H2048/I512 (I256 per TP), TP2 x 32-worker numerical checks passed for baseline/candidate/ASan, with GA4 and dropout 0/0.1. This bounded fixture keeps 9 experts and places both worker groups on NUMA0.

Hardware: dual AMD EPYC 9355, CPU TP=2 / 32 threads per NUMA (64 total), RTX 5090 GPU0 only. This exercises AVX512 BF16; Intel AMX hardware has not been validated. Model: Qwen3.5-35B-A3B BF16, LoRA r8/alpha16/dropout0, batch1/GA4, seed42, FA2 + FLA + Liger fused loss, CPU/GPU activation recompute. Eight fixed synthetic telemetry examples are reused with the same order and seed.

Memory

Each memory diagnostic runs one optimizer step (GA4). PSS snapshots are taken after microbatch 4 backward, before optimizer step. Units are GiB (2^30 bytes); 1k means 1,024 tokens. The last column is a 5-second sampled process-lifetime peak, including load/save, and can miss brief peaks.

Sequence setting Actual input length Baseline PSS Candidate PSS Saved PSS Sampled peak baseline / candidate
16k 16400 83.658 78.402 5.256 86.172 / 79.017
32k 32792 96.777 86.382 10.395 97.478 / 89.078
64k 65552 122.898 102.303 20.595 125.465 / 105.940

The three changes were verified separately using source-defined allocation regions and mincore, without reading/touching the payload. These residency figures are distinct from allocation capacity and process PSS:

Sequence FP32 scratch saved MiB Packed-input scratch saved MiB Duplicate BF16 saved MiB PSS minus summed region savings MiB
16k 2916.38 1193.06 1281.27 -8.96
32k 5748.88 2351.81 2561.90 -18.38
64k 11372.62 4652.44 5121.27 -57.23

The measured 16k-to-64k endpoint PSS growth slope is 837.12 → 509.88 MiB per additional 1k tokens, a reduction of 327.24 MiB/1k.

The old output/A/C regions are no longer allocated in the eligible path. Differences from the 328 MiB/1k logical target include expert padding/alignment, actual input lengths and the touched-page high-water mark. Process PSS additionally includes other allocations and shared-page accounting; the residual is reported without attributing all of it to the allocator. Probe snapshots reported zero errors and zero dropped allocations. 128k is excluded because of the independent GPU-memory limit on this device.

As a quantitative cross-check, dividing shared-C residency by 2 TP * 4 bytes * H2048 * top-k8 gives equivalent touched lengths of approximately 16,968 / 33,448 / 66,168. These are workspace high-water equivalents, not actual input/routing counts. Substituting them in the 176/72 MiB-per-1k formulas and using actual input lengths for the 80 MiB duplicate-output formula explains the three independently measured region deltas to within 25 KiB combined at every length.

Throughput

Memory instrumentation and periodic PSS/GPU sampling are disabled. Each length has three paired runs in B/C, C/B, B/C order, four optimizer steps per run. No compilation or correctness tests run concurrently with these measurements.

The first launch attempt for 64k candidate repeat 2 was rejected by the GPU-idle preflight before training started. Its failure log is retained. The same sequence resumed with a three-second stable-idle wait before Trainer timing; no completed measurement was discarded.

Sequence Native Trainer TPS baseline median Candidate median Change Post-warmup TPS baseline / candidate 3% regression gate
16k 968.68 1005.83 +3.83% 964.72 / 1002.82 PASS
64k 949.73 973.43 +2.50% 946.70 / 968.62 PASS

Native TPS is the final (step 4) native Trainer log, using non-padding input tokens; this installed Trainer's final train_loss summary omits TPS. The separate post-warmup value uses the token/time delta between on_step_end(1) and on_step_end(4), including inter-step logging. Neither includes model loading. Input-token counts match; losses are finite and total gradient norms are finite/nonzero. Independent full-model runs have nondeterministic variation, including within the baseline build. The strict tensor tolerance above is checked with aligned operator inputs/weights, separately from these aggregate training metrics.

Native TPS values by repeat (B/C for each pair):

  • 16k: 974.603/1001.137, 967.827/1011.432, 968.679/1005.827.
  • 64k: 955.189/972.899, 949.732/978.876, 949.189/973.429.

Full-training metric differences (same-build spread is the maximum per-step (max-min)/median across three repeats; its component-level cause has not been isolated):

Sequence Max paired loss delta Max paired grad_norm delta Baseline repeat grad_norm spread Candidate repeat grad_norm spread
16k 1.23% 2.38% 3.74% 1.25%
64k 1.09% 1.87% 1.86% 2.49%

Same-build maximum loss spread (baseline/candidate): 16k 1.23%/0.43%; 64k 0.97%/1.01%.

Repository CI status

The six Book-CI/Deploy documentation jobs passed. The KT CPU job did not run: check-changes failed because this PR lacks the run-ci label. The repository workflow also explicitly rejects Draft PRs at the next gate (workflow at the tested base). This PR remains Draft as requested; the hardware acceptance results above are separate from repository CI.

Before submitting

  • Read the contributor guideline.
  • Added numerical, allocation-policy and lifecycle regression coverage.
  • Completed long-sequence memory and throughput acceptance on the hardware above.
  • Kept this PR in Draft, as requested; Intel AMX hardware coverage remains outstanding.

@yyj6666667
yyj6666667 marked this pull request as ready for review November 13, 2026 05:49
@yyj6666667
yyj6666667 merged commit 12bf63b into kvcache-ai:main Nov 13, 2026
7 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant