Skip to content

[perf](kt-kernel): reuse FP8 LoRA recompute buffers - #2273

Draft
yyj6666667 wants to merge 1 commit into
kvcache-ai:mainfrom
yyj6666667:perf/fp8-lora-recompute-buffers
Draft

yyj6666667 wants to merge 1 commit into
kvcache-ai:mainfrom
yyj6666667:perf/fp8-lora-recompute-buffers

Conversation

@yyj6666667

@yyj6666667 yyj6666667 commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

What does this PR do?

Extends the recompute workspace reuse from #2272 to native E4M3, 128×128 block-FP8 CPU-expert LoRA. #2272 is now merged; this branch contains only the FP8 increment on top of its squash merge, whose tree is identical to the tested baseline c1174e31f0d5be3c68013a7c044b3fdb269396a6.

Per-TP A/C arenas reuse storage across completed forward/backward phases. FP8 BufferC::required_size includes both FP32 c and reduce_buf; both remain present. Gate/up regions stay disjoint while concurrent, and every expert sets max_m before set_data rebinds the two planes. A separate direct-activation capability lets FP8 write unified BF16 outputs without enabling BF16 FullFT cache behavior. Initial checkpoint forward uses storage without publishing a backward cache; recompute records inputs/routing and preserves raw gate separately from intermediate.

Automatic eligibility remains CPU-only LoRA, shared recompute activation storage and the authoritative gradient lifecycle. Unsupported/retain/Full/Hybrid paths retain their previous layout. No new YAML fields/environment variables, checkpoint format, quantization, routing, arithmetic order, input deduplication or token chunking.

Validation in progress (Draft)

  • Independently built baseline/candidate on qj5090, AMD EPYC 9355 AVX512-BF16. Intel AMX hardware is not covered.
  • FP8 96-case matrix and BF16 96-case regression passed, including independent-build comparison. TP1/2, 1/4 workers, top-k1/8, ranks8/16/32, GA1/4, dropout0/0.1, nonaligned lengths, skew/empty experts, alternating layers and growth. FP8 scales are nonunit across blocks; frozen weights/scales remain unchanged.
  • Compares forward, input/router gradients, all six LoRA gradients and AdamW updates. TP1/single-worker comparison is bitwise; multithread tolerance is rtol=1e-2, atol=1e-5, with finite/activity assertions.
  • Both 96-case matrices passed ASan with allocation/free poisoning. Mixed BF16/FP8 pool growth and eligibility checks passed; 262 related regressions passed.
  • Native FP8 Qwen35B baseline short training passed on physical GPU1, with all 40 CPU expert layers verified as FP8 and actual FA2/FLA dispatch checked. GPU1 was explicitly approved after GPU0 became occupied by another service.
  • Additional Qwen H2048/I512, TP2×32-worker GA1 checks passed on baseline/candidate/ASan. A separate GA4 random-gradient stress test is not consistently within the strict tolerance: baseline and candidate each failed 1 of 3 repeated comparisons at the same one of 147,456 down_lora_b elements (0.0014877319 versus 0.0015182495, delta 3.05e-5). The upstream short-token path merges FP32 tile gradients in mutex acquisition order before BF16 GA accumulation; no ordering or tolerance was changed here. Single-thread GA4 and all prescribed 1/4-worker matrix checks pass. The raw failures and controls are retained as a known limitation.
  • 16k/32k/64k memory and 16k/64k paired TPS acceptance are pending. The theoretical saving is 504 MiB per 1,024 tokens for H2048/I512/top-k8/TP2 (352 FP32 + 72 packed BF16 + 80 duplicate BF16), excluding padding/alignment. This is not a measured result. Native Trainer TPS median must not regress by more than 3%.

The installed Transformers non-expert FP8 loader rejects the complete checkpoint's converted keys. For hardware runs only, an isolated BF16 non-expert view is generated from the same FP8 checkpoint: all 963 tensors are bitwise equal to the installed Transformers dequantizer. CPU experts/scales load unchanged from the original 14 shards through existing kt_expert_checkpoint_path. This does not alter the PR's kernel scope or shared environment.

The repository CPU workflow requires a run-ci label and rejects Draft PRs. This PR remains Draft; local hardware validation must not be interpreted as GitHub CPU CI success. 128k is outside this acceptance run because of the previously identified independent GPU-memory limit.

Before submitting

  • Read the contributor guideline.
  • Added necessary numerical, lifecycle, growth and compatibility regressions.
  • Finish and publish long-sequence memory/TPS evidence.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant