Repository navigation
[perf](kt-kernel): reuse FP8 LoRA recompute buffers - #2273
Draft
yyj6666667 wants to merge 1 commit into
Draft
yyj6666667 wants to merge 1 commit into
yyj6666667 wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Extends the recompute workspace reuse from #2272 to native E4M3, 128×128 block-FP8 CPU-expert LoRA. #2272 is now merged; this branch contains only the FP8 increment on top of its squash merge, whose tree is identical to the tested baseline
c1174e31f0d5be3c68013a7c044b3fdb269396a6.Per-TP A/C arenas reuse storage across completed forward/backward phases. FP8
BufferC::required_sizeincludes both FP32candreduce_buf; both remain present. Gate/up regions stay disjoint while concurrent, and every expert setsmax_mbeforeset_datarebinds the two planes. A separate direct-activation capability lets FP8 write unified BF16 outputs without enabling BF16 FullFT cache behavior. Initial checkpoint forward uses storage without publishing a backward cache; recompute records inputs/routing and preserves raw gate separately from intermediate.Automatic eligibility remains CPU-only LoRA, shared recompute activation storage and the authoritative gradient lifecycle. Unsupported/retain/Full/Hybrid paths retain their previous layout. No new YAML fields/environment variables, checkpoint format, quantization, routing, arithmetic order, input deduplication or token chunking.
Validation in progress (Draft)
rtol=1e-2, atol=1e-5, with finite/activity assertions.down_lora_belements (0.0014877319versus0.0015182495, delta3.05e-5). The upstream short-token path merges FP32 tile gradients in mutex acquisition order before BF16 GA accumulation; no ordering or tolerance was changed here. Single-thread GA4 and all prescribed 1/4-worker matrix checks pass. The raw failures and controls are retained as a known limitation.The installed Transformers non-expert FP8 loader rejects the complete checkpoint's converted keys. For hardware runs only, an isolated BF16 non-expert view is generated from the same FP8 checkpoint: all 963 tensors are bitwise equal to the installed Transformers dequantizer. CPU experts/scales load unchanged from the original 14 shards through existing
kt_expert_checkpoint_path. This does not alter the PR's kernel scope or shared environment.The repository CPU workflow requires a
run-cilabel and rejects Draft PRs. This PR remains Draft; local hardware validation must not be interpreted as GitHub CPU CI success. 128k is outside this acceptance run because of the previously identified independent GPU-memory limit.Before submitting