Skip to content

feat: AVX2 Weight Pre-Decoding for FP8 & RAWINT4 MoE Kernels + Anchor Mechanism + Build Fixes - #2267

Open
YuvrajSinghBhadoria2 wants to merge 14 commits into
kvcache-ai:mainfrom
YuvrajSinghBhadoria2:feat/anchor-mechanism-kvcache
Open

YuvrajSinghBhadoria2 wants to merge 14 commits into
kvcache-ai:mainfrom
YuvrajSinghBhadoria2:feat/anchor-mechanism-kvcache

Conversation

@YuvrajSinghBhadoria2

@YuvrajSinghBhadoria2 YuvrajSinghBhadoria2 commented Nov 9, 2026 •

Copy link
Copy Markdown

Summary

This PR implements weight pre-decoding optimization for AVX2 FP8 and RAWINT4 MoE kernels, plus anchor mechanism in KV cache and AVX2 build fixes.


1. Weight Pre-Decoding for FP8 & RAWINT4 MoE (feat)

Problem

FP8/RAWINT4 MoE kernels were decoding weights (FP8→FP32, INT4→FP32) on-the-fly in every GEMM call during inference — redundant work since weights are static.

Solution

Pre-decode quantized weights to FP32 once during load_weights() / from_mat() / from_raw_mat(), store in BufferB::predecoded buffer, and use pre-decoded FP32 weights in GEMM.

Files Changed

File Key Changes
kt-kernel/operators/avx2/fp8-moe.hpp BufferB::predecoded (float[N,K]), required_size() +nksizeof(float), pre-decode in from_mat() via fp8x8_to_fp32x8(), use in gemm_fp8()
kt-kernel/operators/avx2/rawint4-moe.hpp Same pattern, pre-decode via rawint4_dequant_8x4bit_unscaled() + scale application in from_raw_mat()

Why Not MXFP4 Pre-Permutation?

MXFP4 uses kGroupPerm to match weight decode emission order. FP8 (block-wise) and RAWINT4 (K-group) access weights sequentially — no permutation benefit, only 2-3x overhead from extra memory traffic. Weight pre-decoding is the correct pattern.

Performance (AVX2, 4 threads, 1024 token prefill)

Kernel Prefill Decode m=1
FP8 E4M3 546 ms/iter 0.37 ms/iter
RAWINT4 227 ms/iter 0.40 ms/iter

Both kernels pass functional correctness tests.


2. Anchor Mechanism in KV Cache (feat)

  • Connected calc_anchor_all_layers() to attention pipeline (attn_with_kvcache, attn, get_attn_sparsity)
  • Fixed duplicate scale_factor declaration bug
  • Added Python bindings for all KVCache methods with uintptr_t pointer conversion
  • All 5 anchor types work: FIXED_ANCHOR, DYNAMIC, QUEST, BLOCK_MEAN, BLOCK_MAX
  • Tested on CPU with AVX2/FMA in Docker (Ubuntu 24.04)

3. MXFP4/MXFP8/RAWINT4 Input Support (feat)

  • Added mxfp4, mxfp8, rawint4 to --input-type in convert_cpu_weights.py
  • Added RAWINT4ToColumnMajorConverter (raw INT4, no AWQ interleaving)
  • Added MXFP4/MXFP8 dequantization with microscaling in OnlineQuantConverter
  • Enabled online quantization path for MXFP4/MXFP8 → INT4/INT8 via AMXMoEWrapper

4. AVX2 Build Fixes & Docker Test Env (fix)

  • CMakeLists.txt: -mavx2 -mf16c -mfma for 6 llamafile AVX2 sources (fixes compilation)
  • CMakeLists.txt: Conditional NUMA for macOS compatibility
  • CMakeLists.txt: Debug build support (-g -O0 for Debug, -O3 -ffast-math for Release)
  • CMakeLists.txt: AVX2 flags for ext_bindings.cpp, moe-sft-tp.cpp
  • mat_kernel.cpp: Stubs for 6 missing CBLAS INT8/INT4 functions (non-BLIS builds)
  • docker/Dockerfile.cpu-test: CPU-only test container (238MB context)
  • kt-kernel/bench/test_fp8_rawint4.py: Comprehensive test for both kernels

Testing

# In Docker with NUMA permissions
docker run --rm --cap-add=CAP_SYS_RESOURCE --cap-add=CAP_SYS_ADMIN --security-opt seccomp=unconfined \
  -v $(pwd):/workspace/ktransformers ktransformers-cpu-test:debug \
  /opt/venv/bin/python3 /workspace/ktransformers/kt-kernel/bench/test_fp8_rawint4.py

Expected output: All 4 tests pass (FP8/RAWINT4 prefill + decode).

…es TODO)

- Replace hardcoded 256 expert lists with glob-based file discovery
- Dynamically find NUMA folders and expert files per layer
- Parse expert indices from filenames
- Follows same pattern as merge_cpu_weights.py
- Add 'mxfp4', 'mxfp8', 'rawint4' to --input-type choices
- Update load_model_config() to parse quantization_config for new formats
- Add RAWINT4ToColumnMajorConverter for raw INT4 weights (no AWQ interleaving)
- Add MXFP4/MXFP8 dequantization with microscaling support in OnlineQuantConverter
- Enable online quantization path for MXFP4/MXFP8 -> INT4/INT8 via AMXMoEWrapper
- Replace costly LUT gathers (_mm256_i32gather_ps) with exact bit arithmetic
- Normal FP8: mantissa/exponent bits positioned at FP32 bits 20..26 with bias 0x3c000000
- Subnormals: integer mantissa * 2^-9 computed via _mm256_mul_ps
- NaN (0x7F) → zero with sign preservation using blendv
- Scalar fallback retains LUT for tail elements
- ~23% faster single-token, ~2x faster 64-token CPU tails on AVX2 (per issue benchmarks)
- Patch from: https://gist.github.com/f-wunderlich/118bffc7de22aa001d9852947de162a3
- Connect anchor computation (calc_anchor_all_layers) to attention pipeline
- Add calls in attn_with_kvcache, attn, and get_attn_sparsity entry points
- Fix duplicate scale_factor declaration in attn_with_kvcache_one_block_
- Add Python bindings for all KVCache methods with uintptr_t pointer conversion
- Support all 5 anchor types: FIXED, DYNAMIC, QUEST, BLOCK_MEAN, BLOCK_MAX
- Verified: all anchor types work, incremental generation works, no NaN/Inf outputs

Tested on CPU with AVX2/FMA in Docker (Ubuntu 24.04)
- Add -mavx2 -mf16c -mfma flags for llamafile AVX2 sources (fixes compilation)
- Conditional NUMA for macOS compatibility (fixes build on macOS)
- Stub prefill_int4_cblas_gemm_s8s8s32 for non-BLIS builds (prevents linker errors)
- Add CPU-only Docker test environment with .dockerignore
- Add test script for FP8/RAWINT4 MoE kernels

Tested pre-permutation caching for FP8/RAWINT4 following MXFP4 pattern:
reverted as it adds 2-3x overhead without benefit (no matching weight
decode emission order in these kernels).
@YuvrajSinghBhadoria2 YuvrajSinghBhadoria2 changed the title feat: Implement anchor mechanism in KV cache for sparse attention feat: anchor mechanism + MXFP4/MXFP8/RAWINT4 support + fix: AVX2 build, macOS NUMA, INT4 stub, Docker test env Nov 10, 2026
This optimization pre-decodes quantized weights to FP32 once during weight loading
(from_mat/from_raw_mat), avoiding on-the-fly decode in every GEMM call.

Changes:
- Added BufferB::predecoded float buffer to store FP32 weights [N, K]
- Updated required_size() to include pre-decoded buffer size (n * k * sizeof(float))
- Updated BufferB constructors to set up predecoded pointer in memory layout
- Added pre-decode logic in from_mat() for FP8 (using fp8x8_to_fp32x8 + scalar fallback)
- Added pre-decode logic in from_raw_mat() for RAWINT4 (using rawint4_dequant_8x4bit_unscaled with scales applied)
- Updated gemm_fp8() and gemm_rawint4() to use pre-decoded weights when available
  with fallback to on-the-fly decode for scale-only mode

Expected: 20-40% prefill speedup by eliminating repeated FP8/INT4->FP32 conversion.

Follows the same pattern as MXFP4 pre-permutation caching but adapted for
FP8/RAWINT4's sequential weight access pattern (no kGroupPerm needed).
- Add Debug build configuration with -g -O0 flags (instead of hardcoded -O3)
- Add AVX2/F16C/FMA compile flags for ext_bindings.cpp and moe-sft-tp.cpp
- Add missing stubs for decode_cblas_gemm_s8s8s32, prefill_cblas_gemm_s8s8s32, decode_int4_cblas_gemm_s8s8s32
- Add get_reorder_B_size and reorder_B_gemm stubs for non-BLIS builds
- These prevent linker errors when building without AOCL/BLIS
- Update Dockerfile.cpu-test to use Debug build type (consistent with CMakeLists.txt Debug support)
- Fix import in bench_moe.py to use direct import kt_kernel_ext
- Update test_fp8_rawint4.py with improved API usage (AVX2FP8_MOE, AVX2RawInt4_MOE with MOEConfig)
- Add vectorized RAWINT4 quantization for faster test execution
@YuvrajSinghBhadoria2 YuvrajSinghBhadoria2 changed the title feat: anchor mechanism + MXFP4/MXFP8/RAWINT4 support + fix: AVX2 build, macOS NUMA, INT4 stub, Docker test env feat: AVX2 Weight Pre-Decoding for FP8 & RAWINT4 MoE Kernels + Anchor Mechanism + Build Fixes Nov 10, 2026
Add OffloadEngine that keeps quantized weights in system RAM and
pre-decodes only active experts on-demand, with async prefetch and
LRU eviction, enabling models larger than CPU cache/RAM.

- offload_engine.hpp/.cpp: OffloadConfig, ExpertCacheEntry, OffloadEngine
  with cache, prefetch thread, LRU eviction, FP8/RAWINT4 weight decode
- fp8_moe_offload.hpp / rawint4_moe_offload.hpp: MoE variants using
  pre-decoded weights from OffloadEngine
- fp8-moe.hpp: gemm_fp8_with_predecoded (direct FP32 weights, no decode)
- rawint4-moe.hpp: gemm_rawint4_with_predecoded; BufferA 2/3-arg ctors
- CMakeLists.txt: add offload_engine.cpp to AVX2 build
- ext_bindings.cpp: bind AVX2FP8_MOE_OFFLOAD / AVX2RawInt4_MOE_OFFLOAD
- bench/test_offload_moe.py: Python test driving AVX2FP8_MOE_OFFLOAD

macOS: hwloc NUMA binding is unsupported on Darwin; use the Ubuntu CPU
Docker test path (docker/Dockerfile.cpu-test). Offload engine headers
guard NUMA includes with __linux__ / __APPLE__ fallbacks.
… reference tests

Fix FP8 offload engine double-scaling: decode_fp8_weight_matrix was
applying *scale during decode AND gemm_fp8_with_predecoded applied *scale
again, producing ~1e21 output vs expected ~1e3. Removed scale from decode
(gemm applies it once, matching original gemm_fp8).

Fix RAWINT4 scale element count: gate_up_scale_elems/down_scale_elems
were computed as rows * num_col_blocks (e.g. 128) instead of
num_row_blocks * num_col_blocks (e.g. 4), causing scale pointer to read
garbage memory -> nan/inf output. Fixed to use block-count formula.

Add reference tests:
- bench/test_offload_reference.py: FP8 vs PyTorch reference (passes,
  max diff ~10 on values up to ~2266)
- bench/test_offload_reference_rawint4.py: RAWINT4 vs PyTorch reference
  (exposes forward-pass buffer wiring issue in MoE base class; offload
  engine decode/gemm produce correct values)

Also removes debug prints from fp8-moe.hpp, fp8_moe_offload.hpp,
rawint4_moe_offload.hpp, and offload_engine.cpp.
Fix RAWINT4 numerical correctness issues:

1. decode_rawint4_weight_matrix: scale indexing used per-row (scales +
   r * k_group_count) but scales are block-wise
   [div_up(rows, k_group_size), div_up(cols, k_group_size)]. Fixed to
   use scales + (r / k_group_size) * k_group_count, matching FP8's
   block-indexed scale access.

2. OffloadConfig: added explicit is_rawint4 flag instead of inferring
   from k_group_size != block_size, which breaks when block_size=0
   (RAWINT4) or k_group_size == block_size.

3. bench/test_offload_reference_rawint4.py: fixed Python reference
   to use block-wise scales (matching C++ allocation) and nested loops
   over rows and columns within each block (was computing only
   diagonal elements).

FP8 and RAWINT4 both now match PyTorch reference within tolerance
(~0.4% relative error, expected for quantized weights + BF16 output).
@YuvrajSinghBhadoria2

Copy link
Copy Markdown
Author

Hi — pinging on this PR. Happy to rebase onto latest main, add benchmarks, or split into smaller chunks if that helps unblock review. Let me know what's needed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant