Repository navigation
feat: AVX2 Weight Pre-Decoding for FP8 & RAWINT4 MoE Kernels + Anchor Mechanism + Build Fixes - #2267
Open
YuvrajSinghBhadoria2 wants to merge 14 commits into
Open
YuvrajSinghBhadoria2 wants to merge 14 commits into
YuvrajSinghBhadoria2 wants to merge 14 commits into
Conversation
…es TODO) - Replace hardcoded 256 expert lists with glob-based file discovery - Dynamically find NUMA folders and expert files per layer - Parse expert indices from filenames - Follows same pattern as merge_cpu_weights.py
- Add 'mxfp4', 'mxfp8', 'rawint4' to --input-type choices - Update load_model_config() to parse quantization_config for new formats - Add RAWINT4ToColumnMajorConverter for raw INT4 weights (no AWQ interleaving) - Add MXFP4/MXFP8 dequantization with microscaling support in OnlineQuantConverter - Enable online quantization path for MXFP4/MXFP8 -> INT4/INT8 via AMXMoEWrapper
- Replace costly LUT gathers (_mm256_i32gather_ps) with exact bit arithmetic - Normal FP8: mantissa/exponent bits positioned at FP32 bits 20..26 with bias 0x3c000000 - Subnormals: integer mantissa * 2^-9 computed via _mm256_mul_ps - NaN (0x7F) → zero with sign preservation using blendv - Scalar fallback retains LUT for tail elements - ~23% faster single-token, ~2x faster 64-token CPU tails on AVX2 (per issue benchmarks) - Patch from: https://gist.github.com/f-wunderlich/118bffc7de22aa001d9852947de162a3
- Connect anchor computation (calc_anchor_all_layers) to attention pipeline - Add calls in attn_with_kvcache, attn, and get_attn_sparsity entry points - Fix duplicate scale_factor declaration in attn_with_kvcache_one_block_ - Add Python bindings for all KVCache methods with uintptr_t pointer conversion - Support all 5 anchor types: FIXED, DYNAMIC, QUEST, BLOCK_MEAN, BLOCK_MAX - Verified: all anchor types work, incremental generation works, no NaN/Inf outputs Tested on CPU with AVX2/FMA in Docker (Ubuntu 24.04)
- Add -mavx2 -mf16c -mfma flags for llamafile AVX2 sources (fixes compilation) - Conditional NUMA for macOS compatibility (fixes build on macOS) - Stub prefill_int4_cblas_gemm_s8s8s32 for non-BLIS builds (prevents linker errors) - Add CPU-only Docker test environment with .dockerignore - Add test script for FP8/RAWINT4 MoE kernels Tested pre-permutation caching for FP8/RAWINT4 following MXFP4 pattern: reverted as it adds 2-3x overhead without benefit (no matching weight decode emission order in these kernels).
This optimization pre-decodes quantized weights to FP32 once during weight loading (from_mat/from_raw_mat), avoiding on-the-fly decode in every GEMM call. Changes: - Added BufferB::predecoded float buffer to store FP32 weights [N, K] - Updated required_size() to include pre-decoded buffer size (n * k * sizeof(float)) - Updated BufferB constructors to set up predecoded pointer in memory layout - Added pre-decode logic in from_mat() for FP8 (using fp8x8_to_fp32x8 + scalar fallback) - Added pre-decode logic in from_raw_mat() for RAWINT4 (using rawint4_dequant_8x4bit_unscaled with scales applied) - Updated gemm_fp8() and gemm_rawint4() to use pre-decoded weights when available with fallback to on-the-fly decode for scale-only mode Expected: 20-40% prefill speedup by eliminating repeated FP8/INT4->FP32 conversion. Follows the same pattern as MXFP4 pre-permutation caching but adapted for FP8/RAWINT4's sequential weight access pattern (no kGroupPerm needed).
- Add Debug build configuration with -g -O0 flags (instead of hardcoded -O3) - Add AVX2/F16C/FMA compile flags for ext_bindings.cpp and moe-sft-tp.cpp - Add missing stubs for decode_cblas_gemm_s8s8s32, prefill_cblas_gemm_s8s8s32, decode_int4_cblas_gemm_s8s8s32 - Add get_reorder_B_size and reorder_B_gemm stubs for non-BLIS builds - These prevent linker errors when building without AOCL/BLIS
- Update Dockerfile.cpu-test to use Debug build type (consistent with CMakeLists.txt Debug support) - Fix import in bench_moe.py to use direct import kt_kernel_ext - Update test_fp8_rawint4.py with improved API usage (AVX2FP8_MOE, AVX2RawInt4_MOE with MOEConfig) - Add vectorized RAWINT4 quantization for faster test execution
Add OffloadEngine that keeps quantized weights in system RAM and pre-decodes only active experts on-demand, with async prefetch and LRU eviction, enabling models larger than CPU cache/RAM. - offload_engine.hpp/.cpp: OffloadConfig, ExpertCacheEntry, OffloadEngine with cache, prefetch thread, LRU eviction, FP8/RAWINT4 weight decode - fp8_moe_offload.hpp / rawint4_moe_offload.hpp: MoE variants using pre-decoded weights from OffloadEngine - fp8-moe.hpp: gemm_fp8_with_predecoded (direct FP32 weights, no decode) - rawint4-moe.hpp: gemm_rawint4_with_predecoded; BufferA 2/3-arg ctors - CMakeLists.txt: add offload_engine.cpp to AVX2 build - ext_bindings.cpp: bind AVX2FP8_MOE_OFFLOAD / AVX2RawInt4_MOE_OFFLOAD - bench/test_offload_moe.py: Python test driving AVX2FP8_MOE_OFFLOAD macOS: hwloc NUMA binding is unsupported on Darwin; use the Ubuntu CPU Docker test path (docker/Dockerfile.cpu-test). Offload engine headers guard NUMA includes with __linux__ / __APPLE__ fallbacks.
… reference tests Fix FP8 offload engine double-scaling: decode_fp8_weight_matrix was applying *scale during decode AND gemm_fp8_with_predecoded applied *scale again, producing ~1e21 output vs expected ~1e3. Removed scale from decode (gemm applies it once, matching original gemm_fp8). Fix RAWINT4 scale element count: gate_up_scale_elems/down_scale_elems were computed as rows * num_col_blocks (e.g. 128) instead of num_row_blocks * num_col_blocks (e.g. 4), causing scale pointer to read garbage memory -> nan/inf output. Fixed to use block-count formula. Add reference tests: - bench/test_offload_reference.py: FP8 vs PyTorch reference (passes, max diff ~10 on values up to ~2266) - bench/test_offload_reference_rawint4.py: RAWINT4 vs PyTorch reference (exposes forward-pass buffer wiring issue in MoE base class; offload engine decode/gemm produce correct values) Also removes debug prints from fp8-moe.hpp, fp8_moe_offload.hpp, rawint4_moe_offload.hpp, and offload_engine.cpp.
Fix RAWINT4 numerical correctness issues: 1. decode_rawint4_weight_matrix: scale indexing used per-row (scales + r * k_group_count) but scales are block-wise [div_up(rows, k_group_size), div_up(cols, k_group_size)]. Fixed to use scales + (r / k_group_size) * k_group_count, matching FP8's block-indexed scale access. 2. OffloadConfig: added explicit is_rawint4 flag instead of inferring from k_group_size != block_size, which breaks when block_size=0 (RAWINT4) or k_group_size == block_size. 3. bench/test_offload_reference_rawint4.py: fixed Python reference to use block-wise scales (matching C++ allocation) and nested loops over rows and columns within each block (was computing only diagonal elements). FP8 and RAWINT4 both now match PyTorch reference within tolerance (~0.4% relative error, expected for quantized weights + BF16 output).
Author
|
Hi — pinging on this PR. Happy to rebase onto latest main, add benchmarks, or split into smaller chunks if that helps unblock review. Let me know what's needed. |
2 tasks done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements weight pre-decoding optimization for AVX2 FP8 and RAWINT4 MoE kernels, plus anchor mechanism in KV cache and AVX2 build fixes.
1. Weight Pre-Decoding for FP8 & RAWINT4 MoE (feat)
Problem
FP8/RAWINT4 MoE kernels were decoding weights (FP8→FP32, INT4→FP32) on-the-fly in every GEMM call during inference — redundant work since weights are static.
Solution
Pre-decode quantized weights to FP32 once during
load_weights()/from_mat()/from_raw_mat(), store inBufferB::predecodedbuffer, and use pre-decoded FP32 weights in GEMM.Files Changed
kt-kernel/operators/avx2/fp8-moe.hppBufferB::predecoded(float[N,K]),required_size()+nksizeof(float), pre-decode infrom_mat()viafp8x8_to_fp32x8(), use ingemm_fp8()kt-kernel/operators/avx2/rawint4-moe.hpprawint4_dequant_8x4bit_unscaled()+ scale application infrom_raw_mat()Why Not MXFP4 Pre-Permutation?
MXFP4 uses
kGroupPermto match weight decode emission order. FP8 (block-wise) and RAWINT4 (K-group) access weights sequentially — no permutation benefit, only 2-3x overhead from extra memory traffic. Weight pre-decoding is the correct pattern.Performance (AVX2, 4 threads, 1024 token prefill)
Both kernels pass functional correctness tests.
2. Anchor Mechanism in KV Cache (feat)
calc_anchor_all_layers()to attention pipeline (attn_with_kvcache,attn,get_attn_sparsity)scale_factordeclaration buguintptr_tpointer conversion3. MXFP4/MXFP8/RAWINT4 Input Support (feat)
mxfp4,mxfp8,rawint4to--input-typeinconvert_cpu_weights.pyRAWINT4ToColumnMajorConverter(raw INT4, no AWQ interleaving)OnlineQuantConverterAMXMoEWrapper4. AVX2 Build Fixes & Docker Test Env (fix)
CMakeLists.txt:-mavx2 -mf16c -mfmafor 6 llamafile AVX2 sources (fixes compilation)CMakeLists.txt: Conditional NUMA for macOS compatibilityCMakeLists.txt: Debug build support (-g -O0for Debug,-O3 -ffast-mathfor Release)CMakeLists.txt: AVX2 flags forext_bindings.cpp,moe-sft-tp.cppmat_kernel.cpp: Stubs for 6 missing CBLAS INT8/INT4 functions (non-BLIS builds)docker/Dockerfile.cpu-test: CPU-only test container (238MB context)kt-kernel/bench/test_fp8_rawint4.py: Comprehensive test for both kernelsTesting
Expected output: All 4 tests pass (FP8/RAWINT4 prefill + decode).