For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
nvfp4_kv_dequant
NVFP4 KV-cache dequantization helpers for the SM100 sparse MLA decode kernel.
Target family: Blackwell sm_100.
An NVFP4 KV operand is a uint8 cache that packs two e2m1 elements per byte
(element 0 in the low nibble) under one float8_e4m3fn scale per 16 elements.
The kernel dequantizes it to FP8 in shared memory, so Q@K and P@V stay on the
FP8 tcgen05.mma.kind::f8f6f4 path.
An MLA row keeps its trailing NVFP4_MLA_ROPE_DIM rope dims as raw FP8 bytes
instead: the packed latent comes first, then the rope bytes, and only the
latent has scales. The rope bytes are exactly what an FP8 cache holds, so the
kernel copies them to shared memory without a conversion.
Each FP8 value is e2m1 * sf rounded once to e4m3. A 16-element group shares
one scale and e2m1 has only 8 magnitudes, so the eight products are converted
once and every element is then a byte lookup.
The e4m3_magnitude_lut / e2m1x8_to_e4m3x8 pair mirrors the helpers in
Kernels/lib/msa/nvfp4_kv.mojo, which cannot be imported from max/kernels.
comptime values
NVFP4_MLA_ROPE_DIM
comptime NVFP4_MLA_ROPE_DIM = 64
NVFP4_SF_VECTOR_SIZE
comptime NVFP4_SF_VECTOR_SIZE = 16
Functions
-
e2m1x8_scaled_to_e4m3x8: Converts eight e2m1 (element 0 in the low nibble) times a broadcast f16 scale pair to eight e4m3 bytes, through the hardware converts. -
e2m1x8_to_e4m3x8: Converts eight e2m1 (element 0 in the low nibble) to eight e4m3 bytes. -
e4m3_magnitude_lut:e4m3(m * scale)for the e2m1 magnitudes{0, .5, 1, 1.5, 2, 3, 4, 6}. -
e4m3_magnitude_lut_f16:e4m3_magnitude_lutfrom a broadcast f16 scale pair. -
e4m3x2_to_f16x2: Widens the two E4M3 bytes in the low half ofpairto an f16 pair. -
mla_kv_is_nvfp4: Whether the KV operand stores packed NVFP4 (uint8+ E4M3 scales). -
nvfp4_dequant_group: Expands one scale group (8 packed bytes, 16 e2m1) to 16 e4m3 bytes. -
nvfp4_dequant_group_f16:nvfp4_dequant_groupfrom a broadcast f16 scale pair. -
nvfp4_mla_row_bytes: Bytes of one NVFP4 MLA cache row: the packed latent, then FP8 rope. -
nvfp4_mla_scale_row_bytes: Scale bytes of one NVFP4 MLA row; only the latent has scales.