For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
nvfp4_dequant
NVFP4 weight dequantization.
Converts packed NVFP4 weights (uint8, two E2M1 values per byte, low nibble
first) with float8_e4m3fn block scales and a tensor-wide float32 scale into
bfloat16 or float32:
out[r, c] = (e2m1(nibble) * |block_scale[r, c // 16]|) * tensor_sftensor_sf is the decode scale, i.e. the checkpoint's weight_scale_2, the
same value the NVFP4 matmul applies to its output. It is not the inverted
factor the quantize op takes. The multiplication order matches the host-side
reference (max.pipelines.weights.fp4_quantization.dequantize_nvfp4): the
E2M1 value times the block scale is exact in float32, so only the final
multiply rounds, and float32 output is bit-identical to the reference.
Two block-scale layouts are accepted, selected by the rank of scales:
- Rank 2,
[N, K // 16]: the checkpoint layout. - Rank 5,
[ceildiv(N, 128), ceildiv(K, 64), 32, 4, 4]: the SM100 TCGEN05 interleaved layout thatquantize_dynamic_block_scaledemits on SM100/SM120.
E4M3 NaN scale codes are not special-cased: they propagate as NaN.
The kernel is an elementwise over [N, K] with the scale-block width as the
SIMD width, so one thread dequantizes one 16-element scale block: one 8-byte
packed load, one scale load, and one contiguous store of 16 outputs. Occupancy
does not depend on K, and consecutive threads touch consecutive bytes of
every tensor.
GPU only: the graph compiler does not support FP8 tensors on a CPU device, so there is no way to reach a CPU path from a graph.
Functions
-
dequant_nvfp4: Dequantize NVFP4 packed weights to bfloat16 or float32.