For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
mx_dequant
MXFP4 and MXFP8 weight dequantization (E8M0 block-32 scales).
Both OCP microscaling formats share the scale structure: one float8_e8m0fnu
scale (a pure power of two) per 32 consecutive elements along K, and no
tensor-wide scale. They differ only in the element encoding, which is selected
by the dtype of input:
- MXFP4:
uint8, two E2M1 values per byte (low nibble first),[N, K // 2]. - MXFP8:
float8_e4m3fn, one value per element,[N, K].
The output is bfloat16 or float32:
out[r, c] = element[r, c] * 2^(scale_code[r, c // 32] - 127)Because the scale is a power of two the multiply is exact in float32 (barring overflow), so float32 output is bit-identical to a host reference. bfloat16 output rounds once, at the final cast.
Two block-scale layouts are accepted, selected by the rank of scales:
- Rank 2,
[N, K // 32]: the checkpoint layout. - Rank 5,
[ceildiv(N, 128), ceildiv(K, 128), 32, 4, 4]: the SM100/SM120 TCGEN05 interleaved layout. With 32-element blocks one atom spans 128 columns, so the second dimension differs from NVFP4'sceildiv(K, 64).
An E8M0 NaN code (0xFF) propagates as NaN.
The kernel is an elementwise over [N, K] with the scale-block width as the
SIMD width, so one thread dequantizes one 32-element scale block, occupancy
does not depend on K, and consecutive threads touch consecutive bytes of
every tensor. This is separate from the older H100-only mxfp4_dequant
kernel, which stays the path for bf16/fp8 output with rank-2 scales on GPU.
GPU only: the graph compiler does not support FP8 or E8M0 tensors on a CPU device, so there is no way to reach a CPU path from a graph.
Functions
-
dequant_mx: Dequantize MXFP4 or MXFP8 weights to bfloat16 or float32.