IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

mx_dequant

MXFP4 and MXFP8 weight dequantization (E8M0 block-32 scales).

Both OCP microscaling formats share the scale structure: one float8_e8m0fnu scale (a pure power of two) per 32 consecutive elements along K, and no tensor-wide scale. They differ only in the element encoding, which is selected by the dtype of input:

  • MXFP4: uint8, two E2M1 values per byte (low nibble first), [N, K // 2].
  • MXFP8: float8_e4m3fn, one value per element, [N, K].

The output is bfloat16 or float32:

out[r, c] = element[r, c] * 2^(scale_code[r, c // 32] - 127)

Because the scale is a power of two the multiply is exact in float32 (barring overflow), so float32 output is bit-identical to a host reference. bfloat16 output rounds once, at the final cast.

Two block-scale layouts are accepted, selected by the rank of scales:

  • Rank 2, [N, K // 32]: the checkpoint layout.
  • Rank 5, [ceildiv(N, 128), ceildiv(K, 128), 32, 4, 4]: the SM100/SM120 TCGEN05 interleaved layout. With 32-element blocks one atom spans 128 columns, so the second dimension differs from NVFP4's ceildiv(K, 64).

An E8M0 NaN code (0xFF) propagates as NaN.

The kernel is an elementwise over [N, K] with the scale-block width as the SIMD width, so one thread dequantizes one 32-element scale block, occupancy does not depend on K, and consecutive threads touch consecutive bytes of every tensor. This is separate from the older H100-only mxfp4_dequant kernel, which stays the path for bf16/fp8 output with rank-2 scales on GPU.

GPU only: the graph compiler does not support FP8 or E8M0 tensors on a CPU device, so there is no way to reach a CPU path from a graph.

Functions​

  • ​dequant_mx: Dequantize MXFP4 or MXFP8 weights to bfloat16 or float32.

Was this page helpful?