For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
mxfp4_dequant_matmul_amd
MXFP4 matmul on AMD CDNA GPUs with BF16 activations.
Two routes, split on M:
- Decode and low batch (
M <= DECODE_MAX_M): cast the BF16 activations to FP8 and run the native block-scaled MMA against the still-packed MXFP4 weights. The CDNA4mfma.scale.f8f6f4instruction consumes E4M3 A and E2M1 B with their E8M0 block scales in hardware, so the weights are never widened to one byte per element and no temporary weight buffer is allocated. This is the memory-bound regime, where dropping the weight materialization pass and reading B at half width is worth 2.5-4.8x. - Mid batch and up, and every shape on a part without the instruction: the
same dequantize-to-FP8-then-tuned-FP8-GEMM route this file has always
had. It is compute-bound enough there that the extra DRAM pass is hidden
while the GEMM sustains ~2.3 PF/s. The mixed E4M3 x E2M1 tile is not tuned
yet (see the W4A8 TODO on
BlockScaledMmaOp.use_smem_swizzle), so it loses above the decode range.
The first route needs CDNA4 (mfma.scale.f8f6f4, gfx950); on any other part
only the second route exists, so nothing here instantiates the block-scaled
kernel for them.
Numerics match the dequant route exactly while every scaled weight stays
inside E4M3's representable range, which is what MX quantization produces: an
E2M1 value times a power of two is exact in E4M3 there, and the block scale is
derived from the block's largest magnitude. Outside that range the routes
differ, and the dequant route is the one that rounds first: for a block scale
of 2^-10, 0.5 * 2^-10 = 2^-11 is below E4M3's smallest subnormal 2^-9, so
dequant_mxfp4 rounds it to zero before the GEMM while the MMA forms the
product in its exponent domain.
That bound is enforced, not documented: the caller passes the weight's block-scale exponent range — one scalar pair on the packed-weight metadata, computed where the scales are produced, not scanned per call — and the native route only runs when that range proves representability. Callers that do not pass it, or whose weights fall outside it, take the dequant route.
The weight and scale views are the other half of the contract, and it is a type-level one: the native branch bakes both views' static shapes into the kernel and reads the scales as one dense row-major run, so a view with a runtime dimension or a strided scale view cannot instantiate it. Such callers take the dequant route, which indexes both views through their own strides.
comptime values
DECODE_MAX_M
comptime DECODE_MAX_M = 256
MX_BLOCK_SIZE
comptime MX_BLOCK_SIZE = 32
Functions
-
mxfp4_dequant_matmul_amd: MXFP4 matmul: native block-scaled MMA on packed B, or dequant + FP8 GEMM. -
mxfp4_native_route_enabled: Whethermxfp4_dequant_matmul_amdtakes the native block-scaled route. -
mxfp4_native_views_supported: Whether these weight and scale views can instantiate the native route.