IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

mxfp4_dequant_matmul_amd

MXFP4 matmul on AMD CDNA GPUs with BF16 activations.

Two routes, split on M:

  • Decode and low batch (M <= DECODE_MAX_M): cast the BF16 activations to FP8 and run the native block-scaled MMA against the still-packed MXFP4 weights. The CDNA4 mfma.scale.f8f6f4 instruction consumes E4M3 A and E2M1 B with their E8M0 block scales in hardware, so the weights are never widened to one byte per element and no temporary weight buffer is allocated. This is the memory-bound regime, where dropping the weight materialization pass and reading B at half width is worth 2.5-4.8x.
  • Mid batch and up, and every shape on a part without the instruction: the same dequantize-to-FP8-then-tuned-FP8-GEMM route this file has always had. It is compute-bound enough there that the extra DRAM pass is hidden while the GEMM sustains ~2.3 PF/s. The mixed E4M3 x E2M1 tile is not tuned yet (see the W4A8 TODO on BlockScaledMmaOp.use_smem_swizzle), so it loses above the decode range.

The first route needs CDNA4 (mfma.scale.f8f6f4, gfx950); on any other part only the second route exists, so nothing here instantiates the block-scaled kernel for them.

Numerics match the dequant route exactly while every scaled weight stays inside E4M3's representable range, which is what MX quantization produces: an E2M1 value times a power of two is exact in E4M3 there, and the block scale is derived from the block's largest magnitude. Outside that range the routes differ, and the dequant route is the one that rounds first: for a block scale of 2^-10, 0.5 * 2^-10 = 2^-11 is below E4M3's smallest subnormal 2^-9, so dequant_mxfp4 rounds it to zero before the GEMM while the MMA forms the product in its exponent domain.

That bound is enforced, not documented: the caller passes the weight's block-scale exponent range — one scalar pair on the packed-weight metadata, computed where the scales are produced, not scanned per call — and the native route only runs when that range proves representability. Callers that do not pass it, or whose weights fall outside it, take the dequant route.

The weight and scale views are the other half of the contract, and it is a type-level one: the native branch bakes both views' static shapes into the kernel and reads the scales as one dense row-major run, so a view with a runtime dimension or a strided scale view cannot instantiate it. Such callers take the dequant route, which indexes both views through their own strides.

comptime values​

DECODE_MAX_M​

comptime DECODE_MAX_M = 256

MX_BLOCK_SIZE​

comptime MX_BLOCK_SIZE = 32

Functions​

Was this page helpful?