IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

blockwise_scaled_fp8_matmul_amd

Tiled MFMA blockwise-scaled FP8 dense matmul for AMD gfx950 (CDNA4).

Target: AMD MI350/MI355 (gfx950), MFMA 16x16x128 FP8 (E4M3 OCP), or 16x16x32 when the K scale block is 64 wide.

Computes `C[m, n] = sum_k A[m, k] * B[n, k] * a_scale[k//K_SCALE, m]

  • b_scale[n//N_SCALE, k//K_SCALE]for blockwise-scaled FP8 with granularity(m, n, k) = (1, 128, 128)or(1, 64, 64)andtranspose_b=True(B is[N, K], K-major). This mirrors naive_blockwise_scaled_fp8_matmul_kernel (linalg/fp8_quantization.mojo`) exactly, but tiled.

The one hard idea is fp32 per-K-scale-block scale promotion. A single epilogue cannot apply the scales because K spans many scale blocks, each with its own scale. So the K loop steps by BK == 128 (one DRAM->LDS slab holding BK // K_SCALE scale blocks), MFMA-accumulates each block into a temporary fp32 fragment blk_acc (reset per block), then folds promoted += blk_acc * a_scale * b_scale into a persistent fp32 accumulator before the next block. An MFMA cannot straddle two scale blocks, so a 128-wide block is one 16x16x128 MFMA and a 64-wide block is two 16x16x32 MFMAs (blockwise_fp8_mma_k); the load pipeline is the same either way.

Data movement (DRAM->regs->LDS->fragments) reuses the same proven RegTileLoader / RegTileWriterLDS / MmaOp primitives as AMDMatmul (amd_matmul.mojo). The C-fragment lane->(m, n) map used by the scale promotion and the output store matches AMDMatmul.run's epilogue: for a 16x16 MFMA each lane owns one row (lane % 16) and c_frag_size contiguous columns starting at (lane // 16) * c_frag_size. Because a lane's whole fragment shares one row m, a_scale is a single scalar per (lane, m_mma); the 4 columns sit in one N scale block, so b_scale is a single scalar per (lane, n_mma).

The per-output-tile body lives in blockwise_scaled_fp8_matmul_amd_tile so the grouped (MoE) variant in blockwise_scaled_fp8_grouped_matmul_amd can reuse the identical slab-promotion body and lane->(m, n) scale map -- the correctness-critical part is authored once, not copied.

comptime values​

LARGE_M_MIN​

comptime LARGE_M_MIN = 65

Rows from which the default configuration hands a matmul to the large-M kernel (blockwise_scaled_fp8_matmul_amd_large): just past the split-K decode band (M <= 64). Measured faster on MI355X at every M from 65 to 16384 on the GLM-5.3 shapes and on other models' (N, K), K = 512 even.

Functions​

Was this page helpful?