IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

blockwise_scaled_fp8_grouped_matmul_amd

Tiled MFMA blockwise-scaled FP8 grouped (MoE) matmul for AMD gfx950.

Target: AMD MI350/MI355 (gfx950), MFMA 16x16x128 FP8 (E4M3 OCP).

This is the expert-dispatched variant of the dense blockwise_scaled_fp8_matmul_amd. It carries the bulk of a mixed-FP8 MoE model's FLOPs (GLM gate_up / down projections). The grouped expert/tile dispatch shell (one block per (expert, n_tile) striding over M-tiles, with the expert selected by block_idx.z via a_offsets / expert_ids) follows the MXFP4 grouped kernel block_scaled_grouped_matmul_amd; the per-tile MMA body is the dense kernel's E4M3 + fp32-per-128-K-slab-promotion body, reused verbatim via blockwise_scaled_fp8_matmul_amd_tile. The M-tile stride loop keeps grid.y a packing hint rather than a coverage bound, so grid.y can be sized from a capture-safe bound instead of the graph-capture max_num_tokens_per_expert -- at decode that constant (16384 in production) made ~99.6% of the one-wave CTAs exit at the bounds check.

Scale-index and expert-dispatch semantics mirror naive_blockwise_scaled_fp8_grouped_matmul (linalg/fp8_quantization.mojo) at granularity (m, n, k) = (1, 128, 128):

C[e][m, n] = sum_k A[m_global, k] * B[e, n, k] * a_scale[k//128, m_global] * b_scale[e, n//128, k//128]

where m_global = a_offsets[slot] + m_local and e = expert_ids[slot]. a_scales is token-minor [K//128, total_tokens], so the reused body is handed the full a_scales plus this expert's a_start_row column base; b_scales is sliced per expert to [N//128, K//128], matching the dense body's 2D b_scales exactly.

Functionsโ€‹

Was this page helpful?