For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
blockwise_scaled_fp8_grouped_matmul_amd
Tiled MFMA blockwise-scaled FP8 grouped (MoE) matmul for AMD gfx950.
Target: AMD MI350/MI355 (gfx950), MFMA 16x16x128 FP8 (E4M3 OCP).
This is the expert-dispatched variant of the dense
blockwise_scaled_fp8_matmul_amd. It carries the bulk of a mixed-FP8
MoE model's FLOPs (GLM gate_up / down projections). The grouped
expert/tile dispatch shell (one block per (expert, n_tile) striding
over M-tiles, with the expert selected by block_idx.z via a_offsets /
expert_ids) follows the MXFP4 grouped kernel
block_scaled_grouped_matmul_amd; the per-tile MMA body is the dense
kernel's E4M3 + fp32-per-128-K-slab-promotion body, reused verbatim via
blockwise_scaled_fp8_matmul_amd_tile. The M-tile stride loop keeps
grid.y a packing hint rather than a coverage bound, so grid.y can be
sized from a capture-safe bound instead of the graph-capture
max_num_tokens_per_expert -- at decode that constant (16384 in
production) made ~99.6% of the one-wave CTAs exit at the bounds check.
Scale-index and expert-dispatch semantics mirror
naive_blockwise_scaled_fp8_grouped_matmul (linalg/fp8_quantization.mojo)
at granularity (m, n, k) = (1, 128, 128):
C[e][m, n] = sum_k A[m_global, k] * B[e, n, k] * a_scale[k//128, m_global] * b_scale[e, n//128, k//128]
where m_global = a_offsets[slot] + m_local and e = expert_ids[slot].
a_scales is token-minor [K//128, total_tokens], so the reused body is
handed the full a_scales plus this expert's a_start_row column base;
b_scales is sliced per expert to [N//128, K//128], matching the dense
body's 2D b_scales exactly.
Functionsโ
- โ
blockwise_scaled_fp8_grouped_matmul_amd: Enqueues the tiled blockwise-scaled FP8 grouped (MoE) matmul for gfx950. - โ
blockwise_scaled_fp8_grouped_matmul_amd_kernel: Computes the(expert, n_tile)output-tile column, striding over M-tiles, for a grouped matmul.