IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

blockwise_scaled_fp8_matmul_amd_batched_kernel

def blockwise_scaled_fp8_matmul_amd_batched_kernel[c_type: DType, a_type: DType, b_type: DType, a_scales_type: DType, b_scales_type: DType, accum_type: DType, c_layout: TensorLayout, a_layout: TensorLayout, b_layout: TensorLayout, a_scale_layout: TensorLayout, b_scale_layout: TensorLayout, *, BM: Int, BN: Int, BK: Int, WM: Int, WN: Int, MMA_M: Int, MMA_N: Int, MMA_K: Int, N_SCALE: Int, K_SCALE: Int, N: Int, K: Int](c: TileTensor[c_type, c_layout, MutAnyOrigin], a: TileTensor[a_type, a_layout, ImmutAnyOrigin], b: TileTensor[b_type, b_layout, ImmutAnyOrigin], a_scales: TileTensor[a_scales_type, a_scale_layout, ImmutAnyOrigin], b_scales: TileTensor[b_scales_type, b_scale_layout, ImmutAnyOrigin])

One launch for the whole batch of a blockwise-scaled FP8 dense matmul.

Mirrors the SM100 _bmm_sm100_blockwise_scaled_fp8_kernel structure on AMD: block_idx.z selects the batch element, block_idx.y the M-tile, block_idx.x the N-tile, so a batched [B, M, N] problem is one kernel launch of grid (ceildiv(N,BN), ceildiv(M,BM), B) instead of B per- batch launches. The per-tile MMA body is the existing dense body, reused by calling _blockwise_scaled_fp8_matmul_amd_tile with per-batch 2D views.

c/a carry the compile-time trailing extent (N/K) into the tile body as a static last dimension, so the tile body's comptime N = type_of(c).static_shape[1] and comptime K = type_of(a).static_shape[1] still bind. Rows and row strides stay runtime: the MLA absorb output is a strided per-head slab into the latent, so the rank-3 strides are preserved per-batch via ptr_at_offset(Coord(batch, 0, 0)) (which respects the rank-3 strides) and the 2D layout carries the row stride.

Was this page helpful?