For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
BlockScaledMatmulAMD
struct BlockScaledMatmulAMD[BM: Int = Int(128), BN: Int = Int(128), BK_ELEMS: Int = Int(128), WM: Int = Int(64), WN: Int = Int(64), MMA_M: Int = Int(16), MMA_N: Int = Int(16), MMA_K: Int = Int(128), num_stages: Int = Int(1), matrix_format: CDNA4F8F6F4MatrixFormat = CDNA4F8F6F4MatrixFormat.FLOAT4_E2M1, b_matrix_format: CDNA4F8F6F4MatrixFormat = matrix_format]
Native block-scaled matmul for AMD CDNA4.
Uses cdna4_block_scaled_mfma with independent A/B operand encodings.
Single-buffer pipeline, or at num_stages == 2 an LDS ping-pong with a
depth-2 B-fragment ring.
SMEM is plain row-major (no blocked-product), with a conditional XOR
swizzle (BlockScaledMmaOp.use_smem_swizzle) removing LDS bank
conflicts on the A/B fragment read/write when the tile config qualifies.
Parameters
- BM (
Int): Block tile rows (output M per block). Default 128. - BN (
Int): Block tile cols (output N per block). Default 128. - BK_ELEMS (
Int): Block tile K in logical elements. Default 128. - WM (
Int): Warp tile rows. BM must be divisible by WM. Default 64. - WN (
Int): Warp tile cols. BN must be divisible by WN. Default 64. - MMA_M (
Int): MFMA tile rows. WM must be divisible by MMA_M. Default 16. - MMA_N (
Int): MFMA tile cols. WN must be divisible by MMA_N. Default 16. - MMA_K (
Int): MFMA K-depth in logical elements. Default 128. - num_stages (
Int): SMEM pipeline depth. 1 is the single-buffer schedule; 2 ping-pongs LDS (eventiles_per_split, VGPR-bound occupancy). - matrix_format (
CDNA4F8F6F4MatrixFormat): Encoding of A (FP4 E2M1 by default).BK_ELEMScounts logical elements; payload and LDS widths follow each operand's own encoding. - b_matrix_format (
CDNA4F8F6F4MatrixFormat): Encoding of B. Defaults tomatrix_format(A).
Implemented traits
comptime members
a_bits
comptime a_bits = matrix_format.bits_per_element()
A_BK_BYTES
comptime A_BK_BYTES = (Int((mul matrix_format.bits_per_element(), BK_ELEMS)) // Int(8))
a_packed_k_per_mma
comptime a_packed_k_per_mma = (Int((mul matrix_format.bits_per_element(), MMA_K)) // Int(8))
A_SMEM_ROW_BYTES
comptime A_SMEM_ROW_BYTES = _smem_row_bytes[(Int((mul matrix_format.bits_per_element(), BK_ELEMS)) // Int(8))]()
b_bits
comptime b_bits = b_matrix_format.bits_per_element()
B_BK_BYTES
comptime B_BK_BYTES = (Int((mul b_matrix_format.bits_per_element(), BK_ELEMS)) // Int(8))
b_packed_k_per_mma
comptime b_packed_k_per_mma = (Int((mul b_matrix_format.bits_per_element(), MMA_K)) // Int(8))
B_SMEM_ROW_BYTES
comptime B_SMEM_ROW_BYTES = _smem_row_bytes[(Int((mul b_matrix_format.bits_per_element(), BK_ELEMS)) // Int(8))]()
bits_per_element
comptime bits_per_element = BlockScaledMatmulAMD[BM, BN, BK_ELEMS, WM, WN, MMA_M, MMA_N, MMA_K, num_stages, matrix_format, b_matrix_format].a_bits
BK_BYTES
comptime BK_BYTES = BlockScaledMatmulAMD[BM, BN, BK_ELEMS, WM, WN, MMA_M, MMA_N, MMA_K, num_stages, matrix_format, b_matrix_format].A_BK_BYTES
c_frag_size
comptime c_frag_size = ((MMA_M * MMA_N) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)
k_tile_size
comptime k_tile_size = BlockScaledMatmulAMD[BM, BN, BK_ELEMS, WM, WN, MMA_M, MMA_N, MMA_K, num_stages, matrix_format, b_matrix_format].BK_BYTES
lane_bytes
comptime lane_bytes = (Int((mul matrix_format.bits_per_element(), 32)) // Int(8))
num_k_tiles
comptime num_k_tiles = ((Int((mul matrix_format.bits_per_element(), BK_ELEMS)) // Int(8)) // (Int((mul matrix_format.bits_per_element(), MMA_K)) // Int(8)))
num_m_mmas
comptime num_m_mmas = (WM // MMA_M)
num_n_mmas
comptime num_n_mmas = (WN // MMA_N)
num_threads
comptime num_threads = (BlockScaledMatmulAMD[BM, BN, BK_ELEMS, WM, WN, MMA_M, MMA_N, MMA_K, num_stages, matrix_format, b_matrix_format].num_warps * DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)
num_warps
comptime num_warps = (BlockScaledMatmulAMD[BM, BN, BK_ELEMS, WM, WN, MMA_M, MMA_N, MMA_K, num_stages, matrix_format, b_matrix_format].num_warps_m * BlockScaledMatmulAMD[BM, BN, BK_ELEMS, WM, WN, MMA_M, MMA_N, MMA_K, num_stages, matrix_format, b_matrix_format].num_warps_n)
num_warps_m
comptime num_warps_m = (BM // WM)
num_warps_n
comptime num_warps_n = (BN // WN)
packed_k_per_mma
comptime packed_k_per_mma = BlockScaledMatmulAMD[BM, BN, BK_ELEMS, WM, WN, MMA_M, MMA_N, MMA_K, num_stages, matrix_format, b_matrix_format].a_packed_k_per_mma
scales_per_mma
comptime scales_per_mma = (MMA_K // Int(32))
simd_width
comptime simd_width = simd_width_of[DType.uint8]()
Methods
run
static def run[out_dtype: DType, c_layout: TensorLayout, a_layout: TensorLayout, b_layout: TensorLayout, sfa_layout: TensorLayout, sfb_layout: TensorLayout, c_engine: TensorEngine, a_engine: TensorEngine, b_engine: TensorEngine, sfa_engine: TensorEngine, sfb_engine: TensorEngine, num_splits: Int = Int(1)](c: TileTensor[out_dtype, c_layout, MutAnyOrigin, Engine=c_engine], a: TileTensor[.uint8, a_layout, ImmutAnyOrigin, Engine=a_engine], b: TileTensor[.uint8, b_layout, ImmutAnyOrigin, Engine=b_engine], sfa: TileTensor[.float8_e8m0fnu, sfa_layout, ImmutAnyOrigin, Engine=sfa_engine], sfb: TileTensor[.float8_e8m0fnu, sfb_layout, ImmutAnyOrigin, Engine=sfb_engine])
Block-scaled GEMM kernel with SMEM pipeline.
A and B are raw bytes in the matrix_format and b_matrix_format
encodings, which may differ (E4M3 A against E2M1 B), so each operand
has its own byte width for the same logical K.
With num_splits > 1 this is the inter-block split-K body: each
block_idx.z slice accumulates one disjoint K-band into its own
[M, N] region of a stacked (num_splits * M, N) float32
workspace (out_dtype is float32 in that mode). A separate
reduce kernel sums the num_splits partials and casts to the
real output dtype. num_splits == 1 is byte-identical to the
no-split path (split_id == 0, full K range, zero offset).
Parameters:
- out_dtype (
DType): Element type of the output tensorc; must befloat32whennum_splits > 1. - c_layout (
TensorLayout): Compile-time layout of the output tensorc. - a_layout (
TensorLayout): Compile-time layout of the A operand. - b_layout (
TensorLayout): Compile-time layout of the B operand. - sfa_layout (
TensorLayout): Compile-time layout of the A scales tensorsfa. - sfb_layout (
TensorLayout): Compile-time layout of the B scales tensorsfb. - c_engine (
TensorEngine): Engine of the output tensorc. - a_engine (
TensorEngine): Engine of the A operand. - b_engine (
TensorEngine): Engine of the B operand. - sfa_engine (
TensorEngine): Engine of the A scales tensorsfa. - sfb_engine (
TensorEngine): Engine of the B scales tensorsfb. - num_splits (
Int): Number of disjoint K-bands the K dimension is partitioned into (defaults to 1, no split).
Args:
- c (
TileTensor[out_dtype, c_layout, MutAnyOrigin, Engine=c_engine]): Output matrix[M, N]of dtypeout_dtype; in split-K mode a stacked(num_splits * M, N)float32 workspace. - a (
TileTensor[.uint8, a_layout, ImmutAnyOrigin, Engine=a_engine]): A operand[M, K*a_bits//8]as raw uint8 bytes in thematrix_formatencoding (K//2at MXFP4,Kat MXFP8). - b (
TileTensor[.uint8, b_layout, ImmutAnyOrigin, Engine=b_engine]): Transposed B operand[N, K*b_bits//8]as raw uint8 bytes in theb_matrix_formatencoding. - sfa (
TileTensor[.float8_e8m0fnu, sfa_layout, ImmutAnyOrigin, Engine=sfa_engine]): A block scales[M, K//32]asfloat8_e8m0fnu, one scale per 32 logical elements. - sfb (
TileTensor[.float8_e8m0fnu, sfb_layout, ImmutAnyOrigin, Engine=sfb_engine]): B block scales[N, K//32]asfloat8_e8m0fnu, one scale per 32 logical elements.
run_with_epilogue_fn
static def run_with_epilogue_fn[out_dtype: DType, c_layout: TensorLayout, a_layout: TensorLayout, b_layout: TensorLayout, sfa_layout: TensorLayout, sfb_layout: TensorLayout, c_engine: TensorEngine, a_engine: TensorEngine, b_engine: TensorEngine, sfa_engine: TensorEngine, sfb_engine: TensorEngine, EpilogueFnType: ElementwiseEpilogueFn](c: TileTensor[out_dtype, c_layout, MutAnyOrigin, Engine=c_engine], a: TileTensor[.uint8, a_layout, ImmutAnyOrigin, Engine=a_engine], b: TileTensor[.uint8, b_layout, ImmutAnyOrigin, Engine=b_engine], sfa: TileTensor[.float8_e8m0fnu, sfa_layout, ImmutAnyOrigin, Engine=sfa_engine], sfb: TileTensor[.float8_e8m0fnu, sfb_layout, ImmutAnyOrigin, Engine=sfb_engine], epilogue_fn: EpilogueFnType)
Same as run without split-K, storing the output through epilogue_fn.
Launch with host_args=(epilogue_fn,).
Requires MMA_M == 16.
Parameters:
- out_dtype (
DType): Element type of the output tensorc. - c_layout (
TensorLayout): Compile-time layout of the output tensorc. - a_layout (
TensorLayout): Compile-time layout of the A operand. - b_layout (
TensorLayout): Compile-time layout of the B operand. - sfa_layout (
TensorLayout): Compile-time layout of the A scales tensorsfa. - sfb_layout (
TensorLayout): Compile-time layout of the B scales tensorsfb. - c_engine (
TensorEngine): Engine of the output tensorc. - a_engine (
TensorEngine): Engine of the A operand. - b_engine (
TensorEngine): Engine of the B operand. - sfa_engine (
TensorEngine): Engine of the A scales tensorsfa. - sfb_engine (
TensorEngine): Engine of the B scales tensorsfb. - EpilogueFnType (
ElementwiseEpilogueFn): Type ofepilogue_fn.
Args:
- c (
TileTensor[out_dtype, c_layout, MutAnyOrigin, Engine=c_engine]): Output matrix[M, N]. Only its shape is read. - a (
TileTensor[.uint8, a_layout, ImmutAnyOrigin, Engine=a_engine]): A operand, as inrun. - b (
TileTensor[.uint8, b_layout, ImmutAnyOrigin, Engine=b_engine]): Transposed B operand, as inrun. - sfa (
TileTensor[.float8_e8m0fnu, sfa_layout, ImmutAnyOrigin, Engine=sfa_engine]): A block scales, as inrun. - sfb (
TileTensor[.float8_e8m0fnu, sfb_layout, ImmutAnyOrigin, Engine=sfb_engine]): B block scales, as inrun. - epilogue_fn (
EpilogueFnType): Stores each output fragment at its global(m, n).
run_with_compute_fn
static def run_with_compute_fn[out_dtype: DType, c_layout: TensorLayout, a_layout: TensorLayout, b_layout: TensorLayout, sfa_layout: TensorLayout, sfb_layout: TensorLayout, c_engine: TensorEngine, a_engine: TensorEngine, b_engine: TensorEngine, sfa_engine: TensorEngine, sfb_engine: TensorEngine, ComputeFnType: ElementwiseComputeFn](c: TileTensor[out_dtype, c_layout, MutAnyOrigin, Engine=c_engine], a: TileTensor[.uint8, a_layout, ImmutAnyOrigin, Engine=a_engine], b: TileTensor[.uint8, b_layout, ImmutAnyOrigin, Engine=b_engine], sfa: TileTensor[.float8_e8m0fnu, sfa_layout, ImmutAnyOrigin, Engine=sfa_engine], sfb: TileTensor[.float8_e8m0fnu, sfb_layout, ImmutAnyOrigin, Engine=sfb_engine], compute_fn: ComputeFnType)
Same as run without split-K, storing compute_fn(idx, value) into c.
Launch with host_args=(compute_fn,).
Requires MMA_M == 16.
Parameters:
- out_dtype (
DType): Element type of the output tensorc. - c_layout (
TensorLayout): Compile-time layout of the output tensorc. - a_layout (
TensorLayout): Compile-time layout of the A operand. - b_layout (
TensorLayout): Compile-time layout of the B operand. - sfa_layout (
TensorLayout): Compile-time layout of the A scales tensorsfa. - sfb_layout (
TensorLayout): Compile-time layout of the B scales tensorsfb. - c_engine (
TensorEngine): Engine of the output tensorc. - a_engine (
TensorEngine): Engine of the A operand. - b_engine (
TensorEngine): Engine of the B operand. - sfa_engine (
TensorEngine): Engine of the A scales tensorsfa. - sfb_engine (
TensorEngine): Engine of the B scales tensorsfb. - ComputeFnType (
ElementwiseComputeFn): Type ofcompute_fn.
Args:
- c (
TileTensor[out_dtype, c_layout, MutAnyOrigin, Engine=c_engine]): Output matrix[M, N]. - a (
TileTensor[.uint8, a_layout, ImmutAnyOrigin, Engine=a_engine]): A operand, as inrun. - b (
TileTensor[.uint8, b_layout, ImmutAnyOrigin, Engine=b_engine]): Transposed B operand, as inrun. - sfa (
TileTensor[.float8_e8m0fnu, sfa_layout, ImmutAnyOrigin, Engine=sfa_engine]): A block scales, as inrun. - sfb (
TileTensor[.float8_e8m0fnu, sfb_layout, ImmutAnyOrigin, Engine=sfb_engine]): B block scales, as inrun. - compute_fn (
ComputeFnType): Maps each output index and fragment to the value to store.