IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

BlockScaledMatmulAMD

struct BlockScaledMatmulAMD[BM: Int = Int(128), BN: Int = Int(128), BK_ELEMS: Int = Int(128), WM: Int = Int(64), WN: Int = Int(64), MMA_M: Int = Int(16), MMA_N: Int = Int(16), MMA_K: Int = Int(128), num_stages: Int = Int(1), matrix_format: CDNA4F8F6F4MatrixFormat = CDNA4F8F6F4MatrixFormat.FLOAT4_E2M1, b_matrix_format: CDNA4F8F6F4MatrixFormat = matrix_format]

Native block-scaled matmul for AMD CDNA4.

Uses cdna4_block_scaled_mfma with independent A/B operand encodings. Single-buffer pipeline, or at num_stages == 2 an LDS ping-pong with a depth-2 B-fragment ring. SMEM is plain row-major (no blocked-product), with a conditional XOR swizzle (BlockScaledMmaOp.use_smem_swizzle) removing LDS bank conflicts on the A/B fragment read/write when the tile config qualifies.

Parameters​

  • ​BM (Int): Block tile rows (output M per block). Default 128.
  • ​BN (Int): Block tile cols (output N per block). Default 128.
  • ​BK_ELEMS (Int): Block tile K in logical elements. Default 128.
  • ​WM (Int): Warp tile rows. BM must be divisible by WM. Default 64.
  • ​WN (Int): Warp tile cols. BN must be divisible by WN. Default 64.
  • ​MMA_M (Int): MFMA tile rows. WM must be divisible by MMA_M. Default 16.
  • ​MMA_N (Int): MFMA tile cols. WN must be divisible by MMA_N. Default 16.
  • ​MMA_K (Int): MFMA K-depth in logical elements. Default 128.
  • ​num_stages (Int): SMEM pipeline depth. 1 is the single-buffer schedule; 2 ping-pongs LDS (even tiles_per_split, VGPR-bound occupancy).
  • ​matrix_format (CDNA4F8F6F4MatrixFormat): Encoding of A (FP4 E2M1 by default). BK_ELEMS counts logical elements; payload and LDS widths follow each operand's own encoding.
  • ​b_matrix_format (CDNA4F8F6F4MatrixFormat): Encoding of B. Defaults to matrix_format (A).

Implemented traits​

AnyType, Deinitable, Movable

comptime members​

a_bits​

comptime a_bits = matrix_format.bits_per_element()

A_BK_BYTES​

comptime A_BK_BYTES = (Int((mul matrix_format.bits_per_element(), BK_ELEMS)) // Int(8))

a_packed_k_per_mma​

comptime a_packed_k_per_mma = (Int((mul matrix_format.bits_per_element(), MMA_K)) // Int(8))

A_SMEM_ROW_BYTES​

comptime A_SMEM_ROW_BYTES = _smem_row_bytes[(Int((mul matrix_format.bits_per_element(), BK_ELEMS)) // Int(8))]()

b_bits​

comptime b_bits = b_matrix_format.bits_per_element()

B_BK_BYTES​

comptime B_BK_BYTES = (Int((mul b_matrix_format.bits_per_element(), BK_ELEMS)) // Int(8))

b_packed_k_per_mma​

comptime b_packed_k_per_mma = (Int((mul b_matrix_format.bits_per_element(), MMA_K)) // Int(8))

B_SMEM_ROW_BYTES​

comptime B_SMEM_ROW_BYTES = _smem_row_bytes[(Int((mul b_matrix_format.bits_per_element(), BK_ELEMS)) // Int(8))]()

bits_per_element​

comptime bits_per_element = BlockScaledMatmulAMD[BM, BN, BK_ELEMS, WM, WN, MMA_M, MMA_N, MMA_K, num_stages, matrix_format, b_matrix_format].a_bits

BK_BYTES​

comptime BK_BYTES = BlockScaledMatmulAMD[BM, BN, BK_ELEMS, WM, WN, MMA_M, MMA_N, MMA_K, num_stages, matrix_format, b_matrix_format].A_BK_BYTES

c_frag_size​

comptime c_frag_size = ((MMA_M * MMA_N) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)

k_tile_size​

comptime k_tile_size = BlockScaledMatmulAMD[BM, BN, BK_ELEMS, WM, WN, MMA_M, MMA_N, MMA_K, num_stages, matrix_format, b_matrix_format].BK_BYTES

lane_bytes​

comptime lane_bytes = (Int((mul matrix_format.bits_per_element(), 32)) // Int(8))

num_k_tiles​

comptime num_k_tiles = ((Int((mul matrix_format.bits_per_element(), BK_ELEMS)) // Int(8)) // (Int((mul matrix_format.bits_per_element(), MMA_K)) // Int(8)))

num_m_mmas​

comptime num_m_mmas = (WM // MMA_M)

num_n_mmas​

comptime num_n_mmas = (WN // MMA_N)

num_threads​

comptime num_threads = (BlockScaledMatmulAMD[BM, BN, BK_ELEMS, WM, WN, MMA_M, MMA_N, MMA_K, num_stages, matrix_format, b_matrix_format].num_warps * DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)

num_warps​

comptime num_warps = (BlockScaledMatmulAMD[BM, BN, BK_ELEMS, WM, WN, MMA_M, MMA_N, MMA_K, num_stages, matrix_format, b_matrix_format].num_warps_m * BlockScaledMatmulAMD[BM, BN, BK_ELEMS, WM, WN, MMA_M, MMA_N, MMA_K, num_stages, matrix_format, b_matrix_format].num_warps_n)

num_warps_m​

comptime num_warps_m = (BM // WM)

num_warps_n​

comptime num_warps_n = (BN // WN)

packed_k_per_mma​

comptime packed_k_per_mma = BlockScaledMatmulAMD[BM, BN, BK_ELEMS, WM, WN, MMA_M, MMA_N, MMA_K, num_stages, matrix_format, b_matrix_format].a_packed_k_per_mma

scales_per_mma​

comptime scales_per_mma = (MMA_K // Int(32))

simd_width​

comptime simd_width = simd_width_of[DType.uint8]()

Methods​

run​

static def run[out_dtype: DType, c_layout: TensorLayout, a_layout: TensorLayout, b_layout: TensorLayout, sfa_layout: TensorLayout, sfb_layout: TensorLayout, c_engine: TensorEngine, a_engine: TensorEngine, b_engine: TensorEngine, sfa_engine: TensorEngine, sfb_engine: TensorEngine, num_splits: Int = Int(1)](c: TileTensor[out_dtype, c_layout, MutAnyOrigin, Engine=c_engine], a: TileTensor[.uint8, a_layout, ImmutAnyOrigin, Engine=a_engine], b: TileTensor[.uint8, b_layout, ImmutAnyOrigin, Engine=b_engine], sfa: TileTensor[.float8_e8m0fnu, sfa_layout, ImmutAnyOrigin, Engine=sfa_engine], sfb: TileTensor[.float8_e8m0fnu, sfb_layout, ImmutAnyOrigin, Engine=sfb_engine])

Block-scaled GEMM kernel with SMEM pipeline.

A and B are raw bytes in the matrix_format and b_matrix_format encodings, which may differ (E4M3 A against E2M1 B), so each operand has its own byte width for the same logical K.

With num_splits > 1 this is the inter-block split-K body: each block_idx.z slice accumulates one disjoint K-band into its own [M, N] region of a stacked (num_splits * M, N) float32 workspace (out_dtype is float32 in that mode). A separate reduce kernel sums the num_splits partials and casts to the real output dtype. num_splits == 1 is byte-identical to the no-split path (split_id == 0, full K range, zero offset).

Parameters:

  • ​out_dtype (DType): Element type of the output tensor c; must be float32 when num_splits > 1.
  • ​c_layout (TensorLayout): Compile-time layout of the output tensor c.
  • ​a_layout (TensorLayout): Compile-time layout of the A operand.
  • ​b_layout (TensorLayout): Compile-time layout of the B operand.
  • ​sfa_layout (TensorLayout): Compile-time layout of the A scales tensor sfa.
  • ​sfb_layout (TensorLayout): Compile-time layout of the B scales tensor sfb.
  • ​c_engine (TensorEngine): Engine of the output tensor c.
  • ​a_engine (TensorEngine): Engine of the A operand.
  • ​b_engine (TensorEngine): Engine of the B operand.
  • ​sfa_engine (TensorEngine): Engine of the A scales tensor sfa.
  • ​sfb_engine (TensorEngine): Engine of the B scales tensor sfb.
  • ​num_splits (Int): Number of disjoint K-bands the K dimension is partitioned into (defaults to 1, no split).

Args:

run_with_epilogue_fn​

static def run_with_epilogue_fn[out_dtype: DType, c_layout: TensorLayout, a_layout: TensorLayout, b_layout: TensorLayout, sfa_layout: TensorLayout, sfb_layout: TensorLayout, c_engine: TensorEngine, a_engine: TensorEngine, b_engine: TensorEngine, sfa_engine: TensorEngine, sfb_engine: TensorEngine, EpilogueFnType: ElementwiseEpilogueFn](c: TileTensor[out_dtype, c_layout, MutAnyOrigin, Engine=c_engine], a: TileTensor[.uint8, a_layout, ImmutAnyOrigin, Engine=a_engine], b: TileTensor[.uint8, b_layout, ImmutAnyOrigin, Engine=b_engine], sfa: TileTensor[.float8_e8m0fnu, sfa_layout, ImmutAnyOrigin, Engine=sfa_engine], sfb: TileTensor[.float8_e8m0fnu, sfb_layout, ImmutAnyOrigin, Engine=sfb_engine], epilogue_fn: EpilogueFnType)

Same as run without split-K, storing the output through epilogue_fn.

Launch with host_args=(epilogue_fn,). Requires MMA_M == 16.

Parameters:

  • ​out_dtype (DType): Element type of the output tensor c.
  • ​c_layout (TensorLayout): Compile-time layout of the output tensor c.
  • ​a_layout (TensorLayout): Compile-time layout of the A operand.
  • ​b_layout (TensorLayout): Compile-time layout of the B operand.
  • ​sfa_layout (TensorLayout): Compile-time layout of the A scales tensor sfa.
  • ​sfb_layout (TensorLayout): Compile-time layout of the B scales tensor sfb.
  • ​c_engine (TensorEngine): Engine of the output tensor c.
  • ​a_engine (TensorEngine): Engine of the A operand.
  • ​b_engine (TensorEngine): Engine of the B operand.
  • ​sfa_engine (TensorEngine): Engine of the A scales tensor sfa.
  • ​sfb_engine (TensorEngine): Engine of the B scales tensor sfb.
  • ​EpilogueFnType (ElementwiseEpilogueFn): Type of epilogue_fn.

Args:

run_with_compute_fn​

static def run_with_compute_fn[out_dtype: DType, c_layout: TensorLayout, a_layout: TensorLayout, b_layout: TensorLayout, sfa_layout: TensorLayout, sfb_layout: TensorLayout, c_engine: TensorEngine, a_engine: TensorEngine, b_engine: TensorEngine, sfa_engine: TensorEngine, sfb_engine: TensorEngine, ComputeFnType: ElementwiseComputeFn](c: TileTensor[out_dtype, c_layout, MutAnyOrigin, Engine=c_engine], a: TileTensor[.uint8, a_layout, ImmutAnyOrigin, Engine=a_engine], b: TileTensor[.uint8, b_layout, ImmutAnyOrigin, Engine=b_engine], sfa: TileTensor[.float8_e8m0fnu, sfa_layout, ImmutAnyOrigin, Engine=sfa_engine], sfb: TileTensor[.float8_e8m0fnu, sfb_layout, ImmutAnyOrigin, Engine=sfb_engine], compute_fn: ComputeFnType)

Same as run without split-K, storing compute_fn(idx, value) into c.

Launch with host_args=(compute_fn,). Requires MMA_M == 16.

Parameters:

  • ​out_dtype (DType): Element type of the output tensor c.
  • ​c_layout (TensorLayout): Compile-time layout of the output tensor c.
  • ​a_layout (TensorLayout): Compile-time layout of the A operand.
  • ​b_layout (TensorLayout): Compile-time layout of the B operand.
  • ​sfa_layout (TensorLayout): Compile-time layout of the A scales tensor sfa.
  • ​sfb_layout (TensorLayout): Compile-time layout of the B scales tensor sfb.
  • ​c_engine (TensorEngine): Engine of the output tensor c.
  • ​a_engine (TensorEngine): Engine of the A operand.
  • ​b_engine (TensorEngine): Engine of the B operand.
  • ​sfa_engine (TensorEngine): Engine of the A scales tensor sfa.
  • ​sfb_engine (TensorEngine): Engine of the B scales tensor sfb.
  • ​ComputeFnType (ElementwiseComputeFn): Type of compute_fn.

Args:

Was this page helpful?