For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
AMDMatmul
struct AMDMatmul[a_type: DType, b_type: DType, c_type: DType, transpose_b: Bool, config: MatmulConfig[a_type, b_type, c_type, transpose_b]]
Pure TileTensor structured matmul for AMD GPUs.
Schedule-driven single-buffer pipeline. All data movement uses TileTensor.
Parameters
- a_type (
DType): Element type of the A input matrix. - b_type (
DType): Element type of the B input matrix. - c_type (
DType): Element type of the C output matrix. - transpose_b (
Bool): Whether B is stored transposed as[N, K]; must beTrue. - config (
MatmulConfig[a_type, b_type, c_type, transpose_b]): Tile and warp shapes, MMA shape, and thread count for the kernel.
Implemented traits
comptime members
accum_type
comptime accum_type = get_accum_type[a_type]()
BK
comptime BK = ((load_from_mem config.block_tile_shape.__getitem_param__[Int(2)]()) * config)
BM
comptime BM = (load_from_mem config.block_tile_shape.__getitem_param__[Int(0)]())
BN
comptime BN = (load_from_mem config.block_tile_shape.__getitem_param__[Int(1)]())
c_frag_size
comptime c_frag_size = ((AMDMatmul[a_type, b_type, c_type, transpose_b, config].MMA_M * AMDMatmul[a_type, b_type, c_type, transpose_b, config].MMA_N) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)
frag_size
comptime frag_size = ((AMDMatmul[a_type, b_type, c_type, transpose_b, config].MMA_M * AMDMatmul[a_type, b_type, c_type, transpose_b, config].MMA_K) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)
k_group_size
comptime k_group_size = (simd_width_of[a_type]() // (Int((mul (load_from_mem config.mma_shape.__getitem_param__[Int(0)]()), (load_from_mem config.mma_shape.__getitem_param__[Int(2)]()))) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size))
k_tile_size
comptime k_tile_size = (AMDMatmul[a_type, b_type, c_type, transpose_b, config].MMA_K * (simd_width_of[a_type]() // (Int((mul (load_from_mem config.mma_shape.__getitem_param__[Int(0)]()), (load_from_mem config.mma_shape.__getitem_param__[Int(2)]()))) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)))
MMA_K
comptime MMA_K = (load_from_mem config.mma_shape.__getitem_param__[Int(2)]())
MMA_M
comptime MMA_M = (load_from_mem config.mma_shape.__getitem_param__[Int(0)]())
MMA_N
comptime MMA_N = (load_from_mem config.mma_shape.__getitem_param__[Int(1)]())
num_k_mmas
comptime num_k_mmas = (AMDMatmul[a_type, b_type, c_type, transpose_b, config].WK // AMDMatmul[a_type, b_type, c_type, transpose_b, config].MMA_K)
num_k_tiles
comptime num_k_tiles = (AMDMatmul[a_type, b_type, c_type, transpose_b, config].WK // Int((mul (simd_width_of[a_type]() // (Int((mul (load_from_mem config.mma_shape.__getitem_param__[Int(0)]()), (load_from_mem config.mma_shape.__getitem_param__[Int(2)]()))) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)), (load_from_mem config.mma_shape.__getitem_param__[Int(2)]()))))
num_m_mmas
comptime num_m_mmas = (AMDMatmul[a_type, b_type, c_type, transpose_b, config].WM // AMDMatmul[a_type, b_type, c_type, transpose_b, config].MMA_M)
num_n_mmas
comptime num_n_mmas = (AMDMatmul[a_type, b_type, c_type, transpose_b, config].WN // AMDMatmul[a_type, b_type, c_type, transpose_b, config].MMA_N)
num_warps_k
comptime num_warps_k = (Int((mul (load_from_mem config.block_tile_shape.__getitem_param__[Int(2)]()), config.num_warp_k_partitions)) // AMDMatmul[a_type, b_type, c_type, transpose_b, config].WK)
num_warps_m
comptime num_warps_m = (AMDMatmul[a_type, b_type, c_type, transpose_b, config].BM // AMDMatmul[a_type, b_type, c_type, transpose_b, config].WM)
num_warps_n
comptime num_warps_n = (AMDMatmul[a_type, b_type, c_type, transpose_b, config].BN // AMDMatmul[a_type, b_type, c_type, transpose_b, config].WN)
simd_width
comptime simd_width = simd_width_of[a_type]()
WK
comptime WK = (load_from_mem config.warp_tile_shape.__getitem_param__[Int(2)]())
WM
comptime WM = (load_from_mem config.warp_tile_shape.__getitem_param__[Int(0)]())
WN
comptime WN = (load_from_mem config.warp_tile_shape.__getitem_param__[Int(1)]())
Methods
make_mma_swizzle
static def make_mma_swizzle() -> Swizzle
Swizzle for blocked-product SMEM layout (LDS bank conflict avoidance).
The blocked-product layout stores k-tile elements in contiguous blocks. The MMA distribute reads these in col_major[MMA_M, WARP_SIZE/MMA_M] order, giving WARP_SIZE/MMA_M vector columns per block. The swizzle XORs enough bits to spread those column groups across LDS banks.
Unlike the ping-pong make_mma_swizzle (element-space for row-major SMEM with base/shift derived from fragment bytes), this operates in the vector-index space of each blocked-product chunk (base=0, shift=1).
Returns:
Swizzle: Swizzle for bank-conflict-free blocked-product LDS access.
run
static def run[c_layout: TensorLayout, a_layout: TensorLayout, b_layout: TensorLayout, c_engine: TensorEngine, a_engine: TensorEngine, b_engine: TensorEngine](c: TileTensor[c_type, c_layout, MutAnyOrigin, Engine=c_engine], a: TileTensor[a_type, a_layout, ImmutAnyOrigin, Engine=a_engine], b: TileTensor[b_type, b_layout, ImmutAnyOrigin, Engine=b_engine])
TileTensor GEMM matching original kernel config exactly.
Uses StructuredMmaOp with per-k-tile load_frag/mma dispatch, original warp index order, and schedule-driven pipeline.
Parameters:
- c_layout (
TensorLayout): Tensor layout of the output C tile. - a_layout (
TensorLayout): Tensor layout of the input A tile. - b_layout (
TensorLayout): Tensor layout of the input B tile. - c_engine (
TensorEngine): Engine of the output C tile. - a_engine (
TensorEngine): Engine of the input A tile. - b_engine (
TensorEngine): Engine of the input B tile.
Args:
- c (
TileTensor[c_type, c_layout, MutAnyOrigin, Engine=c_engine]): Output tile of shape[M, N]accumulating the matmul result. - a (
TileTensor[a_type, a_layout, ImmutAnyOrigin, Engine=a_engine]): Input A tile of shape[M, K]in row-major block layout. - b (
TileTensor[b_type, b_layout, ImmutAnyOrigin, Engine=b_engine]): Input B tile of shape[N, K](transposed,transpose_bisTrue).
run_with_epilogue_fn
static def run_with_epilogue_fn[c_layout: TensorLayout, a_layout: TensorLayout, b_layout: TensorLayout, c_engine: TensorEngine, a_engine: TensorEngine, b_engine: TensorEngine, EpilogueFnType: ElementwiseEpilogueFn](c: TileTensor[c_type, c_layout, MutAnyOrigin, Engine=c_engine], a: TileTensor[a_type, a_layout, ImmutAnyOrigin, Engine=a_engine], b: TileTensor[b_type, b_layout, ImmutAnyOrigin, Engine=b_engine], epilogue_fn: EpilogueFnType)
Same as run, storing the output through epilogue_fn.
Launch with host_args=(epilogue_fn,).
Parameters:
- c_layout (
TensorLayout): Tensor layout of the output C tile. - a_layout (
TensorLayout): Tensor layout of the input A tile. - b_layout (
TensorLayout): Tensor layout of the input B tile. - c_engine (
TensorEngine): Engine of the output C tile. - a_engine (
TensorEngine): Engine of the input A tile. - b_engine (
TensorEngine): Engine of the input B tile. - EpilogueFnType (
ElementwiseEpilogueFn): Type ofepilogue_fn.
Args:
- c (
TileTensor[c_type, c_layout, MutAnyOrigin, Engine=c_engine]): Output tile of shape[M, N]. Only its shape is read. - a (
TileTensor[a_type, a_layout, ImmutAnyOrigin, Engine=a_engine]): Input A tile of shape[M, K]. - b (
TileTensor[b_type, b_layout, ImmutAnyOrigin, Engine=b_engine]): Input B tile of shape[N, K]. - epilogue_fn (
EpilogueFnType): Stores each output element.
run_with_compute_fn
static def run_with_compute_fn[c_layout: TensorLayout, a_layout: TensorLayout, b_layout: TensorLayout, c_engine: TensorEngine, a_engine: TensorEngine, b_engine: TensorEngine, ComputeFnType: ElementwiseComputeFn](c: TileTensor[c_type, c_layout, MutAnyOrigin, Engine=c_engine], a: TileTensor[a_type, a_layout, ImmutAnyOrigin, Engine=a_engine], b: TileTensor[b_type, b_layout, ImmutAnyOrigin, Engine=b_engine], compute_fn: ComputeFnType)
Same as run, storing compute_fn(idx, value) into c.
Launch with host_args=(compute_fn,).
Parameters:
- c_layout (
TensorLayout): Tensor layout of the output C tile. - a_layout (
TensorLayout): Tensor layout of the input A tile. - b_layout (
TensorLayout): Tensor layout of the input B tile. - c_engine (
TensorEngine): Engine of the output C tile. - a_engine (
TensorEngine): Engine of the input A tile. - b_engine (
TensorEngine): Engine of the input B tile. - ComputeFnType (
ElementwiseComputeFn): Type ofcompute_fn.
Args:
- c (
TileTensor[c_type, c_layout, MutAnyOrigin, Engine=c_engine]): Output tile of shape[M, N]. - a (
TileTensor[a_type, a_layout, ImmutAnyOrigin, Engine=a_engine]): Input A tile of shape[M, K]. - b (
TileTensor[b_type, b_layout, ImmutAnyOrigin, Engine=b_engine]): Input B tile of shape[N, K]. - compute_fn (
ComputeFnType): Maps each output index and value to the value to store.
run_at_tile
static def run_at_tile[c_layout: TensorLayout, a_layout: TensorLayout, b_layout: TensorLayout, c_engine: TensorEngine, a_engine: TensorEngine, b_engine: TensorEngine](c: TileTensor[c_type, c_layout, MutAnyOrigin, Engine=c_engine], a: TileTensor[a_type, a_layout, ImmutAnyOrigin, Engine=a_engine], b: TileTensor[b_type, b_layout, ImmutAnyOrigin, Engine=b_engine], tile_m: Int, tile_n: Int)
Same as run, for a grid whose block_idx is not the tile index.