For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
MatmulTileWriter
struct MatmulTileWriter[dtype: DType, tensor_layout: TensorLayout, output_engine: TensorEngine, linear_idx_type: DType, smem_tile_layout: TensorLayout, //, *, BM: Int, BN: Int, swizzle: TensorMapSwizzle, wgmma_shape: Tuple[Int, Int, Int], num_consumer: Int = Int(1), use_tma_store: Bool = False, swapAB: Bool = False]
Writes WGMMA accumulator register tiles to global memory for SM90 matmul.
Provides two write paths: TMA store (shared memory → global via hardware async copies) and thread-wise store (register → global directly). Handles swizzled shared memory layouts, optional elementwise epilogues, and A/B swap when the problem uses the swapAB small-M strategy.
Parameters
- dtype (
DType): Element type of the output tensor and shared memory tile (inferred). - tensor_layout (
TensorLayout): Memory layout of the output tensor in global memory (inferred). - output_engine (
TensorEngine): Engine backing the output tensor in global memory (inferred). - linear_idx_type (
DType): Integer type used for linear index arithmetic into the output tensor (inferred). - smem_tile_layout (
TensorLayout): Memory layout of the shared memory tile used to stage output before storing to global memory; also defines the workgroup tile dimensionsWG_BMandWG_BN(inferred). - BM (
Int): Row (M) dimension of the matmul output tile per block. - BN (
Int): Column (N) dimension of the matmul output tile per block. - swizzle (
TensorMapSwizzle): TMA swizzle mode applied to shared memory for TMA async stores. - wgmma_shape (
Tuple[Int, Int, Int]): Shape of each WGMMA instruction as(M, N, K). - num_consumer (
Int): Number of consumer warp groups sharing the output tile (defaults to 1). - use_tma_store (
Bool): Whether to use TMA async stores for shared memory to global memory copies (defaults toFalse). - swapAB (
Bool): Whether to swap the A and B operand roles for the small-M strategy, transposing the tile and block coordinate mapping (defaults toFalse).
Fields
- tensor (
MatmulTileWriter[BM=BM, BN=BN, swizzle=swizzle, wgmma_shape=wgmma_shape, num_consumer=num_consumer, use_tma_store=use_tma_store, swapAB=swapAB].CTensorType): - smem_tile (
TileTensor[dtype, smem_tile_layout, MutAnyOrigin, address_space=AddressSpace.SHARED]): - warp_group_thread_idx (
Int): - local_warp_group_idx (
Int): - local_thread_idx (
Int): - block_y (
Int): - block_x (
Int):
Implemented traits
AnyType,
Copyable,
Deinitable,
ImplicitlyCopyable,
Movable,
RegisterPassable,
TrivialRegisterPassable
comptime members
CTensorType
comptime CTensorType = TileTensor[dtype, tensor_layout, MutAnyOrigin, Engine=output_engine, linear_idx_type=linear_idx_type]
frag_size
comptime frag_size = ((SIMD((load_from_mem wgmma_shape.__getitem_param__[Int(0)]())) * SIMD((load_from_mem wgmma_shape.__getitem_param__[Int(1)]()))) // _resolve_warpgroup_size())
lambda_type
comptime lambda_type = def[dtype: DType, width: SIMDLength, *, alignment: Int = Int(1)](Coord[**?], mut SIMD[dtype, width]) -> None
N
comptime N = tensor_layout.static_shape[Int(1)]
num_consumer_threads
comptime num_consumer_threads = (num_consumer * _resolve_warpgroup_size())
num_m_mmas
comptime num_m_mmas = ((BM // SIMD((load_from_mem wgmma_shape.__getitem_param__[Int(0)]()))) // num_consumer)
num_n_mmas
comptime num_n_mmas = (BN // SIMD((load_from_mem wgmma_shape.__getitem_param__[Int(1)]())))
simd_size
comptime simd_size = simd_width_of[dtype]()
WG_BM
comptime WG_BM = smem_tile_layout.static_shape[Int(0)]
WG_BN
comptime WG_BN = smem_tile_layout.static_shape[Int(1)]
Methods
__init__
def __init__(tensor: TileTensor[dtype, tensor_layout, MutAnyOrigin, Engine=output_engine, linear_idx_type=linear_idx_type], smem_tile: TileTensor[dtype, smem_tile_layout, MutAnyOrigin, address_space=AddressSpace.SHARED], warp_group_thread_idx: Int, local_warp_group_idx: Int, local_thread_idx: Int, block_y: Int, block_x: Int) -> Self
write_tile
def write_tile[tma_tile_shape: Coord[*tma_tile_shape.element_types], tma_desc_shape: Coord[*tma_desc_shape.element_types], accum_type: DType, reg_tile_layout: TensorLayout, EpilogueFnType: ElementwiseEpilogueFn = inflated$def[comptime[dtype: DType, width: SIMDLength, *, alignment: Int] #kgen.func.symbol<@linalg::@utils::@"_no_epilogue_body[::DType,::SIMDLength,::SIMD[DType.int, 1],KGENParamList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike],::TypeList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike, $3]](::Coord[$3, $4],::SIMD[$0, $1])"<:!lit.struct<@std::@builtin::@dtype::@DType> dtype, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> width, :!lit.struct<@std::@simd::@SIMD<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype index}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}>> alignment, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?, :!lit.struct<@std::@builtin::@variadics::@TypeList<:meta<!lit.trait<@std::@traits::@anytype::@AnyType>> trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?>> *?>>], ComputeFnType: ElementwiseComputeFn = inflated$def[comptime[dtype: DType, width: SIMDLength, *, alignment: Int] #kgen.func.symbol<@linalg::@utils::@"identity_compute_fn[::DType,::SIMDLength,::SIMD[DType.int, 1],KGENParamList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike],::TypeList[::Intable & ::Defaultable & ::RegisterPassable & ::TrivialRegisterPassable & ::Writable & ::AnyType & ::Copyable & ::ImplicitlyCopyable & ::Deinitable & ::Movable & ::CoordLike, $3]](::Coord[$3, $4],::SIMD[$0, $1])"<:!lit.struct<@std::@builtin::@dtype::@DType> dtype, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> width, :!lit.struct<@std::@simd::@SIMD<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype index}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}>> alignment, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?, :!lit.struct<@std::@builtin::@variadics::@TypeList<:meta<!lit.trait<@std::@traits::@anytype::@AnyType>> trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>, :param_list<trait<@std::@builtin::@int::@Intable, @std::@builtin::@value::@Defaultable, @std::@builtin::@value::@RegisterPassable, @std::@builtin::@value::@TrivialRegisterPassable, @std::@format::@__init__::@Writable, @std::@traits::@anytype::@AnyType, @std::@traits::@copyable::@Copyable, @std::@traits::@copyable::@ImplicitlyCopyable, @std::@traits::@deinitable::@Deinitable, @std::@traits::@movable::@Movable, @std::@utils::@coord::@CoordLike>> *?>> *?>>], //, *, has_epilogue_fn: Bool = False, has_compute_fn: Bool = False](self, tma_op: TMATensorTile[dtype, tma_tile_shape, tma_desc_shape], reg_tile: TileTensor[accum_type, reg_tile_layout, Engine=reg_tile.Engine, address_space=AddressSpace.LOCAL, linear_idx_type=reg_tile.linear_idx_type], epilogue_fn: EpilogueFnType = no_epilogue_fn, compute_fn: ComputeFnType = no_compute_fn)
Write output from registers to global memory.
Selects optimized st.matrix path for bf16 when constraints are met, otherwise uses general register-to-global path.
Parameters:
- tma_tile_shape (
Coord[*element_types]): Shape of each TMA store tile per async copy. - tma_desc_shape (
Coord[*element_types]): Full shape of the TMA tensor descriptor. - accum_type (
DType): Data type of the WGMMA accumulator register tile. - reg_tile_layout (
TensorLayout): Memory layout of the accumulator register tile. - EpilogueFnType (
ElementwiseEpilogueFn): Type ofepilogue_fn(inferred). - ComputeFnType (
ElementwiseComputeFn): Type ofcompute_fn(inferred). - has_epilogue_fn (
Bool): Whetherepilogue_fnstores the output. - has_compute_fn (
Bool): Whethercompute_fnmaps each output before it is stored.
Args:
- tma_op (
TMATensorTile[dtype, tma_tile_shape, tma_desc_shape]): TMA tensor tile descriptor used for async stores from shared memory to global memory. - reg_tile (
TileTensor[accum_type, reg_tile_layout, Engine=reg_tile.Engine, address_space=AddressSpace.LOCAL, linear_idx_type=reg_tile.linear_idx_type]): WGMMA accumulator register tile containing the matmul result to write. - epilogue_fn (
EpilogueFnType): Stores each output element at its global coordinates. - compute_fn (
ComputeFnType): Maps each output element at its global coordinates to the value to store.