For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
KVMmaOp
struct KVMmaOp[in_type: DType, mma_shape: Tuple[Int, Int, Int], num_mmas: Int, num_k_mmas: Int, num_k_tiles: Int, BN: Int, BK: Int, transpose_b: Bool = True, swizzle: Optional[Swizzle] = None, out_type: DType = get_accum_type[in_type]()]
Owns the K or V operand register tile and its SMEM→reg load logic.
Attention has two sequential GEMMs (P = Q @ K^T, O += P @ V). Instantiate
one KVMmaOp per operand role. This keeps KVBuffer focused on SMEM
storage + DMA and moves MMA-side concerns (reg layout, frag size,
fragment loads) here.
The register layout is organized as
[num_k_tiles][num_k_mmas][num_mmas] x input_frag_size: each BK strip
holds num_k_mmas * num_mmas fragments back-to-back.
Parameters
- in_type (
DType): Operand element type (bfloat16 or float8_e4m3fn). - mma_shape (
Tuple[Int, Int, Int]): MMA instruction shape [M, N, K]. - num_mmas (
Int): MMA tiles along the warp's M or N axis (WN/MMA_M for K). - num_k_mmas (
Int): MMA tiles along K within a single BK strip. - num_k_tiles (
Int): Number of BK strips across the full depth. - BN (
Int): KV block height (needed by V load methods for SMEM offset math). - BK (
Int): KV block width (needed by V load methods for SMEM offset math). - transpose_b (
Bool): True for K (transposed load), False for V. - swizzle (
Optional[Swizzle]): Optional SMEM swizzle: vector-space for prefill, element-space for decode. - out_type (
DType): Accumulator data type (defaults to accum(in_type)).
Fields
- reg_tile (
TileTensor[in_type, Layout[TypeList[ComptimeInt[(num_k_tiles * (num_mmas * num_k_mmas))], ComptimeInt[(Int((mul (load_from_mem mma_shape.__getitem_param__[Int(2)]()), (load_from_mem mma_shape.__getitem_param__[Int(1)]()))) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)]](), TypeList[ComptimeInt[(Int((mul (load_from_mem mma_shape.__getitem_param__[Int(2)]()), (load_from_mem mma_shape.__getitem_param__[Int(1)]()))) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]):
Implemented traits
comptime members
input_frag_size
comptime input_frag_size = ((KVMmaOp[in_type, mma_shape, num_mmas, num_k_mmas, num_k_tiles, BN, BK, transpose_b, swizzle, out_type].MMA_K * KVMmaOp[in_type, mma_shape, num_mmas, num_k_mmas, num_k_tiles, BN, BK, transpose_b, swizzle, out_type].MMA_N) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)
MMA_K
comptime MMA_K = (load_from_mem mma_shape.__getitem_param__[Int(2)]())
MMA_M
comptime MMA_M = (load_from_mem mma_shape.__getitem_param__[Int(0)]())
MMA_N
comptime MMA_N = (load_from_mem mma_shape.__getitem_param__[Int(1)]())
Methods
__init__
def __init__(out self)
load_prefill
def load_prefill[bk_tile: Int](self, warp_smem: TileTensor[in_type, Engine=warp_smem.Engine, address_space=AddressSpace.SHARED, linear_idx_type=warp_smem.linear_idx_type])
Load the bk_tile-th BK strip of K fragments from SMEM.
Delegates to TiledMmaLoader.load_b (M-outer iteration,
vector-space swizzle). Handles both BF16 (single load per MMA
tile) and FP8 (two-half-K load + join) via the num_packs
branch inside load_b.
Parameters:
- bk_tile (
Int): Index of the BK strip to load into the register tile.
Args:
- warp_smem (
TileTensor[in_type, Engine=warp_smem.Engine, address_space=AddressSpace.SHARED, linear_idx_type=warp_smem.linear_idx_type]): Source warp tile in shared memory holding the K fragments for this strip.
load_prefill_split
def load_prefill_split[bk_tile: Int, has_hi: Bool](self, warp_smem_lo: TileTensor[in_type, Engine=warp_smem_lo.Engine, address_space=AddressSpace.SHARED, linear_idx_type=warp_smem_lo.linear_idx_type], warp_smem_hi: TileTensor[in_type, Engine=warp_smem_hi.Engine, address_space=AddressSpace.SHARED, linear_idx_type=warp_smem_hi.linear_idx_type])
Load the bk_tile-th MMA K=128 strip composed of two adjacent WN × (MMA_K/2) SMEM blocks.
Used by FP8 16x16x128 MLA decode when the SMEM block width
(bk_smem) is MMA_K/2 = 64 instead of BK = 128. Each MMA
strip's two K-halves live in two separate BN × 64 SMEM blocks
(matching the no-pad K layout at depth=576).
When has_hi == False the hi half is register-zero: this
handles the partial K-tile at the rope tail (strip 4 for
depth=576: lo holds depth 512..575 from block 8, hi has no
backing block, MMA upper-half lane registers get 0).
Parameters:
- bk_tile (
Int): Index of the BK strip to load into the register tile. - has_hi (
Bool): Whether the high K-half has backing SMEM. When False the high half is register-zero for the partial K-tile at the rope tail.
Args:
- warp_smem_lo (
TileTensor[in_type, Engine=warp_smem_lo.Engine, address_space=AddressSpace.SHARED, linear_idx_type=warp_smem_lo.linear_idx_type]): SMEM tile holding the low K half (depth[0, MMA_K/2)of this strip). - warp_smem_hi (
TileTensor[in_type, Engine=warp_smem_hi.Engine, address_space=AddressSpace.SHARED, linear_idx_type=warp_smem_hi.linear_idx_type]): SMEM tile holding the high K half (depth[MMA_K/2, MMA_K)); ignored whenhas_hiis False.
mma_tile_at
def mma_tile_at[bk_tile: Int, kg: Int](self) -> TileTensor[in_type, Layout[TypeList[ComptimeInt[num_mmas], ComptimeInt[(Int((mul (load_from_mem mma_shape.__getitem_param__[Int(2)]()), (load_from_mem mma_shape.__getitem_param__[Int(1)]()))) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)]](), TypeList[ComptimeInt[(Int((mul (load_from_mem mma_shape.__getitem_param__[Int(2)]()), (load_from_mem mma_shape.__getitem_param__[Int(1)]()))) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> _std::_sys::_info::_"_current_target()"), :param_list<trait<_std::_traits::_anytype::_AnyType>> [_max::__plugin::__impl::__NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)], ComptimeInt[Int(1)]]()], MutUntrackedOrigin, address_space=AddressSpace.LOCAL]
Sub-view of the reg tile for a given (bk_tile, k-group) pair.
Parameters:
- bk_tile (
Int): Index of the BK strip in the register tile. - kg (
Int): K-group index within the BK strip.
Returns:
load_v_bf16
def load_v_bf16[bk_tile: Int](self, smem_base: MutPointer[Scalar[in_type], MutAnyOrigin, address_space=AddressSpace.SHARED])
Load the bk_tile-th BK strip of BF16 V fragments from SMEM.
V SMEM is blocked (num_repeats × BN × BK, row-major within each
block). For each (k, i) ∈ [num_k_mmas] × [num_mmas], build an
MMA sub-tile view with the correct (BK, 1) row stride (not
(MMA_M, 1), see smem_mma_subtile header), call
TiledMmaLoader.load_b_tr, and write the fragment into the
reg slot mma_tile_at[bk_tile, k][i].
Only valid when transpose_b == False and in_type == bfloat16.
Parameters:
- bk_tile (
Int): Index of the BK strip to load.
Args:
- smem_base (
MutPointer[Scalar[in_type], MutAnyOrigin, address_space=AddressSpace.SHARED]): Base pointer to the V operand SMEM buffer in shared memory.
load_v_fp8_strip
def load_v_fp8_strip[bk_tile: Int](self, smem_base: MutPointer[Scalar[in_type], MutAnyOrigin, address_space=AddressSpace.SHARED], rel_key: Int, hw_key_shift: Int, depth_base: Int)
Load the bk_tile-th BK strip of FP8 V fragments from SMEM.
Iterates dt over the depth direction and calls
TiledMmaLoader.load_v_fp8_strip per (bk_tile, dt) pair,
writing the joined 32-element SIMD into
mma_tile_at[bk_tile, 0][dt].
Only valid when transpose_b == False and in_type is FP8.
Caller precomputes the lane-only coords (rel_key,
hw_key_shift, depth_base) once before a multi-bk loop:
they don't depend on bk_tile or dt.
Parameters:
- bk_tile (
Int): Index of the BK strip to load.
Args:
- smem_base (
MutPointer[Scalar[in_type], MutAnyOrigin, address_space=AddressSpace.SHARED]): Base pointer to the V operand SMEM buffer in shared memory. - rel_key (
Int): Lane-relative key coordinate, precomputed by the caller (independent ofbk_tileand depth). - hw_key_shift (
Int): Hardware key shift for lane addressing, precomputed by the caller. - depth_base (
Int): Depth base coordinate for lane addressing, precomputed by the caller.
mma
def mma[swap_a_b: Bool = False](self, a: TileTensor[Engine=a.Engine, address_space=AddressSpace.LOCAL, linear_idx_type=a.linear_idx_type], c: TileTensor[Engine=c.Engine, address_space=AddressSpace.LOCAL, linear_idx_type=c.linear_idx_type], bk_tile: Int, kg: Int)
Compute C += A * B using this op's reg tile as B operand.
Parameters:
- swap_a_b (
Bool): Whether to swap A and B operands in the underlyinggpu_mmacall (defaults to False).
Args:
- a (
TileTensor[Engine=a.Engine, address_space=AddressSpace.LOCAL, linear_idx_type=a.linear_idx_type]): A operand register tile in local memory. - c (
TileTensor[Engine=c.Engine, address_space=AddressSpace.LOCAL, linear_idx_type=c.linear_idx_type]): Accumulator register tile, updated in place. - bk_tile (
Int): Index of the BK strip in this op's reg tile to use as the B operand. - kg (
Int): K-group index within the BK strip selecting the MMA fragment row.