For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
KVCacheMHAOperand
struct KVCacheMHAOperand[cache_t: KVCacheT]
An implementation for mo.opaque KVCacheT arguments to MHA kernels.
We can eventually remove this trait and just add it as a sub-trait in the KVCacheT type, but we need to solve some cyclic dependencies first.
Parameters
- cache_t (
KVCacheT): The concreteKVCacheTtype backing this operand and providing the paged KV cache storage and lookup tables.
Fields
- cache (
cache_t):
Implemented traits
AnyType,
Copyable,
Deinitable,
DevicePassable,
ImplicitlyCopyable,
MHAOperand,
Movable,
RegisterPassable,
TrivialRegisterPassable
comptime members
device_type
comptime device_type = KVCacheMHAOperand[cache_t]
dtype
comptime dtype = cache_t.dtype
Engine
comptime Engine = cache_t.Engine
page_size
comptime page_size = cache_t.page_size_
quantization_enabled
comptime quantization_enabled = cache_t.quantization_enabled
quantization_granularity
comptime quantization_granularity = cache_t.quantization_granularity
rows_indexed
comptime rows_indexed = False
Whether each key row lives at its own address (indexed_row_ptr).
False for every operand whose keys are contiguous per tile, which the
AMD loaders address through block_paged_tile. A sparse-selection
operand sets it so KVBuffer gathers rows lane by lane instead.
rows_predicated
comptime rows_predicated = False
Whether an invalid indexed key is skipped rather than read.
Only meaningful with rows_indexed. When set, indexed_row says
whether a slot names a row at all, and the loader zeroes the LDS of
every key it skips instead of reading a substitute row.
scale_dtype
comptime scale_dtype = cache_t.scale_dtype
v_col_base
comptime v_col_base = cache_t.kv_params.v_col_base()
Methods
__init__
def __init__(cache: cache_t) -> Self
get_type_name
block_paged_ptr
def block_paged_ptr[tile_size: Int](self, batch_idx: UInt32, start_tok_idx: UInt32, head_idx: UInt32, head_dim_idx: UInt32 = UInt32(0)) -> ImmPointer[Scalar[Self.dtype], ImmutAnyOrigin]
Returns:
block_paged_storage
def block_paged_storage[tile_size: Int](self, batch_idx: UInt32, start_tok_idx: UInt32, head_idx: UInt32, head_dim_idx: UInt32 = UInt32(0)) -> Self.Engine.StorageType[Self.dtype, ImmutAnyOrigin, .GENERIC]
Returns:
Self.Engine.StorageType[Self.dtype, ImmutAnyOrigin, .GENERIC]
scales_block_paged_ptr
def scales_block_paged_ptr(self, batch_idx: Int, start_tok_idx: Int, head_idx: Int, head_dim_idx: Int = Int(0)) -> ImmPointer[Scalar[Self.scale_dtype], ImmutAnyOrigin]
Returns:
load_scale
def load_scale[width: Int](self, batch_idx: Int, start_tok_idx: Int, head_idx: Int, head_dim_idx: Int) -> SIMD[Self.scale_dtype, width]
Returns:
cache_length
max_context_length
num_kv_rows
def num_kv_rows(self) -> Int
Returns the total number of virtual rows in the KV memory view.
Returns:
kv_tma_coords
def kv_tma_coords(self, batch_idx: UInt32, start_tok_idx: UInt32) -> Tuple[Int32, Int32]
Forwards to the cache, which knows its own paging.
The trait's default folds the block into the row, which is what this
override exists to avoid on a paged cache; see
PagedKVCache.kv_tma_coords.
Args:
- batch_idx (
UInt32): Batch entry to address. - start_tok_idx (
UInt32): First token of the tile, within the entry.
Returns:
Tuple[Int32, Int32]: The row within the block, and the block.
row_idx
def row_idx(self, batch_idx: UInt32, start_tok_idx: UInt32) -> UInt32
Returns the row idx when viewing the memory as a matrix.
Args:
- batch_idx (
UInt32): Batch index of the request. - start_tok_idx (
UInt32): Starting token index within the batch.
Returns:
populate
def populate[BN: Int, base_alignment: Int, pair_cta: Bool = False, is_leader: Bool = True](self, batch_idx: UInt32, base_kv_row: UInt32) -> PagedRowIndices[BN, cache_t.page_size_, pair_cta, is_leader]
Delegate to the underlying cache's populate.
PagedKVCache.populate overrides with a SIMD lookup-table read;
other cache types fall through to the scalar default on
KVCacheT.populate. base_alignment is the comptime alignment
of base_kv_row (typically mask.start_column_alignment[...]()).
Parameters:
- BN (
Int): Number of rows in the tile to populate. - base_alignment (
Int): Comptime promise thatbase_kv_row % base_alignment == 0at runtime. - pair_cta (
Bool): Whether K and V run as a paired CTA pair where K covers only a subset of the range (defaults toFalse). - is_leader (
Bool): Whether this CTA is the leader that computes the row indices (defaults toTrue).
Args:
- batch_idx (
UInt32): Batch index of the request. - base_kv_row (
UInt32): Starting KV row index of theBN-row tile.
Returns:
PagedRowIndices[BN, cache_t.page_size_, pair_cta, is_leader]
get_tma_row
def get_tma_row(self, encoded_index: Int32) -> Int32
Convert an encoded sparse index to a physical TMA row.
Args:
- encoded_index (
Int32): Encoded sparse row index. For paged caches this isphysical_block * page_size + offset.
Returns:
sparse_tma_row
def sparse_tma_row(self, batch_idx: UInt32, logical_index: Int32) -> Int32
Forwards to the cache, which knows its own paging.
Args:
- batch_idx (
UInt32): Batch entry the selection belongs to. - logical_index (
Int32): Token position within the entry, or-1.
Returns:
create_tma_tile
def create_tma_tile[swizzle_mode: TensorMapSwizzle, *, BN: Int, depth: Int, BK: Int = padded_depth[cache_t.dtype, swizzle_mode, depth](), fold_chunks: Int = Int(1), row_major: Bool = False](self, ctx: DeviceContext, out tma: SplitLastDimTMATensorTile[Self.dtype, Coord(), swizzle_mode])
Creates a TMA tile for efficient GPU memory transfers.
Parameters:
- swizzle_mode (
TensorMapSwizzle): TMA swizzle mode for shared memory access pattern. - BN (
Int): Number of rows loaded per TMA instruction. - depth (
Int): Full depth of the K/V dimension in global memory, in elements. - BK (
Int): Block size along the depth dimension in elements (defaults topadded_depth[Self.dtype, swizzle_mode, depth]()). - fold_chunks (
Int): Number of contiguous depth chunks folded into one TMA descriptor (defaults to 1). - row_major (
Bool): WhenTruewithfold_chunks >= 2, use the rank-5 chunk-inner box so a tile can span multiple pages (defaults toFalse).
Args:
- ctx (
DeviceContext): The CUDA device context used to create the TMA descriptor.
Returns:
SplitLastDimTMATensorTile[Self.dtype, Coord(), swizzle_mode]
create_paged_tma_tile
def create_paged_tma_tile[swizzle_mode: TensorMapSwizzle, *, BN: Int, depth: Int, BK: Int = padded_depth[cache_t.dtype, swizzle_mode, depth]()](self, ctx: DeviceContext, out tma: SplitLastDimTMATensorTile[Self.dtype, Coord(), swizzle_mode])
Forwards to the cache, whose blocks the descriptor addresses.
Parameters:
- swizzle_mode (
TensorMapSwizzle): TMA swizzle for the innermost dimension. - BN (
Int): Rows per tile. - depth (
Int): Full depth of a head in global memory, in elements. - BK (
Int): Depth per tile, a multiple of the swizzle granularity.
Args:
- ctx (
DeviceContext): The CUDA device context used to create the TMA descriptor.
Returns:
SplitLastDimTMATensorTile[Self.dtype, Coord(), swizzle_mode]
create_scale_tma_tile
def create_scale_tma_tile[BMN: Int](self, ctx: DeviceContext, out tma: TMATensorTile[Self.scale_dtype, Coord()])
Creates a TMA tile for efficient GPU memory transfers. This is useful for m-major MMA operations where we don't need to mask any extra rows.
Parameters:
- BMN (
Int): Number of scale elements loaded per TMA tile row.
Args:
- ctx (
DeviceContext): The CUDA device context used to create the TMA descriptor.
Returns:
create_rope_tma_tile
def create_rope_tma_tile[swizzle_mode: TensorMapSwizzle, *, BN: Int, BK: Int, padded_depth: Int](self, ctx: DeviceContext, out tma: SplitLastDimTMATensorTile[.bfloat16, Coord(), swizzle_mode])
Delegates to the underlying KVCache to create a BF16 rope TMA tile.
Parameters:
- swizzle_mode (
TensorMapSwizzle): TMA swizzle mode for shared memory access pattern. - BN (
Int): Number of rows loaded per TMA instruction. - BK (
Int): Block size along the depth dimension in BF16 elements. - padded_depth (
Int): Byte offset from row start to the BF16 rope data, skipping the FP8 content prefix.
Args:
- ctx (
DeviceContext): The CUDA device context used to create the TMA descriptor.
Returns:
create_gather4_tma_tile
def create_gather4_tma_tile[tile_width: Int, tile_stride: Int = tile_width, swizzle_mode: TensorMapSwizzle = TensorMapSwizzle.SWIZZLE_NONE, tile_height: Int = Int(4), tma_dtype: DType = KVCacheMHAOperand[cache_t].dtype, l2_promotion: TensorMapL2Promotion = TensorMapL2Promotion.NONE](self, ctx: DeviceContext, out tma: TMATensorTile[tma_dtype, Coord(), Coord()])
Creates a 2D TMA gather4 descriptor for this KV cache operand.
Parameters:
- tile_width (
Int): Number of elements per row to load (box width) intma_dtypeelements. - tile_stride (
Int): Row stride in elements in global memory. Defaults totile_width. Use a larger value when the global row is wider than the portion to load. - swizzle_mode (
TensorMapSwizzle): TMA swizzle mode for shared memory access pattern. Defaults toSWIZZLE_NONE. - tile_height (
Int): Number of rows in the tile. Must be a multiple of 4. Defaults to 4 for backward compatibility. - tma_dtype (
DType): Data type used for the TMA descriptor. Defaults toSelf.dtype. When different, the pointer is bitcast. - l2_promotion (
TensorMapL2Promotion): L2 cache promotion hint for TMA loads. Defaults toNONE.
Args:
- ctx (
DeviceContext): The CUDA device context used to create the TMA descriptor.
Returns:
create_rope_gather4_tma_tile
def create_rope_gather4_tma_tile[tile_width: Int, padded_depth: Int, swizzle_mode: TensorMapSwizzle = TensorMapSwizzle.SWIZZLE_NONE, tile_height: Int = Int(4), l2_promotion: TensorMapL2Promotion = TensorMapL2Promotion.NONE](self, ctx: DeviceContext, out tma: TMATensorTile[.bfloat16, Coord(), Coord()])
Delegates to the underlying KVCache to create a BF16 rope gather4 TMA tile.
For the per-tensor rope-aware layout each token row is padded_depth
FP8 bytes (content) followed by BF16 rope elements.
Parameters:
- tile_width (
Int): Number of BF16 elements per row in global memory. - padded_depth (
Int): Byte offset from row start to the BF16 rope data, skipping the FP8 content prefix. - swizzle_mode (
TensorMapSwizzle): TMA swizzle mode for shared memory access pattern. Defaults toSWIZZLE_NONE. - tile_height (
Int): Number of rows in the tile. Must be a multiple of 4. Defaults to 4. - l2_promotion (
TensorMapL2Promotion): L2 cache promotion hint for TMA loads. Defaults toNONE.
Args:
- ctx (
DeviceContext): The CUDA device context used to create the TMA descriptor.
Returns:
scales_raw_ptr
def scales_raw_ptr(self) -> MutPointer[Float32, MutAnyOrigin]
Returns the base pointer to the quantization scales tensor.
Returns: