IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo trait

MHAOperand

This serves as the trait to support arguments to our MHA kernel.

Implemented traits​

AnyType, Copyable, Deinitable, DevicePassable, ImplicitlyCopyable, Movable, RegisterPassable, TrivialRegisterPassable

comptime members​

dtype​

comptime dtype

Engine​

comptime Engine

page_size​

comptime page_size

quantization_enabled​

comptime quantization_enabled = False

quantization_granularity​

comptime quantization_granularity

rows_indexed​

comptime rows_indexed = False

Whether each key row lives at its own address (indexed_row_ptr).

False for every operand whose keys are contiguous per tile, which the AMD loaders address through block_paged_tile. A sparse-selection operand sets it so KVBuffer gathers rows lane by lane instead.

rows_predicated​

comptime rows_predicated = False

Whether an invalid indexed key is skipped rather than read.

Only meaningful with rows_indexed. When set, indexed_row says whether a slot names a row at all, and the loader zeroes the LDS of every key it skips instead of reading a substitute row.

scale_dtype​

comptime scale_dtype

v_col_base​

comptime v_col_base = Int(0)

The first column of the value within a row of the value operand.

Non-zero only for a split KV cache, whose key and value operands are one view of rows that pack a key and then a narrower value.

Required methods​

block_paged_storage​

def block_paged_storage[tile_size: Int](self, batch_idx: UInt32, start_tok_idx: UInt32, head_idx: UInt32, head_dim_idx: UInt32 = UInt32(0)) -> Self.Engine.StorageType[Self.dtype, ImmutAnyOrigin, .GENERIC]

Returns:

_Self.Engine.StorageType[_Self.dtype, ImmutAnyOrigin, .GENERIC]

block_paged_ptr​

def block_paged_ptr[tile_size: Int](self, batch_idx: UInt32, start_tok_idx: UInt32, head_idx: UInt32, head_dim_idx: UInt32 = UInt32(0)) -> ImmPointer[Scalar[Self.dtype], ImmutAnyOrigin]

Returns:

ImmPointer[Scalar[_Self.dtype], ImmutAnyOrigin]

scales_block_paged_ptr​

def scales_block_paged_ptr(self, batch_idx: Int, start_tok_idx: Int, head_idx: Int, head_dim_idx: Int = Int(0)) -> ImmPointer[Scalar[Self.scale_dtype], ImmutAnyOrigin]

Returns:

ImmPointer[Scalar[_Self.scale_dtype], ImmutAnyOrigin]

load_scale​

def load_scale[width: Int](self, batch_idx: Int, start_tok_idx: Int, head_idx: Int, head_dim_idx: Int) -> SIMD[Self.scale_dtype, width]

Returns:

SIMD[_Self.scale_dtype, width]

cache_length​

def cache_length(self, batch_idx: Int) -> Int

Returns the length of the cache for a given batch index.

Returns:

Int

max_context_length​

def max_context_length(self) -> UInt32

Returns the maximum cache length in a given batch index.

Returns:

UInt32

num_kv_rows​

def num_kv_rows(self) -> Int

Returns the total number of virtual rows in the KV memory view.

For paged caches this accounts for the paging stride so that TMA descriptors can be sized to cover the entire address space.

Returns:

Int

row_idx​

def row_idx(self, batch_idx: UInt32, start_tok_idx: UInt32) -> UInt32

Returns the row idx when viewing the memory as a matrix.

Returns:

UInt32

get_tma_row​

def get_tma_row(self, encoded_index: Int32) -> Int32

Convert an encoded sparse index to a physical TMA row.

For paged caches the encoded index is physical_block * page_size + offset and this method returns physical_block * stride + offset. Non-paged operands return the encoded index unchanged.

Returns:

Int32

create_tma_tile​

def create_tma_tile[swizzle_mode: TensorMapSwizzle, *, BN: Int, depth: Int, BK: Int = padded_depth[_Self.dtype, swizzle_mode, depth](), fold_chunks: Int = Int(1), row_major: Bool = False](self, ctx: DeviceContext) -> SplitLastDimTMATensorTile[Self.dtype, Coord(), swizzle_mode]

Creates a TMA tile for efficient GPU memory transfers. This is useful for k-major MMA operations where we don't need to mask any extra rows.

When fold_chunks >= 2 the contiguous depth chunks are folded into one rank-4 TMA descriptor (SM100 K-only optimization). The caller must pass the value from kv_tma_fold_chunks and use the same value at the tma_copy_k issue site. 1 (default) is the original per-chunk behavior.

When row_major is True (and fold_chunks >= 2) the fold uses the rank-5 chunk-inner (page-dense) box instead of the rank-4 chunk-outer box, so a tile can span multiple pages with one TMA per page. The same value must be used at the tma_copy_k/tma_copy_v issue site and the P@V MMA consumer descriptor.

Returns:

SplitLastDimTMATensorTile[_Self.dtype, Coord(), swizzle_mode]

create_scale_tma_tile​

def create_scale_tma_tile[BMN: Int](self, ctx: DeviceContext) -> TMATensorTile[Self.scale_dtype, Coord()]

Creates a TMA tile for efficient GPU memory transfers. This is useful for m-major MMA operations where we don't need to mask any extra rows.

Returns:

TMATensorTile[_Self.scale_dtype, Coord()]

create_rope_tma_tile​

def create_rope_tma_tile[swizzle_mode: TensorMapSwizzle, *, BN: Int, BK: Int, padded_depth: Int](self, ctx: DeviceContext) -> SplitLastDimTMATensorTile[.bfloat16, Coord(), swizzle_mode]

Creates a BF16 TMA tile for the rope portion of the per-tensor rope-aware KV cache.

Returns:

SplitLastDimTMATensorTile[.bfloat16, Coord(), swizzle_mode]

create_gather4_tma_tile​

def create_gather4_tma_tile[tile_width: Int, tile_stride: Int = tile_width, swizzle_mode: TensorMapSwizzle = TensorMapSwizzle.SWIZZLE_NONE, tile_height: Int = Int(4), tma_dtype: DType = _Self.dtype, l2_promotion: TensorMapL2Promotion = TensorMapL2Promotion.NONE](self, ctx: DeviceContext) -> TMATensorTile[tma_dtype, Coord(), Coord()]

Creates a 2D TMA gather4 descriptor for this operand.

The descriptor views the data as a flat 2D matrix of [num_kv_rows, tile_width] and is configured for gather4 operations that load 4 non-contiguous rows per TMA instruction. The box width is derived from the swizzle mode; for SWIZZLE_NONE it equals tile_width.

When tma_dtype differs from Self.dtype, the underlying data pointer is bitcast to tma_dtype at descriptor creation time.

Parameters:

  • ​tile_width (Int): Number of elements per row to load (box width) in tma_dtype elements.
  • ​tile_stride (Int): Row stride in elements in global memory. Defaults to tile_width. Use a larger value when the global row is wider than the portion to load.
  • ​swizzle_mode (TensorMapSwizzle): TMA swizzle mode for shared memory access pattern. Defaults to SWIZZLE_NONE.
  • ​tile_height (Int): Number of rows in the tile. Must be a multiple of 4. Defaults to 4 for backward compatibility.
  • ​tma_dtype (DType): The data type used for the TMA descriptor. Defaults to Self.dtype. When different, the pointer is bitcast.
  • ​l2_promotion (TensorMapL2Promotion): L2 cache promotion hint for TMA loads. Defaults to NONE.

Args:

  • ​ctx (DeviceContext): The CUDA device context used to create the TMA descriptor.

Returns:

TMATensorTile[tma_dtype, Coord(), Coord()]: A TMATensorTile with box width derived from the swizzle mode.

create_rope_gather4_tma_tile​

def create_rope_gather4_tma_tile[tile_width: Int, padded_depth: Int, swizzle_mode: TensorMapSwizzle = TensorMapSwizzle.SWIZZLE_NONE, tile_height: Int = Int(4), l2_promotion: TensorMapL2Promotion = TensorMapL2Promotion.NONE](self, ctx: DeviceContext) -> TMATensorTile[.bfloat16, Coord(), Coord()]

Creates a BF16 gather4 TMA descriptor for the rope portion of the KV cache.

For the per-tensor rope-aware layout each token row is padded_depth FP8 bytes (content) followed by BF16 rope elements. This method offsets the base pointer by padded_depth bytes, reinterprets as BF16, and creates a gather4 TMA descriptor.

Parameters:

  • ​tile_width (Int): Number of BF16 elements per row in global memory.
  • ​padded_depth (Int): Byte offset from row start to the rope data.
  • ​swizzle_mode (TensorMapSwizzle): TMA swizzle mode for shared memory access pattern.
  • ​tile_height (Int): Number of rows in the tile. Must be a multiple of 4.
  • ​l2_promotion (TensorMapL2Promotion): L2 cache promotion hint for TMA loads. Defaults to NONE.

Args:

  • ​ctx (DeviceContext): The CUDA device context used to create the TMA descriptor.

Returns:

TMATensorTile[.bfloat16, Coord(), Coord()]: A BF16 TMATensorTile configured for gather4.

scales_raw_ptr​

def scales_raw_ptr(self) -> MutPointer[Float32, MutAnyOrigin]

Returns the base pointer to the quantization scales tensor.

Returns a null pointer for operands without quantization support.

Returns:

MutPointer[Float32, MutAnyOrigin]

Provided methods​

indexed_row_ptr​

def indexed_row_ptr(self, batch_idx: UInt32, key_idx: UInt32) -> ImmPointer[Scalar[Self.dtype], ImmutAnyOrigin]

Returns the start of key key_idx's row for batch_idx.

Only read when rows_indexed is set. The default is the dense row, so a contiguous operand answers consistently if asked.

Args:

  • ​batch_idx (UInt32): Batch entry to address.
  • ​key_idx (UInt32): Key position within the entry.

Returns:

ImmPointer[Scalar[_Self.dtype], ImmutAnyOrigin]: A pointer to the key's first element at head 0.

indexed_row​

def indexed_row(self, batch_idx: UInt32, key_idx: UInt32) -> Tuple[Pointer[Scalar[Self.dtype], ImmutAnyOrigin], Bool]

Returns key key_idx's row start and whether its slot names a row.

Only read when rows_predicated is set; the caller applies indexed_valid_length itself. The default is indexed_row_ptr, always named.

Args:

  • ​batch_idx (UInt32): Batch entry to address.
  • ​key_idx (UInt32): Key position within the entry, below the stride.

Returns:

Tuple[Pointer[Scalar[_Self.dtype], ImmutAnyOrigin], Bool]: The row's first element at head 0, and whether to load it.

indexed_valid_length​

def indexed_valid_length(self, batch_idx: UInt32) -> Int

Returns the number of leading keys of batch_idx that are valid.

Read when rows_indexed is set: the decode caps each entry's keys at it. The default bounds nothing.

Args:

  • ​batch_idx (UInt32): Batch entry.

Returns:

Int: The entry's valid key count.

check_indexed_rows​

def check_indexed_rows(self, batch_idx: UInt32)

Asserts the operand's row selection is consistent for batch_idx.

A hook for assert-enabled builds; the default checks nothing.

Args:

  • ​batch_idx (UInt32): Batch entry to check.

block_paged_tile​

def block_paged_tile[layout_t: TensorLayout, //, tile_size: Int](self, batch_idx: UInt32, start_tok_idx: UInt32, head_idx: UInt32, layout_val: layout_t, head_dim_idx: UInt32 = UInt32(0)) -> TileTensor[Self.dtype, layout_t, ImmutAnyOrigin, Engine=Self.Engine]

Wraps block_paged_ptr in a TileTensor with the caller's layout.

Parameters:

  • ​layout_t (TensorLayout): The TensorLayout of the returned TileTensor (inferred).
  • ​tile_size (Int): Tile size in rows used to compute the paged block pointer.

Args:

  • ​batch_idx (UInt32): Batch index of the request.
  • ​start_tok_idx (UInt32): Starting token index within the batch.
  • ​head_idx (UInt32): KV head index.
  • ​layout_val (layout_t): Concrete layout_t instance describing the tile layout.
  • ​head_dim_idx (UInt32): Index along the head dimension (defaults to 0).

Returns:

TileTensor[_Self.dtype, layout_t, ImmutAnyOrigin, Engine=_Self.Engine]

scale_tma_coords​

def scale_tma_coords(self, batch_idx: UInt32, start_tok_idx: UInt32) -> Tuple[Int32, Int32]

The (row, block) coordinate of a token's scale tile.

Defaults to the flat form, which is one block: the row is this operand's own row index rounded down to the 16-byte unit a TMA box must start on, and the block is 0. A paged operand overrides this, because folding its block into the row makes the coordinate scale with the whole pool rather than with one block.

Args:

  • ​batch_idx (UInt32): Batch entry to address.
  • ​start_tok_idx (UInt32): First token of the tile, within the entry.

Returns:

Tuple[Int32, Int32]: The row, and the block.

kv_tma_coords​

def kv_tma_coords(self, batch_idx: UInt32, start_tok_idx: UInt32) -> Tuple[Int32, Int32]

The (row_in_block, block) coordinate of a token's KV tile.

Defaults to the flat form, which is one block: the row is this operand's own row index and the block is 0. A paged operand overrides this, because folding its block into the row makes the coordinate scale with the whole shared slab rather than with one block, and a TMA coordinate is signed 32-bit. Mirrors :meth:scale_tma_coords, which splits for the same reason.

Args:

  • ​batch_idx (UInt32): Batch entry to address.
  • ​start_tok_idx (UInt32): First token of the tile, within the entry.

Returns:

Tuple[Int32, Int32]: The row within the block, and the block.

populate​

def populate[BN: Int, base_alignment: Int, pair_cta: Bool = False, is_leader: Bool = True](self, batch_idx: UInt32, base_kv_row: UInt32) -> PagedRowIndices[BN, _Self.page_size, pair_cta, is_leader]

Populate a full PagedRowIndices[BN, ...] for a BN-row tile.

Returns the precomputed physical row indices for the num_pages sub-tile pages covering the BN-row range starting at base_kv_row for batch_idx. Both K's TMA (which may cover only a subset in pair_cta mode) and V's TMA (which covers the full range) can then consume the result without any lazy LUT lookup.

base_alignment is a comptime promise that base_kv_row % base_alignment == 0 at runtime: typically mask.start_column_alignment[...](). The PagedKVCache override uses it to pick the largest legal SIMD chunk for its LUT vector load and to skip the intra-page divmod when base_alignment % page_size == 0.

Default implementation: scalar loop over num_pages calls to row_idx. Overrides (e.g. PagedKVCache) replace this with a single SIMD load from the underlying lookup table.

Returns:

PagedRowIndices[BN, _Self.page_size, pair_cta, is_leader]

sparse_tma_row​

def sparse_tma_row(self, batch_idx: UInt32, logical_index: Int32) -> Int32

Returns the TMA row of a sparse selection's logical token, or -1.

Default: row_idx, with a negative (unused) slot staying -1. Cache-backed operands forward to KVCacheT.sparse_tma_row, which on a paged cache also maps an unallocated page to -1.

Args:

  • ​batch_idx (UInt32): Batch entry the selection belongs to.
  • ​logical_index (Int32): Token position within the entry, or -1.

Returns:

Int32: The row get_tma_row would return for the slot's physical index, or -1.

create_paged_tma_tile​

def create_paged_tma_tile[swizzle_mode: TensorMapSwizzle, *, BN: Int, depth: Int, BK: Int = padded_depth[_Self.dtype, swizzle_mode, depth]()](self, ctx: DeviceContext, out tma: SplitLastDimTMATensorTile[Self.dtype, Coord(), swizzle_mode])

Creates the KV tile that :meth:kv_tma_coords addresses.

The split counterpart of :meth:create_tma_tile: a paged operand returns its cache's (block, row_in_block) descriptor, and a contiguous one the single-block form of it, so a kernel that reads through this pair has one code path for both. The default refuses at compile time, for operands no such kernel reads.

Parameters:

  • ​swizzle_mode (TensorMapSwizzle): TMA swizzle for the innermost dimension.
  • ​BN (Int): Rows per tile.
  • ​depth (Int): Full depth of a head in global memory, in elements.
  • ​BK (Int): Depth per tile, a multiple of the swizzle granularity.

Args:

  • ​ctx (DeviceContext): The CUDA device context used to create the TMA descriptor.

Returns:

SplitLastDimTMATensorTile[_Self.dtype, Coord(), swizzle_mode]: The TMA descriptor.

create_index_scale_tma_tile​

def create_index_scale_tma_tile[TILE: Int](self, ctx: DeviceContext) -> TMATensorTile[Self.dtype, Coord()]

Creates a flat TMA tile over the scale pool this operand addresses.

Only meaningful for an operand that IS a scale pool, so the default rejects; block_paged_ptr must already address scales. The box is TILE scales plus one alignment unit of slack -- a TMA's global start must be 16-byte aligned and a key index is the innermost coordinate here, so an unaligned caller rounds its base down and skips the residual.

Returns:

TMATensorTile[_Self.dtype, Coord()]

Was this page helpful?