IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

SparseRowsMHAOperand

struct SparseRowsMHAOperand[base_t: MHAOperand, //, row_width: Int, predicated: Bool = False]

A K operand whose key rows are a top-k selection from a paged cache.

Key j of batch entry b is the cache row named by the selection's j-th index, so an attention kernel can read the selected rows in place instead of from a gathered copy. Keys at or past the entry's valid length, and -1 slots, read zero_row: the attention mask drops their scores, and zeros keep 0 * value finite where uninitialized cache bytes might not be.

With predicated, the operand serves one decode token per entry: the valid length is computed in closed form from the cache length (see valid_length) rather than read from a buffer, and invalid keys are reported through indexed_row so the loader skips them instead of reading zero_row.

Everything except the row addressing forwards to base.

Parameters​

  • ​base_t (MHAOperand): The operand over the full paged cache.
  • ​row_width (Int): Elements per cache row (all heads), the stride between consecutive physical rows.
  • ​predicated (Bool): Whether invalid keys are skipped and the valid length is closed-form (decode) rather than read from valid_lengths.

Fields​

  • ​base (base_t):
  • ​blocks_base (ImmPointer[Scalar[SparseRowsMHAOperand[row_width, predicated].dtype], ImmUntrackedOrigin]): Base of the cache storage that base.get_tma_row rows index.
  • ​zero_row (ImmPointer[Scalar[SparseRowsMHAOperand[row_width, predicated].dtype], ImmUntrackedOrigin]): A row_width-element row of zeros; unused when predicated.
  • ​indices (ImmPointer[Int32, ImmUntrackedOrigin, address_space=AddressSpace.GLOBAL]): Encoded physical indices, [batch, indices_stride], -1 unused.
  • ​valid_lengths (ImmPointer[UInt32, ImmUntrackedOrigin, address_space=AddressSpace.GLOBAL]): Per-entry count of leading valid slots; unused when predicated.
  • ​topk_lengths (ImmPointer[Int32, ImmUntrackedOrigin]): Per-entry top-k slot counts, read only when has_topk_lengths.
  • ​indices_stride (Int32):
  • ​selection_lag (Int32): Positions between the token each row was selected for and the token decoding now (predicated only).
  • ​has_topk_lengths (Bool):

Implemented traits​

AnyType, Copyable, Deinitable, DevicePassable, ImplicitlyCopyable, MHAOperand, Movable, RegisterPassable, TrivialRegisterPassable

comptime members​

device_type​

comptime device_type = SparseRowsMHAOperand[row_width, predicated]

dtype​

comptime dtype = base_t.dtype

Engine​

comptime Engine = base_t.Engine

page_size​

comptime page_size = base_t.page_size

quantization_enabled​

comptime quantization_enabled = base_t.quantization_enabled

quantization_granularity​

comptime quantization_granularity = base_t.quantization_granularity

rows_indexed​

comptime rows_indexed = True

rows_predicated​

comptime rows_predicated = predicated

scale_dtype​

comptime scale_dtype = base_t.scale_dtype

v_col_base​

comptime v_col_base = Int(0)

The first column of the value within a row of the value operand.

Non-zero only for a split KV cache, whose key and value operands are one view of rows that pack a key and then a narrower value.

Methods​

__init__​

def __init__(base: base_t, blocks_base: Pointer[Scalar[Self.dtype]], zero_row: Pointer[Scalar[Self.dtype]], indices: Pointer[Int32], valid_lengths: Pointer[UInt32], indices_stride: Int) -> Self

Wraps base with a top-k selection and explicit valid lengths.

Args:

  • ​base (base_t): The operand over the full paged cache.
  • ​blocks_base (Pointer[Scalar[Self.dtype]]): Base of the cache storage base.get_tma_row indexes.
  • ​zero_row (Pointer[Scalar[Self.dtype]]): A row_width-element row of zeros.
  • ​indices (Pointer[Int32]): Encoded physical indices, [batch, indices_stride].
  • ​valid_lengths (Pointer[UInt32]): Per-entry count of leading valid slots.
  • ​indices_stride (Int): Slots per batch entry.

def __init__(base: base_t, blocks_base: Pointer[Scalar[Self.dtype]], indices: Pointer[Int32], indices_stride: Int, topk_lengths: Pointer[Int32], has_topk_lengths: Bool, selection_lag: Int) -> Self

Wraps base with a decode top-k selection (predicated only).

Args:

  • ​base (base_t): The operand over the full paged cache.
  • ​blocks_base (Pointer[Scalar[Self.dtype]]): Base of the cache storage base.get_tma_row indexes.
  • ​indices (Pointer[Int32]): Encoded physical indices, [batch, indices_stride].
  • ​indices_stride (Int): Slots per batch entry.
  • ​topk_lengths (Pointer[Int32]): Per-entry top-k slot counts; not read unless has_topk_lengths.
  • ​has_topk_lengths (Bool): Whether topk_lengths holds the counts; otherwise every entry keeps indices_stride slots.
  • ​selection_lag (Int): Positions between the token each row was selected for and the token decoding now.

get_type_name​

static def get_type_name() -> String

Returns:

String

topk_length​

def topk_length(self, batch_idx: UInt32) -> Int

Returns entry batch_idx's top-k slot count, capped at the stride.

Args:

  • ​batch_idx (UInt32): Batch entry.

Returns:

Int: The number of slots the selection may fill.

valid_length​

def valid_length(self, batch_idx: UInt32) -> UInt32

Returns the number of leading valid slots of entry batch_idx.

Unpredicated, it reads valid_lengths. Predicated, it is the closed form SM100 sparse decode and the AMD gather use: a decode token sees the cached keys plus itself, and a plain top-k row holds min(top_k, visible - selection_lag) valid slots followed by -1 padding.

Args:

  • ​batch_idx (UInt32): Batch entry.

Returns:

UInt32: The entry's valid slot count.

indexed_row_ptr​

def indexed_row_ptr(self, batch_idx: UInt32, key_idx: UInt32) -> ImmPointer[Scalar[Self.dtype], ImmutAnyOrigin]

Returns:

ImmPointer[Scalar[Self.dtype], ImmutAnyOrigin]

indexed_row​

def indexed_row(self, batch_idx: UInt32, key_idx: UInt32) -> Tuple[Pointer[Scalar[Self.dtype], ImmutAnyOrigin], Bool]

Returns:

Tuple[Pointer[Scalar[Self.dtype], ImmutAnyOrigin], Bool]

indexed_valid_length​

def indexed_valid_length(self, batch_idx: UInt32) -> Int

Returns:

Int

check_indexed_rows​

def check_indexed_rows(self, batch_idx: UInt32)

Asserts the closed-form valid length matches the row (predicated).

The closed form assumes the row's valid slots are a prefix ended by -1 padding. A wrong length is silent in place: a predicated load skips the slot and the softmax scores whatever the LDS holds there, so check the boundary. Slot n - 1 must be valid and slot n (when inside the top-k) must be padding.

Args:

  • ​batch_idx (UInt32): Batch entry to check.

block_paged_storage​

def block_paged_storage[tile_size: Int](self, batch_idx: UInt32, start_tok_idx: UInt32, head_idx: UInt32, head_dim_idx: UInt32 = UInt32(0)) -> Self.Engine.StorageType[Self.dtype, ImmutAnyOrigin, .GENERIC]

Returns:

Self.Engine.StorageType[Self.dtype, ImmutAnyOrigin, .GENERIC]

block_paged_ptr​

def block_paged_ptr[tile_size: Int](self, batch_idx: UInt32, start_tok_idx: UInt32, head_idx: UInt32, head_dim_idx: UInt32 = UInt32(0)) -> ImmPointer[Scalar[Self.dtype], ImmutAnyOrigin]

Returns:

ImmPointer[Scalar[Self.dtype], ImmutAnyOrigin]

scales_block_paged_ptr​

def scales_block_paged_ptr(self, batch_idx: Int, start_tok_idx: Int, head_idx: Int, head_dim_idx: Int = Int(0)) -> ImmPointer[Scalar[Self.scale_dtype], ImmutAnyOrigin]

Returns:

ImmPointer[Scalar[Self.scale_dtype], ImmutAnyOrigin]

load_scale​

def load_scale[width: Int](self, batch_idx: Int, start_tok_idx: Int, head_idx: Int, head_dim_idx: Int) -> SIMD[Self.scale_dtype, width]

Returns:

SIMD[Self.scale_dtype, width]

cache_length​

def cache_length(self, batch_idx: Int) -> Int

Returns:

Int

max_context_length​

def max_context_length(self) -> UInt32

Returns:

UInt32

num_kv_rows​

def num_kv_rows(self) -> Int

Returns:

Int

row_idx​

def row_idx(self, batch_idx: UInt32, start_tok_idx: UInt32) -> UInt32

Returns:

UInt32

get_tma_row​

def get_tma_row(self, encoded_index: Int32) -> Int32

Returns:

Int32

create_tma_tile​

def create_tma_tile[swizzle_mode: TensorMapSwizzle, *, BN: Int, depth: Int, BK: Int = padded_depth[base_t.dtype, swizzle_mode, depth](), fold_chunks: Int = Int(1), row_major: Bool = False](self, ctx: DeviceContext) -> SplitLastDimTMATensorTile[Self.dtype, Coord(), swizzle_mode]

Returns:

SplitLastDimTMATensorTile[Self.dtype, Coord(), swizzle_mode]

create_scale_tma_tile​

def create_scale_tma_tile[BMN: Int](self, ctx: DeviceContext) -> TMATensorTile[Self.scale_dtype, Coord()]

Returns:

TMATensorTile[Self.scale_dtype, Coord()]

create_rope_tma_tile​

def create_rope_tma_tile[swizzle_mode: TensorMapSwizzle, *, BN: Int, BK: Int, padded_depth: Int](self, ctx: DeviceContext) -> SplitLastDimTMATensorTile[.bfloat16, Coord(), swizzle_mode]

Returns:

SplitLastDimTMATensorTile[.bfloat16, Coord(), swizzle_mode]

create_gather4_tma_tile​

def create_gather4_tma_tile[tile_width: Int, tile_stride: Int = tile_width, swizzle_mode: TensorMapSwizzle = TensorMapSwizzle.SWIZZLE_NONE, tile_height: Int = Int(4), tma_dtype: DType = SparseRowsMHAOperand[row_width, predicated].dtype, l2_promotion: TensorMapL2Promotion = TensorMapL2Promotion.NONE](self, ctx: DeviceContext) -> TMATensorTile[tma_dtype, Coord(), Coord()]

Returns:

TMATensorTile[tma_dtype, Coord(), Coord()]

create_rope_gather4_tma_tile​

def create_rope_gather4_tma_tile[tile_width: Int, padded_depth: Int, swizzle_mode: TensorMapSwizzle = TensorMapSwizzle.SWIZZLE_NONE, tile_height: Int = Int(4), l2_promotion: TensorMapL2Promotion = TensorMapL2Promotion.NONE](self, ctx: DeviceContext) -> TMATensorTile[.bfloat16, Coord(), Coord()]

Returns:

TMATensorTile[.bfloat16, Coord(), Coord()]

scales_raw_ptr​

def scales_raw_ptr(self) -> MutPointer[Float32, MutAnyOrigin]

Returns:

MutPointer[Float32, MutAnyOrigin]

Was this page helpful?