IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

SparseRowsMHAOperand

struct SparseRowsMHAOperand[base_t: MHAOperand, //, row_width: Int]

A K operand whose key rows are a top-k selection from a paged cache.

Key j of batch entry b is the cache row named by the selection's j-th index, so an attention kernel can read the selected rows in place instead of from a gathered copy. Keys at or past the entry's valid length, and -1 slots, read zero_row: the attention mask drops their scores, and zeros keep 0 * value finite where uninitialized cache bytes might not be.

Everything except the row addressing forwards to base.

Parameters​

  • ​base_t (MHAOperand): The operand over the full paged cache.
  • ​row_width (Int): Elements per cache row (all heads), the stride between consecutive physical rows.

Fields​

  • ​base (base_t):
  • ​blocks_base (Pointer[Scalar[SparseRowsMHAOperand[row_width].dtype], ImmUntrackedOrigin]): Base of the cache storage that base.get_tma_row rows index.
  • ​zero_row (Pointer[Scalar[SparseRowsMHAOperand[row_width].dtype], ImmUntrackedOrigin]): A row_width-element row of zeros.
  • ​indices (Pointer[Int32, ImmUntrackedOrigin]): Encoded physical indices, [batch, indices_stride], -1 unused.
  • ​valid_lengths (Pointer[UInt32, ImmUntrackedOrigin]): Per-entry count of leading valid slots.
  • ​indices_stride (Int32):

Implemented traits​

AnyType, Copyable, Deinitable, DevicePassable, ImplicitlyCopyable, MHAOperand, Movable, RegisterPassable, TrivialRegisterPassable

comptime members​

device_type​

comptime device_type = SparseRowsMHAOperand[row_width]

dtype​

comptime dtype = base_t.dtype

Engine​

comptime Engine = base_t.Engine

page_size​

comptime page_size = base_t.page_size

quantization_enabled​

comptime quantization_enabled = base_t.quantization_enabled

quantization_granularity​

comptime quantization_granularity = base_t.quantization_granularity

rows_indexed​

comptime rows_indexed = True

scale_dtype​

comptime scale_dtype = base_t.scale_dtype

Methods​

__init__​

def __init__(base: base_t, blocks_base: Pointer[Scalar[Self.dtype]], zero_row: Pointer[Scalar[Self.dtype]], indices: Pointer[Int32], valid_lengths: Pointer[UInt32], indices_stride: Int) -> Self

Wraps base with a top-k selection.

Args:

  • ​base (base_t): The operand over the full paged cache.
  • ​blocks_base (Pointer[Scalar[Self.dtype]]): Base of the cache storage base.get_tma_row indexes.
  • ​zero_row (Pointer[Scalar[Self.dtype]]): A row_width-element row of zeros.
  • ​indices (Pointer[Int32]): Encoded physical indices, [batch, indices_stride].
  • ​valid_lengths (Pointer[UInt32]): Per-entry count of leading valid slots.
  • ​indices_stride (Int): Slots per batch entry.

get_type_name​

static def get_type_name() -> String

Returns:

String

indexed_row_ptr​

def indexed_row_ptr(self, batch_idx: UInt32, key_idx: UInt32) -> Pointer[Scalar[Self.dtype], ImmutAnyOrigin]

Returns:

Pointer[Scalar[Self.dtype], ImmutAnyOrigin]

block_paged_storage​

def block_paged_storage[tile_size: Int](self, batch_idx: UInt32, start_tok_idx: UInt32, head_idx: UInt32, head_dim_idx: UInt32 = UInt32(0)) -> Self.Engine.StorageType[False, ImmUnsafeAnyOrigin, Self.dtype, ImmutAnyOrigin, AddressSpace.GENERIC]

Returns:

Self.Engine.StorageType[False, ImmUnsafeAnyOrigin, Self.dtype, ImmutAnyOrigin, AddressSpace.GENERIC]

block_paged_ptr​

def block_paged_ptr[tile_size: Int](self, batch_idx: UInt32, start_tok_idx: UInt32, head_idx: UInt32, head_dim_idx: UInt32 = UInt32(0)) -> Pointer[Scalar[Self.dtype], ImmutAnyOrigin]

Returns:

Pointer[Scalar[Self.dtype], ImmutAnyOrigin]

scales_block_paged_ptr​

def scales_block_paged_ptr(self, batch_idx: Int, start_tok_idx: Int, head_idx: Int, head_dim_idx: Int = Int(0)) -> Pointer[Scalar[Self.scale_dtype], ImmutAnyOrigin]

Returns:

Pointer[Scalar[Self.scale_dtype], ImmutAnyOrigin]

load_scale​

def load_scale[width: Int](self, batch_idx: Int, start_tok_idx: Int, head_idx: Int, head_dim_idx: Int) -> SIMD[Self.scale_dtype, width]

Returns:

SIMD[Self.scale_dtype, width]

cache_length​

def cache_length(self, batch_idx: Int) -> Int

Returns:

Int

max_context_length​

def max_context_length(self) -> UInt32

Returns:

UInt32

num_kv_rows​

def num_kv_rows(self) -> Int

Returns:

Int

row_idx​

def row_idx(self, batch_idx: UInt32, start_tok_idx: UInt32) -> UInt32

Returns:

UInt32

get_tma_row​

def get_tma_row(self, encoded_index: Int32) -> Int32

Returns:

Int32

create_tma_tile​

def create_tma_tile[swizzle_mode: TensorMapSwizzle, *, BN: Int, depth: Int, BK: Int = padded_depth[base_t.dtype, swizzle_mode, depth](), fold_chunks: Int = Int(1), row_major: Bool = False](self, ctx: DeviceContextBase) -> TMATensorTile[Self.dtype, _padded_shape[Self.dtype, Coord(), swizzle_mode](), _ragged_shape[Self.dtype, Coord(), swizzle_mode]()]

Returns:

TMATensorTile[Self.dtype, _padded_shape[Self.dtype, Coord(), swizzle_mode](), _ragged_shape[Self.dtype, Coord(), swizzle_mode]()]

create_scale_tma_tile​

def create_scale_tma_tile[BMN: Int](self, ctx: DeviceContextBase) -> TMATensorTile[Self.scale_dtype, Coord()]

Returns:

TMATensorTile[Self.scale_dtype, Coord()]

create_rope_tma_tile​

def create_rope_tma_tile[swizzle_mode: TensorMapSwizzle, *, BN: Int, BK: Int, padded_depth: Int](self, ctx: DeviceContextBase) -> TMATensorTile[.bfloat16, _padded_shape[DType.bfloat16, Coord(), swizzle_mode](), _ragged_shape[DType.bfloat16, Coord(), swizzle_mode]()]

Returns:

TMATensorTile[.bfloat16, _padded_shape[DType.bfloat16, Coord(), swizzle_mode](), _ragged_shape[DType.bfloat16, Coord(), swizzle_mode]()]

create_gather4_tma_tile​

def create_gather4_tma_tile[tile_width: Int, tile_stride: Int = tile_width, swizzle_mode: TensorMapSwizzle = TensorMapSwizzle.SWIZZLE_NONE, tile_height: Int = Int(4), tma_dtype: DType = SparseRowsMHAOperand[row_width].dtype, l2_promotion: TensorMapL2Promotion = TensorMapL2Promotion.NONE](self, ctx: DeviceContextBase) -> TMATensorTile[tma_dtype, Coord(), Coord()]

Returns:

TMATensorTile[tma_dtype, Coord(), Coord()]

create_rope_gather4_tma_tile​

def create_rope_gather4_tma_tile[tile_width: Int, padded_depth: Int, swizzle_mode: TensorMapSwizzle = TensorMapSwizzle.SWIZZLE_NONE, tile_height: Int = Int(4), l2_promotion: TensorMapL2Promotion = TensorMapL2Promotion.NONE](self, ctx: DeviceContextBase) -> TMATensorTile[.bfloat16, Coord(), Coord()]

Returns:

TMATensorTile[.bfloat16, Coord(), Coord()]

scales_raw_ptr​

def scales_raw_ptr(self) -> Pointer[Float32, MutAnyOrigin]

Returns:

Pointer[Float32, MutAnyOrigin]

Was this page helpful?