IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

KVCacheIterator

struct KVCacheIterator[cache_t: MHAOperand, tile_size: Int, kv_num_heads: Int, depth: Int, cache_depth: Int = depth, head_dim_offset: Int = Int(0), uniform_row_math: Bool = False]

TileTensor-based DRAM tile iterator.

Returns a TileTensor with Scalar for the row dimension (valid-row count) and ComptimeInt for depth and strides.

When cache_depth != depth, the DRAM stride uses cache_depth (e.g., MLA K_rope reads 64 columns from a 576-wide cache row). head_dim_offset shifts the column start (e.g., skip to rope portion at column 512).

Parameters​

  • ​cache_t (MHAOperand): MHA operand wrapping the KV cache pointer and dtype.
  • ​tile_size (Int): Number of rows loaded per DRAM tile.
  • ​kv_num_heads (Int): Number of KV heads in the cache.
  • ​depth (Int): Per-head depth read from each cache row (head dimension).
  • ​cache_depth (Int): Row stride of the paged KV cache in elements (defaults to depth).
  • ​head_dim_offset (Int): Column offset within each cache row, used to skip to a sub-region such as the rope portion (defaults to 0).
  • ​uniform_row_math (Bool): Pin tile_start_row and end to wave-uniform values (defaults to False).

Fields​

  • ​cache (cache_t):
  • ​end (Int):
  • ​tile_start_row (Int):
  • ​batch_idx (Int):
  • ​kv_head_idx (Int):

Implemented traits​

AnyType, Deinitable, Movable

comptime members​

GmemTileLayout​

comptime GmemTileLayout = Layout[TypeList[Int64, ComptimeInt[depth]](), TypeList[ComptimeInt[(kv_num_heads * cache_depth)], ComptimeInt[Int(1)]]()]

GmemTileType​

comptime GmemTileType = TileTensor[cache_t.dtype, Layout[TypeList[Int64, ComptimeInt[depth]](), TypeList[ComptimeInt[(kv_num_heads * cache_depth)], ComptimeInt[Int(1)]]()], ImmutAnyOrigin, Engine=cache_t.Engine]

Methods​

__init__​

def __init__(out self, cache: cache_t, batch_idx: Int, kv_head_idx: Int, end: Int)

set_start_row​

def set_start_row(mut self, row: Int)

Points the iterator at the tile starting at row.

With uniform_row_math, row is pinned to a wave-uniform value, which keeps the next_tile tile clamp and the prefetch clamp (re-read the last valid tile) in SALU for the rest of the loop.

Args:

  • ​row (Int): First key row of the next tile.

next_tile​

def next_tile(mut self) -> Self.GmemTileType

Returns a TileTensor for the next DRAM tile.

Returns:

Self.GmemTileType

increment​

def increment(mut self)

clamp_prefetch_tile​

def clamp_prefetch_tile(mut self)

Rewinds to the last valid tile after a prefetch ran past end.

Call after advancing past the last tile a consumer will use: the pipeline prefetches one tile ahead even from the final tile, which would be a paged-cache OOB read, so it re-reads the previous tile instead. The data is unused; only the DMA count has to stay the same.

valid_rows​

def valid_rows(self) -> Int

Returns the next tile's valid row count, clamped to [0, tile_size].

With uniform_row_math the clamp runs in 32-bit math (gfx9 SALU has 32-bit compares, but only s_cmp_eq/lg_u64 at 64-bit) and the result is pinned to an SGPR, so the loader's bounds math on it stays scalar.

Returns:

Int

Was this page helpful?