For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
KVCacheIterator
struct KVCacheIterator[cache_t: MHAOperand, tile_size: Int, kv_num_heads: Int, depth: Int, cache_depth: Int = depth, head_dim_offset: Int = Int(0), uniform_row_math: Bool = False]
TileTensor-based DRAM tile iterator.
Returns a TileTensor with Scalar for the row dimension (valid-row count) and ComptimeInt for depth and strides.
When cache_depth != depth, the DRAM stride uses cache_depth (e.g., MLA K_rope reads 64 columns from a 576-wide cache row). head_dim_offset shifts the column start (e.g., skip to rope portion at column 512).
Parameters
- cache_t (
MHAOperand): MHA operand wrapping the KV cache pointer and dtype. - tile_size (
Int): Number of rows loaded per DRAM tile. - kv_num_heads (
Int): Number of KV heads in the cache. - depth (
Int): Per-head depth read from each cache row (head dimension). - cache_depth (
Int): Row stride of the paged KV cache in elements (defaults todepth). - head_dim_offset (
Int): Column offset within each cache row, used to skip to a sub-region such as the rope portion (defaults to0). - uniform_row_math (
Bool): Pintile_start_rowandendto wave-uniform values (defaults toFalse).
Fields
- cache (
cache_t): - end (
Int): - tile_start_row (
Int): - batch_idx (
Int): - kv_head_idx (
Int):
Implemented traits
comptime members
GmemTileLayout
comptime GmemTileLayout = Layout[TypeList[Int64, ComptimeInt[depth]](), TypeList[ComptimeInt[(kv_num_heads * cache_depth)], ComptimeInt[Int(1)]]()]
GmemTileType
comptime GmemTileType = TileTensor[cache_t.dtype, Layout[TypeList[Int64, ComptimeInt[depth]](), TypeList[ComptimeInt[(kv_num_heads * cache_depth)], ComptimeInt[Int(1)]]()], ImmutAnyOrigin, Engine=cache_t.Engine]
Methods
__init__
def __init__(out self, cache: cache_t, batch_idx: Int, kv_head_idx: Int, end: Int)
set_start_row
def set_start_row(mut self, row: Int)
Points the iterator at the tile starting at row.
With uniform_row_math, row is pinned to a wave-uniform value, which
keeps the next_tile tile clamp and the prefetch clamp (re-read the
last valid tile) in SALU for the rest of the loop.
Args:
- row (
Int): First key row of the next tile.
next_tile
def next_tile(mut self) -> Self.GmemTileType
Returns a TileTensor for the next DRAM tile.
Returns:
Self.GmemTileType
increment
def increment(mut self)
clamp_prefetch_tile
def clamp_prefetch_tile(mut self)
Rewinds to the last valid tile after a prefetch ran past end.
Call after advancing past the last tile a consumer will use: the pipeline prefetches one tile ahead even from the final tile, which would be a paged-cache OOB read, so it re-reads the previous tile instead. The data is unused; only the DMA count has to stay the same.
valid_rows
def valid_rows(self) -> Int
Returns the next tile's valid row count, clamped to [0, tile_size].
With uniform_row_math the clamp runs in 32-bit math (gfx9 SALU has
32-bit compares, but only s_cmp_eq/lg_u64 at 64-bit) and the result is
pinned to an SGPR, so the loader's bounds math on it stays scalar.
Returns: