For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
ContinuousBatchingKVCache
struct ContinuousBatchingKVCache[dtype_: DType, kv_params_: KVCacheStaticParams, blocks_origin: MutOrigin, blocks_engine: TensorEngine, cache_lengths_origin: ImmOrigin, cache_lengths_engine_: TensorEngine, lookup_table_origin: ImmOrigin, lookup_table_engine: TensorEngine]
Wrapper for the ContinuousKVCache of a given layer in the transformer model.
This abstracts the Pointer indirection for accessing the ContinuousKVCache for a given batch entry.
THIS IS THE TYPE THAT IS PASSED TO KV PROJECTION AND FLASH ATTENTION KERNELS.
Parameters
- dtype_ (
DType): The dtype of the kv-cache. - kv_params_ (
KVCacheStaticParams): The kv-cache static parameters. - blocks_origin (
MutOrigin): Origin of the KV cache blocks buffer. - blocks_engine (
TensorEngine): Engine policy of the KV cache blocks buffer. - cache_lengths_origin (
ImmOrigin): Origin of the cache lengths buffer. - cache_lengths_engine_ (
TensorEngine): Engine policy of the cache lengths buffer. - lookup_table_origin (
ImmOrigin): Origin of the lookup table buffer. - lookup_table_engine (
TensorEngine): Engine policy of the lookup table buffer.
Fields
- blocks (
ContinuousBatchingKVCache[dtype_, kv_params_, blocks_origin, blocks_engine, cache_lengths_origin, cache_lengths_engine_, lookup_table_origin, lookup_table_engine].blocks_tt_type): - cache_lengths (
ContinuousBatchingKVCache[dtype_, kv_params_, blocks_origin, blocks_engine, cache_lengths_origin, cache_lengths_engine_, lookup_table_origin, lookup_table_engine].cache_lengths_tt_type): - lookup_table (
ContinuousBatchingKVCache[dtype_, kv_params_, blocks_origin, blocks_engine, cache_lengths_origin, cache_lengths_engine_, lookup_table_origin, lookup_table_engine].lookup_table_tt_type): - max_seq_length (
UInt32): - max_cache_length (
UInt32):
Implemented traits
AnyType,
Copyable,
Deinitable,
DevicePassable,
ImplicitlyCopyable,
KVCacheT,
Movable,
RegisterPassable,
TrivialRegisterPassable
comptime members
blocks_tt_layout
comptime blocks_tt_layout = Layout[TypeList[Int64, Int64, ComptimeInt[kv_params_.num_heads], ComptimeInt[kv_params_.head_size]](), TypeList[Int64, ComptimeInt[(kv_params_ * kv_params_)], ComptimeInt[kv_params_.head_size], ComptimeInt[Int(1)]]()]
blocks_tt_type
comptime blocks_tt_type = TileTensor[ContinuousBatchingKVCache[dtype_, kv_params_, blocks_origin, blocks_engine, cache_lengths_origin, cache_lengths_engine_, lookup_table_origin, lookup_table_engine].dtype, Layout[TypeList[Int64, Int64, ComptimeInt[kv_params_.num_heads], ComptimeInt[kv_params_.head_size]](), TypeList[Int64, ComptimeInt[(kv_params_ * kv_params_)], ComptimeInt[kv_params_.head_size], ComptimeInt[Int(1)]]()], blocks_origin, Engine=blocks_engine]
cache_lengths_engine
comptime cache_lengths_engine = cache_lengths_engine_
cache_lengths_tt_layout
comptime cache_lengths_tt_layout = Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()]
cache_lengths_tt_type
comptime cache_lengths_tt_type = TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], cache_lengths_origin, Engine=ContinuousBatchingKVCache[dtype_, kv_params_, blocks_origin, blocks_engine, cache_lengths_origin, cache_lengths_engine_, lookup_table_origin, lookup_table_engine].cache_lengths_engine]
device_type
comptime device_type = ContinuousBatchingKVCache[dtype_, kv_params_, blocks_origin, blocks_engine, cache_lengths_origin, cache_lengths_engine_, lookup_table_origin, lookup_table_engine]
dtype
comptime dtype = dtype_
Engine
comptime Engine = blocks_engine
kv_params
comptime kv_params = kv_params_
lookup_table_tt_layout
comptime lookup_table_tt_layout = Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()]
lookup_table_tt_type
comptime lookup_table_tt_type = TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], lookup_table_origin, Engine=lookup_table_engine]
page_size_
comptime page_size_ = 0
quantization_enabled
comptime quantization_enabled = False
quantization_granularity
comptime quantization_granularity = 1
scale_dtype
comptime scale_dtype = DType.float32
Methods
__init__
def __init__(blocks: TileTensor[Self.dtype, blocks_origin, Engine=blocks_engine, linear_idx_type=blocks.linear_idx_type], cache_lengths: TileTensor[.uint32, cache_lengths_origin, Engine=Self.cache_lengths_engine, linear_idx_type=cache_lengths.linear_idx_type], lookup_table: TileTensor[.uint32, lookup_table_origin, Engine=lookup_table_engine, linear_idx_type=lookup_table.linear_idx_type], max_seq_length: UInt32, max_cache_length: UInt32) -> Self
Constructs the cache from tensors of any layout with its shape.
get_type_name
max_tile_size
cache_lengths_nd
def cache_lengths_nd(self) -> Self.cache_lengths_tt_type
Returns:
Self.cache_lengths_tt_type
cache_length
load
def load[width: Int, output_dtype: DType = ContinuousBatchingKVCache[dtype_, kv_params_, blocks_origin, blocks_engine, cache_lengths_origin, cache_lengths_engine_, lookup_table_origin, lookup_table_engine].dtype](self, bs: Int, head_idx: Int, tok_idx: Int, head_dim_idx: Int) -> SIMD[output_dtype, width]
Returns:
store
def store(self, bs: Int, head_idx: Int, tok_idx: Int, head_dim_idx: Int, val: SIMD[Self.dtype])
load_scale
def load_scale[width: Int](self, bs: Int, head_idx: Int, tok_idx: Int, head_dim_idx: Int) -> SIMD[.float32, width]
Loads a quantization scale from the given index.
Note: ContinuousBatchingKVCache does not support KVCache quantization.
Parameters:
- width (
Int): The SIMD vector width in elements.
Args:
- bs (
Int): The batch index selecting which request in the batch. - head_idx (
Int): The attention head index. - tok_idx (
Int): The token index within the sequence. - head_dim_idx (
Int): The element offset within the head dimension.
Returns:
store_scale
def store_scale[scales_dtype: DType = .float32, width: Int = Int(1)](self, bs: Int, head_idx: Int, tok_idx: Int, head_dim_idx: Int, scales: SIMD[scales_dtype, width])
Stores the quantization scales at the given index.
Note: ContinuousBatchingKVCache does not support KVCache quantization.
load_quantized
def load_quantized[width: Int](self, bs: Int, head_idx: Int, tok_idx: Int, head_dim_idx: Int) -> SIMD[Self.dtype, width]
Loads a quantized element from the given index.
Note: ContinuousBatchingKVCache does not support KVCache quantization.
Parameters:
- width (
Int): The SIMD vector width in elements.
Args:
- bs (
Int): The batch index selecting which request in the batch. - head_idx (
Int): The attention head index. - tok_idx (
Int): The token index within the sequence. - head_dim_idx (
Int): The element offset within the head dimension.
Returns:
empty_cache
def empty_cache(self) -> Bool
Returns true if the cache_lengths for all requests is 0, false otherwise.
Returns:
max_prompt_length
def max_prompt_length(self) -> UInt32
Returns the maximum sequence length across all batches of the current request.
Returns:
max_context_length
def max_context_length(self) -> UInt32
Returns the maximum cache length used across all batches of the current request.
Returns:
get_tma_row
def get_tma_row(self, encoded_index: Int32) -> Int32
Convert an encoded sparse index to a physical TMA row.
For non-paged caches the encoded index is already the row, so this is an identity operation.
Args:
- encoded_index (
Int32): The encoded sparse index to convert.
Returns:
num_kv_rows
def num_kv_rows(self) -> Int
Returns the total number of virtual rows in this KV cache view.
Returns:
row_idx
def row_idx(self, batch_idx: UInt32, tok_idx: UInt32) -> UInt32
Returns the row idx when viewing the memory as a matrix.
Returns:
kv_tma_coords
def kv_tma_coords(self, batch_idx: UInt32, tok_idx: UInt32) -> Tuple[Int32, Int32]
The (row_in_block, block) coordinate of a token's KV tile.
What :meth:create_paged_tma_tile is addressed by. A batch entry's
sequence is one whole block, so the row is the token. The trait's
flat default folds the block into the row, past that descriptor's
rows for every block but the first.
Args:
- batch_idx (
UInt32): Batch entry whose lookup table is read. - tok_idx (
UInt32): Token within the entry.
Returns:
Tuple[Int32, Int32]: The row within the block, and the block.
scale_row_idx
def scale_row_idx(self, batch_idx: UInt32, start_tok_idx: UInt32) -> UInt32
Not supported: this cache carries no quantization scales.
Returns:
create_index_scale_tma_tile
def create_index_scale_tma_tile[TILE: Int](self, ctx: DeviceContext) -> TMATensorTile[.float32, Coord()]
Not supported: this cache carries no quantization scales.
Returns:
create_paged_tma_tile
def create_paged_tma_tile[swizzle_mode: TensorMapSwizzle, *, BN: Int, BK: Int = padded_depth[dtype_, swizzle_mode, kv_params_.head_size](), pool_row_tokens: Int = Int(1)](self, ctx: DeviceContext, out res: SplitLastDimTMATensorTile[Self.dtype, Coord(), swizzle_mode])
Builds the split descriptor over this cache's blocks.
The continuous cache is [num_blocks, num_layers, seq_len, num_heads, head_size], subset at the layer, so a block's rows are its sequence
and the pitch is the whole per-block span.
Parameters:
- swizzle_mode (
TensorMapSwizzle): TMA swizzle for the innermost dimension. - BN (
Int): Rows per tile. - BK (
Int): Depth per tile, padded to the swizzle granularity. - pool_row_tokens (
Int): Must be 1; only a paged cache holds a key spread over token rows.
Args:
- ctx (
DeviceContext): The CUDA device context used to create the TMA descriptor.
Returns:
SplitLastDimTMATensorTile[Self.dtype, Coord(), swizzle_mode]: The TMA descriptor.
create_tma_tile
def create_tma_tile[swizzle_mode: TensorMapSwizzle, *, BN: Int, BK: Int = padded_depth[dtype_, swizzle_mode, kv_params_.head_size](), fold_chunks: Int = Int(1), row_major: Bool = False](self, ctx: DeviceContext) -> SplitLastDimTMATensorTile[Self.dtype, Coord(), swizzle_mode]
Creates a TMA tile for this KV cache.
Parameters:
- swizzle_mode (
TensorMapSwizzle): TMA swizzle mode for shared memory access pattern. - BN (
Int): Number of rows in the SMEM tile box. - BK (
Int): Contiguous depth of the tile in elements. Defaults tohead_sizealigned up to the swizzle granularity. - fold_chunks (
Int): Depth-chunk fold factor.1(default) keeps the original 3D descriptor;>= 2folds the depth chunks into one rank-4cp.async.bulk.tensor. - row_major (
Bool): WhenTruewithfold_chunks >= 2, builds the rank-5 chunk-inner box;False(default) builds the rank-4 chunk-outer box.
Args:
- ctx (
DeviceContext): The CUDA device context used to create the TMA descriptor.
Returns:
SplitLastDimTMATensorTile[Self.dtype, Coord(), swizzle_mode]
create_gather4_tma_tile
def create_gather4_tma_tile[*, tile_height: Int = Int(4), tile_width: Int, tile_stride: Int = tile_width, swizzle_mode: TensorMapSwizzle = TensorMapSwizzle.SWIZZLE_NONE, tma_dtype: DType = ContinuousBatchingKVCache[dtype_, kv_params_, blocks_origin, blocks_engine, cache_lengths_origin, cache_lengths_engine_, lookup_table_origin, lookup_table_engine].dtype, l2_promotion: TensorMapL2Promotion = TensorMapL2Promotion.NONE](self, ctx: DeviceContext) -> TMATensorTile[tma_dtype, Coord(), Coord()]
Creates a 2D TMA gather4 descriptor for this KV cache.
The descriptor views the KV cache as a flat 2D matrix of
[num_kv_rows, tile_width] and is configured for gather4 operations
that load 4 non-contiguous rows per TMA instruction. The box width
is derived from the swizzle mode; for SWIZZLE_NONE it equals
tile_width.
When tma_dtype differs from Self.dtype, the underlying data
pointer is bitcast to tma_dtype at descriptor creation time.
Parameters:
- tile_height (
Int): Number of rows in the tile. Must be a multiple of 4. Defaults to 4 for backward compatibility. - tile_width (
Int): Number of elements per row to load (box width) intma_dtypeelements. - tile_stride (
Int): Row stride in elements in global memory. Defaults totile_width. Use a larger value when the global row is wider than the portion to load. - swizzle_mode (
TensorMapSwizzle): TMA swizzle mode for shared memory access pattern. Defaults to SWIZZLE_NONE. - tma_dtype (
DType): The data type used for the TMA descriptor. Defaults toSelf.dtype. When different, the pointer is bitcast. - l2_promotion (
TensorMapL2Promotion): L2 cache promotion hint for TMA loads. Defaults to NONE.
Args:
- ctx (
DeviceContext): The CUDA device context used to create the TMA descriptor.
Returns:
TMATensorTile[tma_dtype, Coord(), Coord()]: A TMATensorTile with box width derived from the swizzle mode.
create_rope_tma_tile
def create_rope_tma_tile[swizzle_mode: TensorMapSwizzle, *, BN: Int, BK: Int, padded_depth: Int](self, ctx: DeviceContext, out tma: SplitLastDimTMATensorTile[.bfloat16, Coord(), swizzle_mode])
Not supported for ContinuousBatchingKVCache.
Parameters:
- swizzle_mode (
TensorMapSwizzle): TMA swizzle mode for shared memory access pattern. - BN (
Int): Number of rows in the SMEM tile box. - BK (
Int): Number of BF16 rope elements per row (the rope depth). - padded_depth (
Int): Byte offset from row start to the rope data.
Args:
- ctx (
DeviceContext): The CUDA device context used to create the TMA descriptor.
Returns:
create_rope_gather4_tma_tile
def create_rope_gather4_tma_tile[*, tile_height: Int = Int(4), tile_width: Int, padded_depth: Int, swizzle_mode: TensorMapSwizzle = TensorMapSwizzle.SWIZZLE_NONE, l2_promotion: TensorMapL2Promotion = TensorMapL2Promotion.NONE](self, ctx: DeviceContext) -> TMATensorTile[.bfloat16, Coord(), Coord()]
Not supported for ContinuousBatchingKVCache.
Returns:
block_paged_ptr
def block_paged_ptr[tile_size: Int](self, batch_idx: Int, start_tok_idx: Int, head_idx: Int, head_dim_idx: Int = Int(0)) -> MutPointer[Scalar[Self.dtype], MutAnyOrigin]
Returns:
block_paged_storage
def block_paged_storage[tile_size: Int](self, batch_idx: Int, start_tok_idx: Int, head_idx: Int, head_dim_idx: Int = Int(0)) -> Self.Engine.StorageType[Self.dtype, MutAnyOrigin, .GENERIC]
Offsets the blocks handle to a block through the engine.
Parameters:
- tile_size (
Int): Tile size in rows (unused; kept for trait parity).
Args:
- batch_idx (
Int): Batch index of the request. - start_tok_idx (
Int): Starting token index within the batch. - head_idx (
Int): KV head index. - head_dim_idx (
Int): Index along the head dimension (defaults to 0).
Returns:
Self.Engine.StorageType[Self.dtype, MutAnyOrigin, .GENERIC]: The blocks storage handle advanced to the block.
scales_block_paged_ptr
def scales_block_paged_ptr(self, batch_idx: Int, start_tok_idx: Int, head_idx: Int, head_dim_idx: Int = Int(0)) -> MutPointer[Float32, MutAnyOrigin]
Returns a pointer to the scales block at the requested indices.
Note: ContinuousBatchingKVCache does not support KVCache quantization. This function returns a dangling pointer.
Returns:
scales_raw_ptr
def scales_raw_ptr(self) -> MutPointer[Float32, MutAnyOrigin]
Returns a dangling pointer. ContinuousBatchingKVCache does not support quantization.
Returns:
blocks_raw_ptr
def blocks_raw_ptr(self) -> MutPointer[Scalar[Self.dtype], MutAnyOrigin]
Returns the base pointer to the blocks tensor.
Returns: