For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
PagedKVCacheCollection
struct PagedKVCacheCollection[dtype_: DType, kv_params_: KVCacheStaticParams, page_size: Int, blocks_origin: MutOrigin, cache_lengths_origin: ImmOrigin, lookup_table_origin: ImmOrigin, scales_origin: MutOrigin, *, scale_dtype_: Optional[DType] = None, quantization_granularity_: Int = Int(1)]
Paged pair of key and value caches backed by a block-allocated tensor.
Stores both the K and V caches in a single 6D block tensor of shape
[total_num_blocks, 2, num_layers, page_size, num_heads, head_size]
(the 2 collapses to 1 under Multi-head Latent Attention), along with
per-request cache lengths and a lookup table mapping logical batches to
physical blocks. Supports optional quantization scales stored in a parallel
tensor with head_dim_granularity as the inner dimension.
Padded pages and page_stride
page_stride is the distance in elements from one page to the next; -1
means the pages are packed and the product of the dimensions inside a page
is already right. Padding lands entirely in that one stride.
The tensor passed in still describes packed pages, so a padded
page_stride addresses beyond the extent it declares. The allocator owes
total_num_blocks * page_stride elements behind it.
Fields
- scales (
OptionalReg[TileTensor[PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].scale_dtype, Layout[TypeList[Int64, ComptimeInt[kv_params_.num_planes()], Int64, ComptimeInt[page_size], ComptimeInt[kv_params_.num_heads], ComptimeInt[ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_)]](), TypeList[Int64, Int64, ComptimeInt[(Int((mul page_size, kv_params_.num_heads)) * ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_))], ComptimeInt[(kv_params_ * ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_))], ComptimeInt[ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_)], ComptimeInt[Int(1)]]()], scales_origin]]): - kv_cache_scales_dynamic_shape (
DynamicCoord[.int64, Int(4)]): - kv_cache_scales_dynamic_strides (
DynamicCoord[.int64, Int(4)]): - blocks (
PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].blocks_tt_type): - cache_lengths (
PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].CacheType.cache_lengths_tt_type): - lookup_table (
PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].CacheType.lookup_table_tt_type): - scales_lookup_table (
PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].CacheType.lookup_table_tt_type): - max_seq_length (
UInt32): - max_cache_length (
UInt32): - kv_cache_dynamic_shape (
DynamicCoord[.int64, Int(4)]): - kv_cache_dynamic_strides (
DynamicCoord[.int64, Int(4)]):
Implemented traits
AnyType,
Copyable,
Deinitable,
ImplicitlyCopyable,
KVCollectionT,
Movable
comptime members
blocks_tt_layout
comptime blocks_tt_layout = Layout[TypeList[Int64, ComptimeInt[kv_params_.num_planes()], Int64, ComptimeInt[page_size], ComptimeInt[kv_params_.num_heads], ComptimeInt[kv_params_.head_size]](), TypeList[Int64, Int64, ComptimeInt[(Int((mul page_size, kv_params_.num_heads)) * kv_params_)], ComptimeInt[(kv_params_ * kv_params_)], ComptimeInt[kv_params_.head_size], ComptimeInt[Int(1)]]()]
blocks_tt_type
comptime blocks_tt_type = TileTensor[PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].dtype, Layout[TypeList[Int64, ComptimeInt[kv_params_.num_planes()], Int64, ComptimeInt[page_size], ComptimeInt[kv_params_.num_heads], ComptimeInt[kv_params_.head_size]](), TypeList[Int64, Int64, ComptimeInt[(Int((mul page_size, kv_params_.num_heads)) * kv_params_)], ComptimeInt[(kv_params_ * kv_params_)], ComptimeInt[kv_params_.head_size], ComptimeInt[Int(1)]]()], blocks_origin]
CacheType
comptime CacheType = PagedKVCache[PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].dtype, PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].kv_params, page_size, blocks_origin, DefaultEngine, cache_lengths_origin, DefaultEngine, lookup_table_origin, DefaultEngine, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_]
dtype
comptime dtype = dtype_
head_dim_granularity
comptime head_dim_granularity = ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].CacheType.quantization_granularity)
kv_params
comptime kv_params = kv_params_
name_str
comptime name_str = "paged"
scale_dtype
comptime scale_dtype = scale_dtype_.or_else(dtype_)
scales_tt_layout
comptime scales_tt_layout = Layout[TypeList[Int64, ComptimeInt[kv_params_.num_planes()], Int64, ComptimeInt[page_size], ComptimeInt[kv_params_.num_heads], ComptimeInt[ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_)]](), TypeList[Int64, Int64, ComptimeInt[(Int((mul page_size, kv_params_.num_heads)) * ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_))], ComptimeInt[(kv_params_ * ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_))], ComptimeInt[ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_)], ComptimeInt[Int(1)]]()]
scales_tt_type
comptime scales_tt_type = TileTensor[PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].scale_dtype, Layout[TypeList[Int64, ComptimeInt[kv_params_.num_planes()], Int64, ComptimeInt[page_size], ComptimeInt[kv_params_.num_heads], ComptimeInt[ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_)]](), TypeList[Int64, Int64, ComptimeInt[(Int((mul page_size, kv_params_.num_heads)) * ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_))], ComptimeInt[(kv_params_ * ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_))], ComptimeInt[ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_)], ComptimeInt[Int(1)]]()], scales_origin]
Methods
__init__
def __init__(out self, blocks: TileTensor[Self.dtype, blocks_origin, linear_idx_type=blocks.linear_idx_type], cache_lengths: TileTensor[.uint32, cache_lengths_origin, linear_idx_type=cache_lengths.linear_idx_type], lookup_table: TileTensor[.uint32, lookup_table_origin, linear_idx_type=lookup_table.linear_idx_type], max_seq_length: UInt32, max_cache_length: UInt32, scales: OptionalReg[TileTensor[Self.scale_dtype, Layout[TypeList[Int64, ComptimeInt[kv_params_.num_planes()], Int64, ComptimeInt[page_size], ComptimeInt[kv_params_.num_heads], ComptimeInt[ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_)]](), TypeList[Int64, Int64, ComptimeInt[(Int((mul page_size, kv_params_.num_heads)) * ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_))], ComptimeInt[(kv_params_ * ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_))], ComptimeInt[ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_)], ComptimeInt[Int(1)]]()], scales_origin]] = None, scales_lookup_table: OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64, Int64](), TypeList[Int64, ComptimeInt[Int(1)]]()], lookup_table_origin]] = None, page_stride: Int = Int(-1), scales_page_stride: Int = Int(-1))
Constructs the collection from tensors of any layout with its shape, such as the graph compiler's to_tile_tensor() views.
def __init__[scales_dtype: DType = PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].scale_dtype](out self, blocks: TileTensor[Self.dtype, blocks_origin, linear_idx_type=blocks.linear_idx_type], cache_lengths: TileTensor[.uint32, cache_lengths_origin, linear_idx_type=cache_lengths.linear_idx_type], lookup_table: TileTensor[.uint32, lookup_table_origin, linear_idx_type=lookup_table.linear_idx_type], max_seq_length: UInt32, max_cache_length: UInt32, scales: TileTensor[scales_dtype, scales_origin, linear_idx_type=scales.linear_idx_type], scales_lookup_table: TileTensor[.uint32, lookup_table_origin, linear_idx_type=scales_lookup_table.linear_idx_type], page_stride: Int = Int(-1), scales_page_stride: Int = Int(-1))
Constructs a quantized collection from tensors of any layout with its shape. scales_lookup_table is lookup_table itself when the scales share the values' block-id space.
get_key_cache
def get_key_cache(self, layer_idx: Int) -> Self.CacheType
Returns:
Self.CacheType
get_value_cache
def get_value_cache(self, layer_idx: Int) -> Self.CacheType
Returns the layer's value view.
A split cache's value shares its key's plane, so this is the key view;
callers add kv_params.v_col_base() to the column.
Returns:
Self.CacheType
cache_length
rows_view
def rows_view(self, first_row: Int, num_rows: Int, max_seq_length: UInt32, max_cache_length: UInt32) -> Self
Returns a view of batch rows [first_row, first_row + num_rows).
The blocks and the scales are shared with self, not copied.
Args:
- first_row (
Int): First batch row the view covers. - num_rows (
Int): Number of batch rows the view covers. - max_seq_length (
UInt32): Max sequence length the view reports. - max_cache_length (
UInt32): Max cache length the view reports.
Returns:
Self: A view of the collection covering the requested rows only.