IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

PagedKVCacheCollection

struct PagedKVCacheCollection[dtype_: DType, kv_params_: KVCacheStaticParams, page_size: Int, blocks_origin: MutOrigin, cache_lengths_origin: ImmOrigin, lookup_table_origin: ImmOrigin, scales_origin: MutOrigin, *, scale_dtype_: Optional[DType] = None, quantization_granularity_: Int = Int(1)]

Paged pair of key and value caches backed by a block-allocated tensor.

Stores both the K and V caches in a single 6D block tensor of shape [total_num_blocks, 2, num_layers, page_size, num_heads, head_size] (the 2 collapses to 1 under Multi-head Latent Attention), along with per-request cache lengths and a lookup table mapping logical batches to physical blocks. Supports optional quantization scales stored in a parallel tensor with head_dim_granularity as the inner dimension.

Padded pages and page_stride​

page_stride is the distance in elements from one page to the next; -1 means the pages are packed and the product of the dimensions inside a page is already right. Padding lands entirely in that one stride.

The tensor passed in still describes packed pages, so a padded page_stride addresses beyond the extent it declares. The allocator owes total_num_blocks * page_stride elements behind it.

Fields​

  • ​scales (OptionalReg[TileTensor[PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].scale_dtype, Layout[TypeList[Int64, ComptimeInt[kv_params_.num_planes()], Int64, ComptimeInt[page_size], ComptimeInt[kv_params_.num_heads], ComptimeInt[ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_)]](), TypeList[Int64, Int64, ComptimeInt[(Int((mul page_size, kv_params_.num_heads)) * ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_))], ComptimeInt[(kv_params_ * ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_))], ComptimeInt[ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_)], ComptimeInt[Int(1)]]()], scales_origin]]):
  • ​kv_cache_scales_dynamic_shape (DynamicCoord[.int64, Int(4)]):
  • ​kv_cache_scales_dynamic_strides (DynamicCoord[.int64, Int(4)]):
  • ​blocks (PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].blocks_tt_type):
  • ​cache_lengths (PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].CacheType.cache_lengths_tt_type):
  • ​lookup_table (PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].CacheType.lookup_table_tt_type):
  • ​scales_lookup_table (PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].CacheType.lookup_table_tt_type):
  • ​max_seq_length (UInt32):
  • ​max_cache_length (UInt32):
  • ​kv_cache_dynamic_shape (DynamicCoord[.int64, Int(4)]):
  • ​kv_cache_dynamic_strides (DynamicCoord[.int64, Int(4)]):

Implemented traits​

AnyType, Copyable, Deinitable, ImplicitlyCopyable, KVCollectionT, Movable

comptime members​

blocks_tt_layout​

comptime blocks_tt_layout = Layout[TypeList[Int64, ComptimeInt[kv_params_.num_planes()], Int64, ComptimeInt[page_size], ComptimeInt[kv_params_.num_heads], ComptimeInt[kv_params_.head_size]](), TypeList[Int64, Int64, ComptimeInt[(Int((mul page_size, kv_params_.num_heads)) * kv_params_)], ComptimeInt[(kv_params_ * kv_params_)], ComptimeInt[kv_params_.head_size], ComptimeInt[Int(1)]]()]

blocks_tt_type​

comptime blocks_tt_type = TileTensor[PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].dtype, Layout[TypeList[Int64, ComptimeInt[kv_params_.num_planes()], Int64, ComptimeInt[page_size], ComptimeInt[kv_params_.num_heads], ComptimeInt[kv_params_.head_size]](), TypeList[Int64, Int64, ComptimeInt[(Int((mul page_size, kv_params_.num_heads)) * kv_params_)], ComptimeInt[(kv_params_ * kv_params_)], ComptimeInt[kv_params_.head_size], ComptimeInt[Int(1)]]()], blocks_origin]

CacheType​

comptime CacheType = PagedKVCache[PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].dtype, PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].kv_params, page_size, blocks_origin, DefaultEngine, cache_lengths_origin, DefaultEngine, lookup_table_origin, DefaultEngine, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_]

dtype​

comptime dtype = dtype_

head_dim_granularity​

comptime head_dim_granularity = ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].CacheType.quantization_granularity)

kv_params​

comptime kv_params = kv_params_

name_str​

comptime name_str = "paged"

scale_dtype​

comptime scale_dtype = scale_dtype_.or_else(dtype_)

scales_tt_layout​

comptime scales_tt_layout = Layout[TypeList[Int64, ComptimeInt[kv_params_.num_planes()], Int64, ComptimeInt[page_size], ComptimeInt[kv_params_.num_heads], ComptimeInt[ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_)]](), TypeList[Int64, Int64, ComptimeInt[(Int((mul page_size, kv_params_.num_heads)) * ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_))], ComptimeInt[(kv_params_ * ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_))], ComptimeInt[ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_)], ComptimeInt[Int(1)]]()]

scales_tt_type​

comptime scales_tt_type = TileTensor[PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].scale_dtype, Layout[TypeList[Int64, ComptimeInt[kv_params_.num_planes()], Int64, ComptimeInt[page_size], ComptimeInt[kv_params_.num_heads], ComptimeInt[ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_)]](), TypeList[Int64, Int64, ComptimeInt[(Int((mul page_size, kv_params_.num_heads)) * ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_))], ComptimeInt[(kv_params_ * ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_))], ComptimeInt[ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_)], ComptimeInt[Int(1)]]()], scales_origin]

Methods​

__init__​

def __init__(out self, blocks: TileTensor[Self.dtype, blocks_origin, linear_idx_type=blocks.linear_idx_type], cache_lengths: TileTensor[.uint32, cache_lengths_origin, linear_idx_type=cache_lengths.linear_idx_type], lookup_table: TileTensor[.uint32, lookup_table_origin, linear_idx_type=lookup_table.linear_idx_type], max_seq_length: UInt32, max_cache_length: UInt32, scales: OptionalReg[TileTensor[Self.scale_dtype, Layout[TypeList[Int64, ComptimeInt[kv_params_.num_planes()], Int64, ComptimeInt[page_size], ComptimeInt[kv_params_.num_heads], ComptimeInt[ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_)]](), TypeList[Int64, Int64, ComptimeInt[(Int((mul page_size, kv_params_.num_heads)) * ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_))], ComptimeInt[(kv_params_ * ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_))], ComptimeInt[ceildiv(Int((add (mul kv_params_.fp8_tail_size, -1), kv_params_.head_size)), quantization_granularity_)], ComptimeInt[Int(1)]]()], scales_origin]] = None, scales_lookup_table: OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64, Int64](), TypeList[Int64, ComptimeInt[Int(1)]]()], lookup_table_origin]] = None, page_stride: Int = Int(-1), scales_page_stride: Int = Int(-1))

Constructs the collection from tensors of any layout with its shape, such as the graph compiler's to_tile_tensor() views.

def __init__[scales_dtype: DType = PagedKVCacheCollection[dtype_, kv_params_, page_size, blocks_origin, cache_lengths_origin, lookup_table_origin, scales_origin, scale_dtype_=scale_dtype_, quantization_granularity_=quantization_granularity_].scale_dtype](out self, blocks: TileTensor[Self.dtype, blocks_origin, linear_idx_type=blocks.linear_idx_type], cache_lengths: TileTensor[.uint32, cache_lengths_origin, linear_idx_type=cache_lengths.linear_idx_type], lookup_table: TileTensor[.uint32, lookup_table_origin, linear_idx_type=lookup_table.linear_idx_type], max_seq_length: UInt32, max_cache_length: UInt32, scales: TileTensor[scales_dtype, scales_origin, linear_idx_type=scales.linear_idx_type], scales_lookup_table: TileTensor[.uint32, lookup_table_origin, linear_idx_type=scales_lookup_table.linear_idx_type], page_stride: Int = Int(-1), scales_page_stride: Int = Int(-1))

Constructs a quantized collection from tensors of any layout with its shape. scales_lookup_table is lookup_table itself when the scales share the values' block-id space.

get_key_cache​

def get_key_cache(self, layer_idx: Int) -> Self.CacheType

Returns:

Self.CacheType

get_value_cache​

def get_value_cache(self, layer_idx: Int) -> Self.CacheType

Returns the layer's value view.

A split cache's value shares its key's plane, so this is the key view; callers add kv_params.v_col_base() to the column.

Returns:

Self.CacheType

cache_length​

def cache_length(self, bs_idx: Int) -> Int

Returns:

Int

rows_view​

def rows_view(self, first_row: Int, num_rows: Int, max_seq_length: UInt32, max_cache_length: UInt32) -> Self

Returns a view of batch rows [first_row, first_row + num_rows).

The blocks and the scales are shared with self, not copied.

Args:

  • ​first_row (Int): First batch row the view covers.
  • ​num_rows (Int): Number of batch rows the view covers.
  • ​max_seq_length (UInt32): Max sequence length the view reports.
  • ​max_cache_length (UInt32): Max cache length the view reports.

Returns:

Self: A view of the collection covering the requested rows only.

Was this page helpful?