For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
generic_get_paged_cache
def generic_get_paged_cache[dtype: DType, v_head_size: Int = Int(-1)](blocks: ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=blocks.static_spec], page_stride: InputTensor[static_spec=page_stride.static_spec], cache_lengths: InputTensor[static_spec=cache_lengths.static_spec], lookup_table: InputTensor[static_spec=lookup_table.static_spec], max_prompt_length: InputTensor[static_spec=max_prompt_length.static_spec], max_cache_length: InputTensor[static_spec=max_cache_length.static_spec], out result: PagedKVCacheCollection[dtype, _paged_cache_static_params(SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(4))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(4)]())), SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(5))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(5)]())), SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(1))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(1)]())), v_head_size), SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(3))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(3)]())), MutUntrackedOrigin, ImmUntrackedOrigin, ImmUntrackedOrigin, MutUntrackedOrigin])
Returns:
def generic_get_paged_cache[v_head_size: Int = Int(-1)](blocks: TileTensor[Engine=blocks.Engine, linear_idx_type=blocks.linear_idx_type], page_stride: TileTensor[.int64, Engine=page_stride.Engine, linear_idx_type=page_stride.linear_idx_type], cache_lengths: TileTensor[.uint32, Engine=cache_lengths.Engine, linear_idx_type=cache_lengths.linear_idx_type], lookup_table: TileTensor[.uint32, Engine=lookup_table.Engine, linear_idx_type=lookup_table.linear_idx_type], max_prompt_length: TileTensor[.uint32, Engine=max_prompt_length.Engine, linear_idx_type=max_prompt_length.linear_idx_type], max_cache_length: TileTensor[.uint32, Engine=max_cache_length.Engine, linear_idx_type=max_cache_length.linear_idx_type], out result: PagedKVCacheCollection[blocks.dtype, _paged_cache_static_params(blocks.LayoutType.static_shape[Int(4)], blocks.LayoutType.static_shape[Int(5)], blocks.LayoutType.static_shape[Int(1)], v_head_size), blocks.LayoutType.static_shape[Int(3)], blocks.origin, cache_lengths.origin, lookup_table.origin, MutUntrackedOrigin])
Builds a paged KV cache collection from projected tiles of any storage.
The blocks' static shape supplies the cache's head count, head size, page size and MLA layout.
Parameters:
- v_head_size (
Int): The value head size of a split cache, whose single plane packs each key and value row; -1 otherwise.
Args:
- blocks (
TileTensor[Engine=blocks.Engine, linear_idx_type=blocks.linear_idx_type]): KV values[num_blocks, kv_dim, num_layers, page_size, num_heads, head_size]. - page_stride (
TileTensor[.int64, Engine=page_stride.Engine, linear_idx_type=page_stride.linear_idx_type]): Page-to-page distance forblocks, in elements. - cache_lengths (
TileTensor[.uint32, Engine=cache_lengths.Engine, linear_idx_type=cache_lengths.linear_idx_type]): Cache lengths per request. - lookup_table (
TileTensor[.uint32, Engine=lookup_table.Engine, linear_idx_type=lookup_table.linear_idx_type]): Page lookup table[batch_size, max_pages]. - max_prompt_length (
TileTensor[.uint32, Engine=max_prompt_length.Engine, linear_idx_type=max_prompt_length.linear_idx_type]): Max prompt length,[1]. - max_cache_length (
TileTensor[.uint32, Engine=max_cache_length.Engine, linear_idx_type=max_cache_length.linear_idx_type]): Max cache length,[1].
Returns:
def generic_get_paged_cache[dtype: DType, kv_params: KVCacheStaticParams, page_size: Int](blocks: TileTensor[dtype, linear_idx_type=blocks.linear_idx_type], page_stride: TileTensor[.int64, linear_idx_type=page_stride.linear_idx_type], cache_lengths: TileTensor[.uint32, linear_idx_type=cache_lengths.linear_idx_type], lookup_table: TileTensor[.uint32, linear_idx_type=lookup_table.linear_idx_type], max_prompt_length: TileTensor[.uint32, linear_idx_type=max_prompt_length.linear_idx_type], max_cache_length: TileTensor[.uint32, linear_idx_type=max_cache_length.linear_idx_type], out result: PagedKVCacheCollection[dtype, kv_params, page_size, blocks.origin, cache_lengths.origin, lookup_table.origin, MutUntrackedOrigin])
Returns: