IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

generic_get_paged_cache

def generic_get_paged_cache[dtype: DType, v_head_size: Int = Int(-1)](blocks: ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=blocks.static_spec], page_stride: InputTensor[static_spec=page_stride.static_spec], cache_lengths: InputTensor[static_spec=cache_lengths.static_spec], lookup_table: InputTensor[static_spec=lookup_table.static_spec], max_prompt_length: InputTensor[static_spec=max_prompt_length.static_spec], max_cache_length: InputTensor[static_spec=max_cache_length.static_spec], out result: PagedKVCacheCollection[dtype, _paged_cache_static_params(SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(4))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(4)]())), SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(5))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(5)]())), SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(1))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(1)]())), v_head_size), SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(3))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(3)]())), MutUntrackedOrigin, ImmUntrackedOrigin, ImmUntrackedOrigin, MutUntrackedOrigin])

Returns:

PagedKVCacheCollection[dtype, _paged_cache_static_params(SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(4))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(4)]())), SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(5))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(5)]())), SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(1))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(1)]())), v_head_size), SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(3))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(3)]())), MutUntrackedOrigin, ImmUntrackedOrigin, ImmUntrackedOrigin, MutUntrackedOrigin]

def generic_get_paged_cache[v_head_size: Int = Int(-1)](blocks: TileTensor[Engine=blocks.Engine, linear_idx_type=blocks.linear_idx_type], page_stride: TileTensor[.int64, Engine=page_stride.Engine, linear_idx_type=page_stride.linear_idx_type], cache_lengths: TileTensor[.uint32, Engine=cache_lengths.Engine, linear_idx_type=cache_lengths.linear_idx_type], lookup_table: TileTensor[.uint32, Engine=lookup_table.Engine, linear_idx_type=lookup_table.linear_idx_type], max_prompt_length: TileTensor[.uint32, Engine=max_prompt_length.Engine, linear_idx_type=max_prompt_length.linear_idx_type], max_cache_length: TileTensor[.uint32, Engine=max_cache_length.Engine, linear_idx_type=max_cache_length.linear_idx_type], out result: PagedKVCacheCollection[blocks.dtype, _paged_cache_static_params(blocks.LayoutType.static_shape[Int(4)], blocks.LayoutType.static_shape[Int(5)], blocks.LayoutType.static_shape[Int(1)], v_head_size), blocks.LayoutType.static_shape[Int(3)], blocks.origin, cache_lengths.origin, lookup_table.origin, MutUntrackedOrigin])

Builds a paged KV cache collection from projected tiles of any storage.

The blocks' static shape supplies the cache's head count, head size, page size and MLA layout.

Parameters:

  • ​v_head_size (Int): The value head size of a split cache, whose single plane packs each key and value row; -1 otherwise.

Args:

Returns:

PagedKVCacheCollection[blocks.dtype, _paged_cache_static_params(blocks.LayoutType.static_shape[Int(4)], blocks.LayoutType.static_shape[Int(5)], blocks.LayoutType.static_shape[Int(1)], v_head_size), blocks.LayoutType.static_shape[Int(3)], blocks.origin, cache_lengths.origin, lookup_table.origin, MutUntrackedOrigin]: The collection over blocks.

def generic_get_paged_cache[dtype: DType, kv_params: KVCacheStaticParams, page_size: Int](blocks: TileTensor[dtype, linear_idx_type=blocks.linear_idx_type], page_stride: TileTensor[.int64, linear_idx_type=page_stride.linear_idx_type], cache_lengths: TileTensor[.uint32, linear_idx_type=cache_lengths.linear_idx_type], lookup_table: TileTensor[.uint32, linear_idx_type=lookup_table.linear_idx_type], max_prompt_length: TileTensor[.uint32, linear_idx_type=max_prompt_length.linear_idx_type], max_cache_length: TileTensor[.uint32, linear_idx_type=max_cache_length.linear_idx_type], out result: PagedKVCacheCollection[dtype, kv_params, page_size, blocks.origin, cache_lengths.origin, lookup_table.origin, MutUntrackedOrigin])

Returns:

PagedKVCacheCollection[dtype, kv_params, page_size, blocks.origin, cache_lengths.origin, lookup_table.origin, MutUntrackedOrigin]

Was this page helpful?