For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
generic_get_paged_cache_with_scales
def generic_get_paged_cache_with_scales[dtype: DType, scale_dtype: DType](blocks: ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=blocks.static_spec], page_stride: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=page_stride.static_spec], cache_lengths: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=cache_lengths.static_spec], lookup_table: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=lookup_table.static_spec], max_prompt_length: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=max_prompt_length.static_spec], max_cache_length: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=max_cache_length.static_spec], scales: ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=scales.static_spec], scales_page_stride: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=scales_page_stride.static_spec], scales_lookup_table: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=scales_lookup_table.static_spec], out result: PagedKVCacheCollection[dtype, KVCacheStaticParams(SIMD[IntTuple](StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion, blocks.static_spec.ComputeFusionTile, blocks.static_spec.OutFusionTile].shape_tuple[4]), SIMD[IntTuple](StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion, blocks.static_spec.ComputeFusionTile, blocks.static_spec.OutFusionTile].shape_tuple[5]), (SIMD[IntTuple](StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion, blocks.static_spec.ComputeFusionTile, blocks.static_spec.OutFusionTile].shape_tuple[1]) == Int(1))), SIMD[IntTuple](StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion, blocks.static_spec.ComputeFusionTile, blocks.static_spec.OutFusionTile].shape_tuple[3]), MutAnyOrigin, ImmutAnyOrigin, ImmutAnyOrigin, MutAnyOrigin, scale_dtype_=scale_dtype, quantization_granularity_=(SIMD[IntTuple](StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion, blocks.static_spec.ComputeFusionTile, blocks.static_spec.OutFusionTile].shape_tuple[5]) // SIMD[IntTuple](StaticTensorSpec[scale_dtype, Int(6), scales.static_spec.static_layout, scales.static_spec.InFusion, scales.static_spec.OutFusion, scales.static_spec.ComputeFusion, scales.static_spec.ComputeFusionTile, scales.static_spec.OutFusionTile].shape_tuple[5]))])
Builds a scaled paged KV cache collection from graph operands.
The operands come in the order flatten_without_attention_dispatch_metadata
emits for quantized cache params. The quantization granularity is the
values' row width over the scales' row width, in value elements.
Args:
- blocks (
ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=blocks.static_spec]): KV values[num_blocks, kv_dim, num_layers, page_size, num_heads, head_size]. - page_stride (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=page_stride.static_spec]): Page-to-page distance forblocks, in elements. - cache_lengths (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=cache_lengths.static_spec]): Cache lengths per request. - lookup_table (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=lookup_table.static_spec]): Page lookup table[batch_size, max_pages]. - max_prompt_length (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=max_prompt_length.static_spec]): Max prompt length,[1]. - max_cache_length (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=max_cache_length.static_spec]): Max cache length,[1]. - scales (
ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=scales.static_spec]): Scales, the shape ofblockswith the last dimensionhead_size / granularity. - scales_page_stride (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=scales_page_stride.static_spec]): Page-to-page distance forscales. - scales_lookup_table (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=scales_lookup_table.static_spec]): Page lookup table of the scales.
Returns:
def generic_get_paged_cache_with_scales[dtype: DType, scale_dtype: DType, kv_params: KVCacheStaticParams, page_size: Int, quantization_granularity: Int](blocks: TileTensor[dtype, linear_idx_type=blocks.linear_idx_type], page_stride: TileTensor[.int64, linear_idx_type=page_stride.linear_idx_type], cache_lengths: TileTensor[.uint32, linear_idx_type=cache_lengths.linear_idx_type], lookup_table: TileTensor[.uint32, linear_idx_type=lookup_table.linear_idx_type], max_prompt_length: TileTensor[.uint32, linear_idx_type=max_prompt_length.linear_idx_type], max_cache_length: TileTensor[.uint32, linear_idx_type=max_cache_length.linear_idx_type], scales: TileTensor[scale_dtype, linear_idx_type=scales.linear_idx_type], scales_page_stride: TileTensor[.int64, linear_idx_type=scales_page_stride.linear_idx_type], scales_lookup_table: TileTensor[.uint32, lookup_table.origin, linear_idx_type=scales_lookup_table.linear_idx_type], out result: PagedKVCacheCollection[dtype, kv_params, page_size, blocks.origin, cache_lengths.origin, lookup_table.origin, scales.origin, scale_dtype_=scale_dtype, quantization_granularity_=quantization_granularity])
Create a PagedKVCacheCollection with scales for MLA attention.
Args:
- blocks (
TileTensor[dtype, linear_idx_type=blocks.linear_idx_type]): KV cache blocks tensor [num_blocks, kv_dim, num_layers, page_size, num_heads, head_dim]. - page_stride (
TileTensor[.int64, linear_idx_type=page_stride.linear_idx_type]): Page-to-page distance forblocks, in elements; -1 when packed. - cache_lengths (
TileTensor[.uint32, linear_idx_type=cache_lengths.linear_idx_type]): Cache lengths per batch [batch_size]. - lookup_table (
TileTensor[.uint32, linear_idx_type=lookup_table.linear_idx_type]): Page lookup table [batch_size, max_pages]. - max_prompt_length (
TileTensor[.uint32, linear_idx_type=max_prompt_length.linear_idx_type]): Max prompt (query) length scalar tensor [1]. - max_cache_length (
TileTensor[.uint32, linear_idx_type=max_cache_length.linear_idx_type]): Max cache length scalar tensor [1]. - scales (
TileTensor[scale_dtype, linear_idx_type=scales.linear_idx_type]): Scales tensor [num_blocks, kv_dim, num_layers, page_size, num_heads, head_dim_granularity]. - scales_page_stride (
TileTensor[.int64, linear_idx_type=scales_page_stride.linear_idx_type]): The same distance forscales, which is its own pool leaf and pads independently of the values. - scales_lookup_table (
TileTensor[.uint32, lookup_table.origin, linear_idx_type=scales_lookup_table.linear_idx_type]): Page lookup table for the scales [batch_size, max_pages]. Passlookup_tableitself while the scales share the values' block-id space, and a distinct table when the scales are paged independently so a request's scale pages carry their own ids.
Returns: