For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
generic_get_paged_cache_with_scales
def generic_get_paged_cache_with_scales[dtype: DType, scale_dtype: DType](blocks: ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=blocks.static_spec], page_stride: InputTensor[static_spec=page_stride.static_spec], cache_lengths: InputTensor[static_spec=cache_lengths.static_spec], lookup_table: InputTensor[static_spec=lookup_table.static_spec], max_prompt_length: InputTensor[static_spec=max_prompt_length.static_spec], max_cache_length: InputTensor[static_spec=max_cache_length.static_spec], scales: ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=scales.static_spec], scales_page_stride: InputTensor[static_spec=scales_page_stride.static_spec], scales_lookup_table: InputTensor[static_spec=scales_lookup_table.static_spec], out result: PagedKVCacheCollection[dtype, KVCacheStaticParams(SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(4))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(4)]())), SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(5))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(5)]())), (SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(1))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(1)]())) == Int(1)), _scaled_cache_fp8_tail_size[dtype, scale_dtype](SIMD[blocks.static_spec.static_layout.__shape_types[5]]((load_from_mem _coord_sentinel_value[*blocks.static_spec.static_layout._shape_types]().__getitem_param__[Int(5)]())), SIMD[scales.static_spec.static_layout.__shape_types[5]]((load_from_mem _coord_sentinel_value[*scales.static_spec.static_layout._shape_types]().__getitem_param__[Int(5)]()))), Int(0)), SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(3))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(3)]())), MutUntrackedOrigin, ImmUntrackedOrigin, ImmUntrackedOrigin, MutUntrackedOrigin, scale_dtype_=scale_dtype, quantization_granularity_=_scaled_cache_granularity[dtype, scale_dtype](SIMD[blocks.static_spec.static_layout.__shape_types[5]]((load_from_mem _coord_sentinel_value[*blocks.static_spec.static_layout._shape_types]().__getitem_param__[Int(5)]())), SIMD[scales.static_spec.static_layout.__shape_types[5]]((load_from_mem _coord_sentinel_value[*scales.static_spec.static_layout._shape_types]().__getitem_param__[Int(5)]())))])
Builds a scaled paged KV cache collection from graph operands.
The operands come in the order flatten_without_attention_dispatch_metadata
emits for quantized cache params. The quantization granularity is the
values' row width over the scales' row width, in value elements. A packed
NVFP4 row may end in FP8 bytes that carry no scales.
Args:
- blocks (
ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=blocks.static_spec]): KV values[num_blocks, kv_dim, num_layers, page_size, num_heads, head_size]. - page_stride (
InputTensor[static_spec=page_stride.static_spec]): Page-to-page distance forblocks, in elements. - cache_lengths (
InputTensor[static_spec=cache_lengths.static_spec]): Cache lengths per request. - lookup_table (
InputTensor[static_spec=lookup_table.static_spec]): Page lookup table[batch_size, max_pages]. - max_prompt_length (
InputTensor[static_spec=max_prompt_length.static_spec]): Max prompt length,[1]. - max_cache_length (
InputTensor[static_spec=max_cache_length.static_spec]): Max cache length,[1]. - scales (
ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=scales.static_spec]): Scales, the shape ofblockswith the last dimensionhead_size / granularity. - scales_page_stride (
InputTensor[static_spec=scales_page_stride.static_spec]): Page-to-page distance forscales. - scales_lookup_table (
InputTensor[static_spec=scales_lookup_table.static_spec]): Page lookup table of the scales.
Returns:
def generic_get_paged_cache_with_scales(blocks: TileTensor[Engine=blocks.Engine, linear_idx_type=blocks.linear_idx_type], page_stride: TileTensor[.int64, Engine=page_stride.Engine, linear_idx_type=page_stride.linear_idx_type], cache_lengths: TileTensor[.uint32, Engine=cache_lengths.Engine, linear_idx_type=cache_lengths.linear_idx_type], lookup_table: TileTensor[.uint32, Engine=lookup_table.Engine, linear_idx_type=lookup_table.linear_idx_type], max_prompt_length: TileTensor[.uint32, Engine=max_prompt_length.Engine, linear_idx_type=max_prompt_length.linear_idx_type], max_cache_length: TileTensor[.uint32, Engine=max_cache_length.Engine, linear_idx_type=max_cache_length.linear_idx_type], scales: TileTensor[Engine=scales.Engine, linear_idx_type=scales.linear_idx_type], scales_page_stride: TileTensor[.int64, Engine=scales_page_stride.Engine, linear_idx_type=scales_page_stride.linear_idx_type], scales_lookup_table: TileTensor[.uint32, lookup_table.origin, Engine=scales_lookup_table.Engine, linear_idx_type=scales_lookup_table.linear_idx_type], out result: PagedKVCacheCollection[blocks.dtype, KVCacheStaticParams(blocks.LayoutType.static_shape[Int(4)], blocks.LayoutType.static_shape[Int(5)], (blocks.LayoutType.static_shape[Int(1)] == Int(1)), _scaled_cache_fp8_tail_size[blocks.dtype, scales.dtype](blocks.LayoutType.static_shape[Int(5)], scales.LayoutType.static_shape[Int(5)]), Int(0)), blocks.LayoutType.static_shape[Int(3)], blocks.origin, cache_lengths.origin, lookup_table.origin, scales.origin, scale_dtype_=scales.dtype, quantization_granularity_=_scaled_cache_granularity[blocks.dtype, scales.dtype](blocks.LayoutType.static_shape[Int(5)], scales.LayoutType.static_shape[Int(5)])])
Builds a scaled paged KV cache collection from projected tiles of any storage.
The operands come in the order flatten_without_attention_dispatch_metadata
emits for quantized cache params. The blocks' static shape supplies the
cache's head count, head size, page size and MLA layout. The quantization
granularity is the values' row width over the scales' row width, in value
elements. A packed NVFP4 row may end in FP8 bytes that carry no scales.
Args:
- blocks (
TileTensor[Engine=blocks.Engine, linear_idx_type=blocks.linear_idx_type]): KV values[num_blocks, kv_dim, num_layers, page_size, num_heads, head_size]. - page_stride (
TileTensor[.int64, Engine=page_stride.Engine, linear_idx_type=page_stride.linear_idx_type]): Page-to-page distance forblocks, in elements. - cache_lengths (
TileTensor[.uint32, Engine=cache_lengths.Engine, linear_idx_type=cache_lengths.linear_idx_type]): Cache lengths per request. - lookup_table (
TileTensor[.uint32, Engine=lookup_table.Engine, linear_idx_type=lookup_table.linear_idx_type]): Page lookup table[batch_size, max_pages]. - max_prompt_length (
TileTensor[.uint32, Engine=max_prompt_length.Engine, linear_idx_type=max_prompt_length.linear_idx_type]): Max prompt length,[1]. - max_cache_length (
TileTensor[.uint32, Engine=max_cache_length.Engine, linear_idx_type=max_cache_length.linear_idx_type]): Max cache length,[1]. - scales (
TileTensor[Engine=scales.Engine, linear_idx_type=scales.linear_idx_type]): Scales, the shape ofblockswith the last dimensionhead_size / granularity. - scales_page_stride (
TileTensor[.int64, Engine=scales_page_stride.Engine, linear_idx_type=scales_page_stride.linear_idx_type]): Page-to-page distance forscales. - scales_lookup_table (
TileTensor[.uint32, lookup_table.origin, Engine=scales_lookup_table.Engine, linear_idx_type=scales_lookup_table.linear_idx_type]): Page lookup table of the scales.
Returns:
def generic_get_paged_cache_with_scales[dtype: DType, scale_dtype: DType, kv_params: KVCacheStaticParams, page_size: Int, quantization_granularity: Int](blocks: TileTensor[dtype, linear_idx_type=blocks.linear_idx_type], page_stride: TileTensor[.int64, linear_idx_type=page_stride.linear_idx_type], cache_lengths: TileTensor[.uint32, linear_idx_type=cache_lengths.linear_idx_type], lookup_table: TileTensor[.uint32, linear_idx_type=lookup_table.linear_idx_type], max_prompt_length: TileTensor[.uint32, linear_idx_type=max_prompt_length.linear_idx_type], max_cache_length: TileTensor[.uint32, linear_idx_type=max_cache_length.linear_idx_type], scales: TileTensor[scale_dtype, linear_idx_type=scales.linear_idx_type], scales_page_stride: TileTensor[.int64, linear_idx_type=scales_page_stride.linear_idx_type], scales_lookup_table: TileTensor[.uint32, lookup_table.origin, linear_idx_type=scales_lookup_table.linear_idx_type], out result: PagedKVCacheCollection[dtype, kv_params, page_size, blocks.origin, cache_lengths.origin, lookup_table.origin, scales.origin, scale_dtype_=scale_dtype, quantization_granularity_=quantization_granularity])
Create a PagedKVCacheCollection with scales for MLA attention.
Args:
- blocks (
TileTensor[dtype, linear_idx_type=blocks.linear_idx_type]): KV cache blocks tensor [num_blocks, kv_dim, num_layers, page_size, num_heads, head_dim]. - page_stride (
TileTensor[.int64, linear_idx_type=page_stride.linear_idx_type]): Page-to-page distance forblocks, in elements; -1 when packed. - cache_lengths (
TileTensor[.uint32, linear_idx_type=cache_lengths.linear_idx_type]): Cache lengths per batch [batch_size]. - lookup_table (
TileTensor[.uint32, linear_idx_type=lookup_table.linear_idx_type]): Page lookup table [batch_size, max_pages]. - max_prompt_length (
TileTensor[.uint32, linear_idx_type=max_prompt_length.linear_idx_type]): Max prompt (query) length scalar tensor [1]. - max_cache_length (
TileTensor[.uint32, linear_idx_type=max_cache_length.linear_idx_type]): Max cache length scalar tensor [1]. - scales (
TileTensor[scale_dtype, linear_idx_type=scales.linear_idx_type]): Scales tensor [num_blocks, kv_dim, num_layers, page_size, num_heads, head_dim_granularity]. - scales_page_stride (
TileTensor[.int64, linear_idx_type=scales_page_stride.linear_idx_type]): The same distance forscales, which is its own pool leaf and pads independently of the values. - scales_lookup_table (
TileTensor[.uint32, lookup_table.origin, linear_idx_type=scales_lookup_table.linear_idx_type]): Page lookup table for the scales [batch_size, max_pages]. Passlookup_tableitself while the scales share the values' block-id space, and a distinct table when the scales are paged independently so a request's scale pages carry their own ids.
Returns: