IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

generic_get_paged_cache_with_scales

def generic_get_paged_cache_with_scales[dtype: DType, scale_dtype: DType](blocks: ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=blocks.static_spec], page_stride: InputTensor[static_spec=page_stride.static_spec], cache_lengths: InputTensor[static_spec=cache_lengths.static_spec], lookup_table: InputTensor[static_spec=lookup_table.static_spec], max_prompt_length: InputTensor[static_spec=max_prompt_length.static_spec], max_cache_length: InputTensor[static_spec=max_cache_length.static_spec], scales: ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=scales.static_spec], scales_page_stride: InputTensor[static_spec=scales_page_stride.static_spec], scales_lookup_table: InputTensor[static_spec=scales_lookup_table.static_spec], out result: PagedKVCacheCollection[dtype, KVCacheStaticParams(SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(4))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(4)]())), SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(5))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(5)]())), (SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(1))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(1)]())) == Int(1)), _scaled_cache_fp8_tail_size[dtype, scale_dtype](SIMD[blocks.static_spec.static_layout.__shape_types[5]]((load_from_mem _coord_sentinel_value[*blocks.static_spec.static_layout._shape_types]().__getitem_param__[Int(5)]())), SIMD[scales.static_spec.static_layout.__shape_types[5]]((load_from_mem _coord_sentinel_value[*scales.static_spec.static_layout._shape_types]().__getitem_param__[Int(5)]()))), Int(0)), SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(3))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(3)]())), MutUntrackedOrigin, ImmUntrackedOrigin, ImmUntrackedOrigin, MutUntrackedOrigin, scale_dtype_=scale_dtype, quantization_granularity_=_scaled_cache_granularity[dtype, scale_dtype](SIMD[blocks.static_spec.static_layout.__shape_types[5]]((load_from_mem _coord_sentinel_value[*blocks.static_spec.static_layout._shape_types]().__getitem_param__[Int(5)]())), SIMD[scales.static_spec.static_layout.__shape_types[5]]((load_from_mem _coord_sentinel_value[*scales.static_spec.static_layout._shape_types]().__getitem_param__[Int(5)]())))])

Builds a scaled paged KV cache collection from graph operands.

The operands come in the order flatten_without_attention_dispatch_metadata emits for quantized cache params. The quantization granularity is the values' row width over the scales' row width, in value elements. A packed NVFP4 row may end in FP8 bytes that carry no scales.

Args:

Returns:

PagedKVCacheCollection[dtype, KVCacheStaticParams(SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(4))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(4)]())), SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(5))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(5)]())), (SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(1))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(1)]())) == Int(1)), _scaled_cache_fp8_tail_size[dtype, scale_dtype](SIMD[blocks.static_spec.static_layout.__shape_types[5]]((load_from_mem _coord_sentinel_value[*blocks.static_spec.static_layout._shape_types]().__getitem_param__[Int(5)]())), SIMD[scales.static_spec.static_layout.__shape_types[5]]((load_from_mem _coord_sentinel_value[*scales.static_spec.static_layout._shape_types]().__getitem_param__[Int(5)]()))), Int(0)), SIMD[blocks.static_spec.static_layout.__shape_types[SIMDLength(Int(3))]]((load_from_mem StaticTensorSpec[dtype, Int(6), blocks.static_spec.static_layout, blocks.static_spec.InFusion, blocks.static_spec.OutFusion, blocks.static_spec.ComputeFusion].shape_tuple.__getitem_param__[Int(3)]())), MutUntrackedOrigin, ImmUntrackedOrigin, ImmUntrackedOrigin, MutUntrackedOrigin, scale_dtype_=scale_dtype, quantization_granularity_=_scaled_cache_granularity[dtype, scale_dtype](SIMD[blocks.static_spec.static_layout.__shape_types[5]]((load_from_mem _coord_sentinel_value[*blocks.static_spec.static_layout._shape_types]().__getitem_param__[Int(5)]())), SIMD[scales.static_spec.static_layout.__shape_types[5]]((load_from_mem _coord_sentinel_value[*scales.static_spec.static_layout._shape_types]().__getitem_param__[Int(5)]())))]: The collection over the values and their scales.

def generic_get_paged_cache_with_scales(blocks: TileTensor[Engine=blocks.Engine, linear_idx_type=blocks.linear_idx_type], page_stride: TileTensor[.int64, Engine=page_stride.Engine, linear_idx_type=page_stride.linear_idx_type], cache_lengths: TileTensor[.uint32, Engine=cache_lengths.Engine, linear_idx_type=cache_lengths.linear_idx_type], lookup_table: TileTensor[.uint32, Engine=lookup_table.Engine, linear_idx_type=lookup_table.linear_idx_type], max_prompt_length: TileTensor[.uint32, Engine=max_prompt_length.Engine, linear_idx_type=max_prompt_length.linear_idx_type], max_cache_length: TileTensor[.uint32, Engine=max_cache_length.Engine, linear_idx_type=max_cache_length.linear_idx_type], scales: TileTensor[Engine=scales.Engine, linear_idx_type=scales.linear_idx_type], scales_page_stride: TileTensor[.int64, Engine=scales_page_stride.Engine, linear_idx_type=scales_page_stride.linear_idx_type], scales_lookup_table: TileTensor[.uint32, lookup_table.origin, Engine=scales_lookup_table.Engine, linear_idx_type=scales_lookup_table.linear_idx_type], out result: PagedKVCacheCollection[blocks.dtype, KVCacheStaticParams(blocks.LayoutType.static_shape[Int(4)], blocks.LayoutType.static_shape[Int(5)], (blocks.LayoutType.static_shape[Int(1)] == Int(1)), _scaled_cache_fp8_tail_size[blocks.dtype, scales.dtype](blocks.LayoutType.static_shape[Int(5)], scales.LayoutType.static_shape[Int(5)]), Int(0)), blocks.LayoutType.static_shape[Int(3)], blocks.origin, cache_lengths.origin, lookup_table.origin, scales.origin, scale_dtype_=scales.dtype, quantization_granularity_=_scaled_cache_granularity[blocks.dtype, scales.dtype](blocks.LayoutType.static_shape[Int(5)], scales.LayoutType.static_shape[Int(5)])])

Builds a scaled paged KV cache collection from projected tiles of any storage.

The operands come in the order flatten_without_attention_dispatch_metadata emits for quantized cache params. The blocks' static shape supplies the cache's head count, head size, page size and MLA layout. The quantization granularity is the values' row width over the scales' row width, in value elements. A packed NVFP4 row may end in FP8 bytes that carry no scales.

Args:

Returns:

PagedKVCacheCollection[blocks.dtype, KVCacheStaticParams(blocks.LayoutType.static_shape[Int(4)], blocks.LayoutType.static_shape[Int(5)], (blocks.LayoutType.static_shape[Int(1)] == Int(1)), _scaled_cache_fp8_tail_size[blocks.dtype, scales.dtype](blocks.LayoutType.static_shape[Int(5)], scales.LayoutType.static_shape[Int(5)]), Int(0)), blocks.LayoutType.static_shape[Int(3)], blocks.origin, cache_lengths.origin, lookup_table.origin, scales.origin, scale_dtype_=scales.dtype, quantization_granularity_=_scaled_cache_granularity[blocks.dtype, scales.dtype](blocks.LayoutType.static_shape[Int(5)], scales.LayoutType.static_shape[Int(5)])]: The collection over the values and their scales.

def generic_get_paged_cache_with_scales[dtype: DType, scale_dtype: DType, kv_params: KVCacheStaticParams, page_size: Int, quantization_granularity: Int](blocks: TileTensor[dtype, linear_idx_type=blocks.linear_idx_type], page_stride: TileTensor[.int64, linear_idx_type=page_stride.linear_idx_type], cache_lengths: TileTensor[.uint32, linear_idx_type=cache_lengths.linear_idx_type], lookup_table: TileTensor[.uint32, linear_idx_type=lookup_table.linear_idx_type], max_prompt_length: TileTensor[.uint32, linear_idx_type=max_prompt_length.linear_idx_type], max_cache_length: TileTensor[.uint32, linear_idx_type=max_cache_length.linear_idx_type], scales: TileTensor[scale_dtype, linear_idx_type=scales.linear_idx_type], scales_page_stride: TileTensor[.int64, linear_idx_type=scales_page_stride.linear_idx_type], scales_lookup_table: TileTensor[.uint32, lookup_table.origin, linear_idx_type=scales_lookup_table.linear_idx_type], out result: PagedKVCacheCollection[dtype, kv_params, page_size, blocks.origin, cache_lengths.origin, lookup_table.origin, scales.origin, scale_dtype_=scale_dtype, quantization_granularity_=quantization_granularity])

Create a PagedKVCacheCollection with scales for MLA attention.

Args:

Returns:

PagedKVCacheCollection[dtype, kv_params, page_size, blocks.origin, cache_lengths.origin, lookup_table.origin, scales.origin, scale_dtype_=scale_dtype, quantization_granularity_=quantization_granularity]

Was this page helpful?