For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
kv_cache_store_scales_ragged
def kv_cache_store_scales_ragged[cache_t: KVCacheT, scale_dtype: DType, //, InputFnType: def[width: Int, alignment: Int](Coord[**?]) -> SIMD[scale_dtype, width] & RegisterPassable & ImplicitlyCopyable, *, dcp_size: Int = Int(1), dcp_interleave: Int = Int(1)](cache: cache_t, input_shape: Coord[*input_shape.element_types], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], input_fn: InputFnType, context: DeviceContext, dcp_rank: Int = Int(0))
Stores ragged per-token quantization scales into a paged KV cache.
The scale counterpart of kv_cache_store_ragged: a row lands at the same
position its values do, through the cache's scale lookup table.
Parameters:
- cache_t (
KVCacheT): The quantized KV cache type (inferred). - scale_dtype (
DType): Element type of the scales (inferred). - InputFnType (
def[width: Int, alignment: Int](Coord[**?]) -> SIMD[scale_dtype, width]&RegisterPassable&ImplicitlyCopyable): Type ofinput_fn(inferred). - dcp_size (
Int): Ranks of the decode context-parallel group sharing each request's KV positions; 1 stores every row. - dcp_interleave (
Int): Consecutive positions a rank owns before the next rank's turn (seeContextParallelLayout).
Args:
- cache (
cache_t): The KVCache object whose scales to write. - input_shape (
Coord[*input_shape.element_types]): Shape of the scales as a 3D coordinate (tokens, heads, scale groups). - input_row_offsets (
TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type]): Tensor with shape (batch_size + 1,) denoting the start of each sequence along the ragged sequence dimension. - input_fn (
InputFnType): Callback that loads a SIMD vector of scales at the given 3D index. - context (
DeviceContext): The call context pointer, passed by the graph compiler. - dcp_rank (
Int): This rank's index in the decode context-parallel group; seekv_cache_store_ragged.