IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

kv_cache_store_scales_ragged

def kv_cache_store_scales_ragged[cache_t: KVCacheT, scale_dtype: DType, //, InputFnType: def[width: Int, alignment: Int](Coord[**?]) -> SIMD[scale_dtype, width] & RegisterPassable & ImplicitlyCopyable, *, dcp_size: Int = Int(1), dcp_interleave: Int = Int(1)](cache: cache_t, input_shape: Coord[*input_shape.element_types], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], input_fn: InputFnType, context: DeviceContext, dcp_rank: Int = Int(0))

Stores ragged per-token quantization scales into a paged KV cache.

The scale counterpart of kv_cache_store_ragged: a row lands at the same position its values do, through the cache's scale lookup table.

Parameters:

  • ​cache_t (KVCacheT): The quantized KV cache type (inferred).
  • ​scale_dtype (DType): Element type of the scales (inferred).
  • ​InputFnType (def[width: Int, alignment: Int](Coord[**?]) -> SIMD[scale_dtype, width] & RegisterPassable & ImplicitlyCopyable): Type of input_fn (inferred).
  • ​dcp_size (Int): Ranks of the decode context-parallel group sharing each request's KV positions; 1 stores every row.
  • ​dcp_interleave (Int): Consecutive positions a rank owns before the next rank's turn (see ContextParallelLayout).

Args:

  • ​cache (cache_t): The KVCache object whose scales to write.
  • ​input_shape (Coord[*input_shape.element_types]): Shape of the scales as a 3D coordinate (tokens, heads, scale groups).
  • ​input_row_offsets (TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type]): Tensor with shape (batch_size + 1,) denoting the start of each sequence along the ragged sequence dimension.
  • ​input_fn (InputFnType): Callback that loads a SIMD vector of scales at the given 3D index.
  • ​context (DeviceContext): The call context pointer, passed by the graph compiler.
  • ​dcp_rank (Int): This rank's index in the decode context-parallel group; see kv_cache_store_ragged.

Was this page helpful?