IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

kv_cache_store_ragged

def kv_cache_store_ragged[cache_t: KVCacheT, //, InputFnType: def[width: Int, alignment: Int](Coord[**?]) -> SIMD[cache_t.dtype, width] & RegisterPassable & ImplicitlyCopyable, *, head_dim_offset: Int = Int(0), dcp_size: Int = Int(1), dcp_interleave: Int = Int(1)](cache: cache_t, input_shape: Coord[*input_shape.element_types], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], input_fn: InputFnType, context: DeviceContext, dcp_rank: Int = Int(0))

Stores ragged input values into a paged KV cache via an elementwise kernel.

Invokes the supplied input_fn to load values and writes them into the cache at positions determined by the per-batch cache lengths and input row offsets.

Parameters:

  • ​cache_t (KVCacheT): The KV cache type used to store key or value entries (inferred).
  • ​InputFnType (def[width: Int, alignment: Int](Coord[**?]) -> SIMD[cache_t.dtype, width] & RegisterPassable & ImplicitlyCopyable): Type of input_fn (inferred).
  • ​head_dim_offset (Int): Cache column of the input's first column: the value's v_col_base() in a split cache, else 0.
  • ​dcp_size (Int): Ranks of the decode context-parallel group sharing each request's KV positions; 1 stores every row.
  • ​dcp_interleave (Int): Consecutive positions a rank owns before the next rank's turn (see ContextParallelLayout).

Args:

  • ​cache (cache_t): The KVCache object to write key or value entries into.
  • ​input_shape (Coord[*input_shape.element_types]): Shape of the input as a 3D coordinate (tokens, heads, head_size).
  • ​input_row_offsets (TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type]): Tensor with shape (batch_size + 1,) denoting the start of each sequence along the ragged sequence dimension.
  • ​input_fn (InputFnType): Callback that loads a SIMD vector of cache-typed elements at the given 3D index.
  • ​context (DeviceContext): The call context pointer, passed by the graph compiler.
  • ​dcp_rank (Int): This rank's index in the decode context-parallel group. cache_lengths stay global; a row is stored only if this rank owns its position, at the rank's local slot.

Was this page helpful?