For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
kv_cache_store_ragged
def kv_cache_store_ragged[cache_t: KVCacheT, //, InputFnType: def[width: Int, alignment: Int](Coord[**?]) -> SIMD[cache_t.dtype, width] & RegisterPassable & ImplicitlyCopyable, *, head_dim_offset: Int = Int(0), dcp_size: Int = Int(1), dcp_interleave: Int = Int(1)](cache: cache_t, input_shape: Coord[*input_shape.element_types], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], input_fn: InputFnType, context: DeviceContext, dcp_rank: Int = Int(0))
Stores ragged input values into a paged KV cache via an elementwise kernel.
Invokes the supplied input_fn to load values and writes them into the cache at positions determined by the per-batch cache lengths and input row offsets.
Parameters:
- cache_t (
KVCacheT): The KV cache type used to store key or value entries (inferred). - InputFnType (
def[width: Int, alignment: Int](Coord[**?]) -> SIMD[cache_t.dtype, width]&RegisterPassable&ImplicitlyCopyable): Type ofinput_fn(inferred). - head_dim_offset (
Int): Cache column of the input's first column: the value'sv_col_base()in a split cache, else 0. - dcp_size (
Int): Ranks of the decode context-parallel group sharing each request's KV positions; 1 stores every row. - dcp_interleave (
Int): Consecutive positions a rank owns before the next rank's turn (seeContextParallelLayout).
Args:
- cache (
cache_t): The KVCache object to write key or value entries into. - input_shape (
Coord[*input_shape.element_types]): Shape of the input as a 3D coordinate (tokens, heads, head_size). - input_row_offsets (
TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type]): Tensor with shape (batch_size + 1,) denoting the start of each sequence along the ragged sequence dimension. - input_fn (
InputFnType): Callback that loads a SIMD vector of cache-typed elements at the given 3D index. - context (
DeviceContext): The call context pointer, passed by the graph compiler. - dcp_rank (
Int): This rank's index in the decode context-parallel group.cache_lengthsstay global; a row is stored only if this rank owns its position, at the rank's local slot.