For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
kv_matmul_ragged_paged
def kv_matmul_ragged_paged[dtype: DType, params: KVCacheStaticParams, page_size: Int, //, target: StringSpan[ImmStaticOrigin]](hidden_state: TileTensor[dtype, Engine=hidden_state.Engine, linear_idx_type=hidden_state.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], weight: TileTensor[dtype, Engine=weight.Engine, linear_idx_type=weight.linear_idx_type], kv_collection: PagedKVCacheCollection[dtype, params, page_size, scale_dtype_=kv_collection.scale_dtype_, quantization_granularity_=kv_collection.quantization_granularity_], layer_idx: UInt32, ctx: DeviceContextBase)
Performs a matmul, writing the output into a mutable PagedKVCacheCollection object.
Parameters:
- dtype (
DType): Element type of thehidden_stateandweighttensors and of the KV cache entries (inferred). - params (
KVCacheStaticParams): Static shape parameters of the paged KV cache, including the attention-head count, per-head size, and MLA flag (inferred). - page_size (
Int): Number of tokens stored per cache page (inferred). - target (
StringSpan[ImmStaticOrigin]): Compilation target string used to dispatch GPU versus CPU paths.
Args:
- hidden_state (
TileTensor[dtype, Engine=hidden_state.Engine, linear_idx_type=hidden_state.linear_idx_type]): Ragged input tensor with shape (sum(seq_lens), hidden_size). - input_row_offsets (
TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type]): Tensor with shape (batch_size + 1,) denoting the start of each sequence along the seq_len dimension. - weight (
TileTensor[dtype, Engine=weight.Engine, linear_idx_type=weight.linear_idx_type]): Concatenated K/V weights with shape (2 * num_kv_heads * head_size, hidden_size). - kv_collection (
PagedKVCacheCollection[dtype, params, page_size, scale_dtype_=kv_collection.scale_dtype_, quantization_granularity_=kv_collection.quantization_granularity_]): The historical KVCache for keys and values. The KVCache for this layer is retrieved via layer_idx. - layer_idx (
UInt32): The index of the layer being executed. Used to retrieve the KVCache for the given layer from kv_collection. - ctx (
DeviceContextBase): The call context pointer, passed by the graph compiler.