For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
k_matmul_ragged_paged
def k_matmul_ragged_paged[dtype: DType, params: KVCacheStaticParams, page_size: Int, //, target: StringSpan[ImmStaticOrigin]](hidden_state: TileTensor[dtype, Engine=hidden_state.Engine, linear_idx_type=hidden_state.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], weight: TileTensor[dtype, Engine=weight.Engine, linear_idx_type=weight.linear_idx_type], kv_collection: PagedKVCacheCollection[dtype, params, page_size, scale_dtype_=kv_collection.scale_dtype_, quantization_granularity_=kv_collection.quantization_granularity_], layer_idx: UInt32, ctx: DeviceContextBase)
Performs a matmul, writing the output into a mutable PagedKVCacheCollection object.
Parameters:
- dtype (
DType): Element type of thehidden_stateinput, theweighttensor, and the KV cache entries (inferred). - params (
KVCacheStaticParams): StaticKVCacheStaticParamsdescribing the cache layout, such as head count and head dimension (inferred). - page_size (
Int): Number of tokens stored per cache page in the paged KV cache (inferred). - target (
StringSpan[ImmStaticOrigin]): Compilation target string used to dispatch GPU versus CPU paths.
Args:
- hidden_state (
TileTensor[dtype, Engine=hidden_state.Engine, linear_idx_type=hidden_state.linear_idx_type]): Ragged input tensor with shape (sum(seq_lens), hidden_size). - input_row_offsets (
TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type]): Tensor with shape (batch_size + 1,) denoting the start of each sequence along the seq_len dimension. - weight (
TileTensor[dtype, Engine=weight.Engine, linear_idx_type=weight.linear_idx_type]): Key projection weights with shape (num_kv_heads * head_size, hidden_size). - kv_collection (
PagedKVCacheCollection[dtype, params, page_size, scale_dtype_=kv_collection.scale_dtype_, quantization_granularity_=kv_collection.quantization_granularity_]): The historical KVCache for keys and values. The KVCache for this layer is retrieved via layer_idx. - layer_idx (
UInt32): The index of the layer being executed. Used to retrieve the KVCache for the given layer from kv_collection. - ctx (
DeviceContextBase): The call context pointer, passed by the graph compiler.