For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
unfused_qkv_matmul_ragged_paged_gguf_quantized
def unfused_qkv_matmul_ragged_paged_gguf_quantized[dtype: DType, params: KVCacheStaticParams, page_size: Int, //, quantization_encoding_q: StringSpan[ImmStaticOrigin], quantization_encoding_k: StringSpan[ImmStaticOrigin], quantization_encoding_v: StringSpan[ImmStaticOrigin]](hidden_state: TileTensor[.float32, Engine=hidden_state.Engine, linear_idx_type=hidden_state.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], q_weight: TileTensor[.uint8, Engine=q_weight.Engine, linear_idx_type=q_weight.linear_idx_type], k_weight: TileTensor[.uint8, Engine=k_weight.Engine, linear_idx_type=k_weight.linear_idx_type], v_weight: TileTensor[.uint8, Engine=v_weight.Engine, linear_idx_type=v_weight.linear_idx_type], kv_collection: PagedKVCacheCollection[dtype, params, page_size, scale_dtype_=kv_collection.scale_dtype_, quantization_granularity_=kv_collection.quantization_granularity_], layer_idx: UInt32, output: TileTensor[.float32, Engine=output.Engine, linear_idx_type=output.linear_idx_type], ctx: DeviceContextBase)
Performs a quantized matmul, writing the output into a mutable PagedKVCacheCollection object.
Unlike the un-quantized version (kv_matmul_ragged_continuous_batching), this implementation does not concat the q, k, and v weights together. Instead, it performs three matmuls. This allows the q, k, and v weights to have different quantization encodings.
This is only supported on CPU.
Parameters:
- dtype (
DType): Element type of the KV cache collection entries (inferred). - params (
KVCacheStaticParams): Static shape parameters of the paged KV cache, including the attention-head count, per-head size, and MLA flag (inferred). - page_size (
Int): Number of tokens stored per cache page (inferred). - quantization_encoding_q (
StringSpan[ImmStaticOrigin]): GGUF quantization encoding name applied to theq_weighttensor. - quantization_encoding_k (
StringSpan[ImmStaticOrigin]): GGUF quantization encoding name applied to thek_weighttensor. - quantization_encoding_v (
StringSpan[ImmStaticOrigin]): GGUF quantization encoding name applied to thev_weighttensor.
Args:
- hidden_state (
TileTensor[.float32, Engine=hidden_state.Engine, linear_idx_type=hidden_state.linear_idx_type]): Tensor with shape (sum(seq_lens), hidden_size). - input_row_offsets (
TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type]): Tensor with shape (batch_size + 1,) denoting the start of each sequence along the seq_len dimension. - q_weight (
TileTensor[.uint8, Engine=q_weight.Engine, linear_idx_type=q_weight.linear_idx_type]): Packed uint8 Q projection weights, one row per output channel. - k_weight (
TileTensor[.uint8, Engine=k_weight.Engine, linear_idx_type=k_weight.linear_idx_type]): Packed uint8 K projection weights, one row per output channel. - v_weight (
TileTensor[.uint8, Engine=v_weight.Engine, linear_idx_type=v_weight.linear_idx_type]): Packed uint8 V projection weights, one row per output channel. - kv_collection (
PagedKVCacheCollection[dtype, params, page_size, scale_dtype_=kv_collection.scale_dtype_, quantization_granularity_=kv_collection.quantization_granularity_]): The Collection object storing KVCache entries. - layer_idx (
UInt32): The index of the layer being executed. Used to retrieve the KVCache for the given layer from kv_collection. - output (
TileTensor[.float32, Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Tensor with shape (sum(seq_lens), num_query_heads * head_size). This is the output buffer for the Q matmul. - ctx (
DeviceContextBase): The call context pointer, passed by the graph compiler.