For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
generic_fused_qkv_matmul_kv_cache_paged_ragged
def generic_fused_qkv_matmul_kv_cache_paged_ragged[dtype: DType, weight_dtype: DType, group_size: Optional[Int] = None, has_zp: Optional[Bool] = None](hidden_state: TileTensor[dtype, Engine=hidden_state.Engine, linear_idx_type=hidden_state.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], weight: TileTensor[weight_dtype, Engine=weight.Engine, linear_idx_type=weight.linear_idx_type], kv_collection: PagedKVCacheCollection[scale_dtype_=kv_collection.scale_dtype_, quantization_granularity_=kv_collection.quantization_granularity_], layer_idx: UInt32, output: TileTensor[dtype, Engine=output.Engine, linear_idx_type=output.linear_idx_type], ctx: DeviceContext)
Performs a fused QKV matmul. Q outputs are written to the output argument while K and V outputs are written in-place into k_cache and v_cache.
Parameters:
- dtype (
DType): Element type of thehidden_stateinput andoutputtensors. - weight_dtype (
DType): Element type of theweighttensor. - group_size (
Optional[Int]): Block size for GPTQ-style quantization ofweight; when set,weightmust beuint8(defaults toNonefor no quantization). - has_zp (
Optional[Bool]): Whether the weight quantization uses a zero point; currently must be falsy whengroup_sizeis set (defaults toNone).
Args:
- hidden_state (
TileTensor[dtype, Engine=hidden_state.Engine, linear_idx_type=hidden_state.linear_idx_type]): Ragged input tensor with shape (sum(seq_lens), hidden_size). - input_row_offsets (
TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type]): Tensor with shape (batch_size + 1). The value at each index is the start_idx of the corresponding batch in hidden_state. - weight (
TileTensor[weight_dtype, Engine=weight.Engine, linear_idx_type=weight.linear_idx_type]): Projection weights, logically shaped (projection_size, hidden_size), ordered Q, K for MLA or Q, K, V otherwise. Quantized weights use the corresponding packed storage. - kv_collection (
PagedKVCacheCollection[scale_dtype_=kv_collection.scale_dtype_, quantization_granularity_=kv_collection.quantization_granularity_]): The object storing the KVCache for this layer. - layer_idx (
UInt32): The current layer, used to retrieve the KVCache object from kv_collection. - output (
TileTensor[dtype, Engine=output.Engine, linear_idx_type=output.linear_idx_type]): The pre-allocated output buffer for Q projections. K and V projections are written in-place to k_cache and v_cache. Shape: (sum(seq_lens), num_heads * head_size). - ctx (
DeviceContext): The call context pointer, passed by the graph compiler.