For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
generic_fused_qkv_matmul_kv_cache_bshd_paged
def generic_fused_qkv_matmul_kv_cache_bshd_paged[dtype: DType](hidden_state: TileTensor[dtype, Engine=hidden_state.Engine, linear_idx_type=hidden_state.linear_idx_type], weight: TileTensor[dtype, Engine=weight.Engine, linear_idx_type=weight.linear_idx_type], kv_collection: PagedKVCacheCollection[scale_dtype_=kv_collection.scale_dtype_, quantization_granularity_=kv_collection.quantization_granularity_], layer_idx: UInt32, valid_lengths: TileTensor[.uint32, Engine=valid_lengths.Engine, linear_idx_type=valid_lengths.linear_idx_type], output: TileTensor[dtype, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type], ctx: DeviceContext)
Performs a fused QKV matmul. Q outputs are written to the output argument while K and V outputs are written in-place into k_cache and v_cache.
Only positions within valid_lengths are written to the KV cache.
Args:
- hidden_state (
TileTensor[dtype, Engine=hidden_state.Engine, linear_idx_type=hidden_state.linear_idx_type]): Tensor with shape (batch_size, seq_len, num_heads * head_size). - weight (
TileTensor[dtype, Engine=weight.Engine, linear_idx_type=weight.linear_idx_type]): Tensor with shape (num_heads * head_size, num_kv_heads * head_size). - kv_collection (
PagedKVCacheCollection[scale_dtype_=kv_collection.scale_dtype_, quantization_granularity_=kv_collection.quantization_granularity_]): The historical KVCache for keys and values. The KVCache for this layer is retrieved via layer_idx. - layer_idx (
UInt32): The index of the layer being executed. Used to retrieve the KVCache for the given layer from kv_collection. - valid_lengths (
TileTensor[.uint32, Engine=valid_lengths.Engine, linear_idx_type=valid_lengths.linear_idx_type]): Tensor of shape [batch] containing the valid length for each sequence. K and V are only written to cache for positions within these lengths. - output (
TileTensor[dtype, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type]): The pre-allocated output buffer for Q projections. K and V projections are written in-place to k_cache and v_cache. - ctx (
DeviceContext): The call context pointer, passed by the graph compiler.