For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
generic_cross_attention_kv_cache
def generic_cross_attention_kv_cache[collection_t: KVCollectionT, dtype: DType, //, target: StringSpan[ImmStaticOrigin], mask_str: StringSpan[ImmStaticOrigin], local_window_size: Int = Int(-1), output_dtype: DType = dtype](q: TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type], q_input_row_offsets: TileTensor[.uint32, Engine=q_input_row_offsets.Engine, linear_idx_type=q_input_row_offsets.linear_idx_type], q_max_seq_len: TileTensor[.uint32, Engine=q_max_seq_len.Engine, linear_idx_type=q_max_seq_len.linear_idx_type], kv_input_row_offsets: TileTensor[.uint32, Engine=kv_input_row_offsets.Engine, linear_idx_type=kv_input_row_offsets.linear_idx_type], kv_collection: collection_t, layer_idx: UInt32, scale: Float32, output: TileTensor[output_dtype, Engine=output.Engine, linear_idx_type=output.linear_idx_type], context: DeviceContextBase, sink_weights: OptionalReg[TileTensor[dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None)
Dispatches cross-attention flash attention over a ragged batch against a paged KV cache.
Parameters:
- collection_t (
KVCollectionT): The KV cache collection type storing the K and V caches for this layer (inferred). - dtype (
DType): Data type of the query tensor (inferred). - target (
StringSpan[ImmStaticOrigin]): Target device string for kernel dispatch, such as "cpu" or "gpu". - mask_str (
StringSpan[ImmStaticOrigin]): Attention mask name selecting the masking strategy, such as "causal", "null", or "sliding_window_causal". - local_window_size (
Int): Sliding-window size in tokens for windowed masks; -1 for masks that ignore it (defaults to -1). - output_dtype (
DType): Data type of the output tensor; defaults todtype.
Args:
- q (
TileTensor[dtype, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): Query tensor with shape (sum(q_seq_lens), num_heads, head_size). - q_input_row_offsets (
TileTensor[.uint32, Engine=q_input_row_offsets.Engine, linear_idx_type=q_input_row_offsets.linear_idx_type]): Tensor with shape (batch_size + 1,) denoting the start of each query sequence along the ragged sequence dimension. - q_max_seq_len (
TileTensor[.uint32, Engine=q_max_seq_len.Engine, linear_idx_type=q_max_seq_len.linear_idx_type]): Scalar tensor holding the maximum query sequence length. - kv_input_row_offsets (
TileTensor[.uint32, Engine=kv_input_row_offsets.Engine, linear_idx_type=kv_input_row_offsets.linear_idx_type]): Tensor with shape (batch_size + 1,) denoting the start of each KV sequence along the ragged sequence dimension. GPU dispatch requires unit stride. - kv_collection (
collection_t): The collection storing the KVCache entries for this layer, retrieved via layer_idx. - layer_idx (
UInt32): The index of the layer being executed, used to retrieve the KVCache objects from kv_collection. - scale (
Float32): The scaling factor in scaled dot-product attention, usually rsqrt(head_size). - output (
TileTensor[output_dtype, Engine=output.Engine, linear_idx_type=output.linear_idx_type]): The pre-allocated output buffer to write results to, with shape (sum(q_seq_lens), num_heads, head_size). - context (
DeviceContextBase): The call context pointer, passed by the graph compiler. - sink_weights (
OptionalReg[TileTensor[dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Optional per-batch attention sink weights applied to the leading cache slots.