For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
flash_attention_kv_cache
def flash_attention_kv_cache[dtype: DType, cache_t: KVCacheT, q_origin: ImmOrigin, output_origin: MutOrigin, //](q: TileTensor[dtype, q_origin, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: cache_t, v: cache_t, mask: TileTensor[dtype, Engine=mask.Engine, linear_idx_type=mask.linear_idx_type], scale: Float32, output: TileTensor[dtype, output_origin, Engine=output.Engine, linear_idx_type=output.linear_idx_type], sink_weights: OptionalReg[TileTensor[dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None)
Computes flash attention on CPU using a KV cache with an additive TileTensor mask.
Args:
- q (
TileTensor[dtype, q_origin, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): Query tensor in BSHD layout. - k (
cache_t): Key cache. - v (
cache_t): Value cache. - mask (
TileTensor[dtype, Engine=mask.Engine, linear_idx_type=mask.linear_idx_type]): Additive attention mask tensor. - scale (
Float32): Scaling factor applied to the query-key dot products. - output (
TileTensor[dtype, output_origin, Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Output tensor to write the attention results into. - sink_weights (
OptionalReg[TileTensor[dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Optional per-head attention sink weights.
def flash_attention_kv_cache[dtype: DType, cache_t: KVCacheT, mask_t: MHAMask, q_origin: ImmOrigin, output_origin: MutOrigin, //](q: TileTensor[dtype, q_origin, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: cache_t, v: cache_t, mask: mask_t, scale: Float32, output: TileTensor[dtype, output_origin, Engine=output.Engine, linear_idx_type=output.linear_idx_type], sink_weights: OptionalReg[TileTensor[dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None)
Computes flash attention on CPU using a KV cache with an MHAMask-based mask.
Args:
- q (
TileTensor[dtype, q_origin, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): Query tensor in BSHD layout. - k (
cache_t): Key cache. - v (
cache_t): Value cache. - mask (
mask_t): MHAMask applied to the attention scores. - scale (
Float32): Scaling factor applied to the query-key dot products. - output (
TileTensor[dtype, output_origin, Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Output tensor to write the attention results into. - sink_weights (
OptionalReg[TileTensor[dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Optional per-head attention sink weights.
def flash_attention_kv_cache[dtype: DType, cache_t: KVCacheT, mask_t: MHAMask, q_origin: ImmOrigin, output_origin: MutOrigin, //](q: TileTensor[dtype, q_origin, Engine=q.Engine, linear_idx_type=q.linear_idx_type], q_input_row_offsets: TileTensor[.uint32, Engine=q_input_row_offsets.Engine, linear_idx_type=q_input_row_offsets.linear_idx_type], kv_input_row_offsets: TileTensor[.uint32, Engine=kv_input_row_offsets.Engine, linear_idx_type=kv_input_row_offsets.linear_idx_type], k: cache_t, v: cache_t, mask: mask_t, scale: Float32, output: TileTensor[dtype, output_origin, Engine=output.Engine, linear_idx_type=output.linear_idx_type], sink_weights: OptionalReg[TileTensor[dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None)
Computes flash attention on CPU for ragged tensors using a KV cache with an MHAMask-based mask.
Parameters:
- dtype (
DType): The element type of the query, key, value, and output tensors (inferred). - cache_t (
KVCacheT): The KV cache type storing key and value states (inferred). - mask_t (
MHAMask): TheMHAMasktype applied to the attention scores (inferred). - q_origin (
ImmOrigin): The memory origin of the read-only query tensor (inferred). - output_origin (
MutOrigin): The memory origin of the writable output tensor (inferred).
Args:
- q (
TileTensor[dtype, q_origin, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): Flattened query tensor indexed by(row_offset, head, depth). - q_input_row_offsets (
TileTensor[.uint32, Engine=q_input_row_offsets.Engine, linear_idx_type=q_input_row_offsets.linear_idx_type]): Per-batch start offsets into the flattened query tensor; batchbspans rows[offsets[b], offsets[b + 1]). - kv_input_row_offsets (
TileTensor[.uint32, Engine=kv_input_row_offsets.Engine, linear_idx_type=kv_input_row_offsets.linear_idx_type]): Per-batch start offsets into the flattened KV tensors; batchbspans rows[offsets[b], offsets[b + 1]). - k (
cache_t): Key cache storing per-head key states. - v (
cache_t): Value cache storing per-head value states. - mask (
mask_t):MHAMaskapplied additively to the query-key attention scores. - scale (
Float32): Scaling factor applied to the query-key dot products. - output (
TileTensor[dtype, output_origin, Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Output tensor to write the attention results into. - sink_weights (
OptionalReg[TileTensor[dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Optional per-head attention sink weights.