For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
flash_attention
def flash_attention[dtype: DType, q_layout: TensorLayout, //, config: MHAConfig[dtype] = MHAConfig(q_layout.static_shape[Int(2)], q_layout.static_shape[Int(3)], Optional(None), Optional(None), Optional(None), Optional(None), Optional(None), Int(4), Int(1), FlashAttentionAlgorithm(Int(-1)), TensorMapSwizzle.SWIZZLE_128B, Int(-1)), decoding_warp_split_k: Bool = False, naive_kernel: Bool = False, sink: Bool = False](output: TileTensor[Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[dtype, q_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: TileTensor[Engine=k.Engine, linear_idx_type=k.linear_idx_type], v: TileTensor[Engine=v.Engine, linear_idx_type=v.linear_idx_type], mask: TileTensor[Engine=mask.Engine, linear_idx_type=mask.linear_idx_type], scale: Float32, context: DeviceContext, num_partitions: Optional[Int] = None, sink_weights: OptionalReg[TileTensor[dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None)
Run flash attention with a dense mask tensor on the current device.
Wraps the mask tensor in a MaterializedMask and delegates to the
shared dense implementation. Selects the flash-attention algorithm variant
(FA2 / FA3 / naive) based on config.algorithm and the detected GPU.
Parameters:
- dtype (
DType): Element type of the query tensor. - q_layout (
TensorLayout): Compile-time layout of the query tensor. - config (
MHAConfig[dtype]): Tile/pipeline configuration; defaults are derived from the query layout's last two dimensions. - decoding_warp_split_k (
Bool): Enable warp-level split-K for decode. - naive_kernel (
Bool): Force the fallback naive attention kernel. - sink (
Bool): Enable attention-sink mode (first tokens always attend).
Args:
- output (
TileTensor[Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Destination tensor for attention output. - q (
TileTensor[dtype, q_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): Query tensor. - k (
TileTensor[Engine=k.Engine, linear_idx_type=k.linear_idx_type]): Key tensor. - v (
TileTensor[Engine=v.Engine, linear_idx_type=v.linear_idx_type]): Value tensor. - mask (
TileTensor[Engine=mask.Engine, linear_idx_type=mask.linear_idx_type]): Dense attention mask tensor. - scale (
Float32): Softmax temperature scale applied to Q·Kᵀ. - context (
DeviceContext): GPU device context for kernel dispatch. - num_partitions (
Optional[Int]): Override the number of split-K partitions;Noneselects automatically. - sink_weights (
OptionalReg[TileTensor[dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Optional sink-token weight tensor for attention sinks.
def flash_attention[cache_t: KVCacheT, mask_t: MHAMask, dtype: DType, output_type: DType, q_tt_layout: TensorLayout, output_tt_layout: TensorLayout, valid_length_tt_layout: TensorLayout, //, config: MHAConfig[dtype] = MHAConfig(q_tt_layout.static_shape[Int((add q_tt_layout.rank, -2))], q_tt_layout.static_shape[Int((add q_tt_layout.rank, -1))], Optional(None), Optional(None), Optional(None), Optional(None), Optional(None), Int(4), Int(1), FlashAttentionAlgorithm(Int(-1)), TensorMapSwizzle.SWIZZLE_128B, Int(-1)), ragged: Bool = False, sink: Bool = False, decoding_warp_split_k: Bool = False, naive_kernel: Bool = False, cross_p: Bool = True](output: TileTensor[output_type, output_tt_layout, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[dtype, q_tt_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: cache_t, v: cache_t, mask_functor: mask_t, valid_length: TileTensor[.uint32, valid_length_tt_layout, Engine=valid_length.Engine, linear_idx_type=valid_length.linear_idx_type], scale: Float32, ctx: DeviceContext, q_max_seq_len: Optional[Int] = None, kv_input_row_offsets: OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None, num_partitions: Optional[Int] = None, sink_weights: OptionalReg[TileTensor[dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None, decode_dispatch_metadata: OptionalReg[MHADecodeDispatchMetadata] = None)
Runs flash attention with KV-cache operands.
Delegates native tensor operands to the shared KV-cache implementation.
Parameters:
- cache_t (
KVCacheT): KV-cache type backing the key and value tensors (inferred). - mask_t (
MHAMask): Attention mask type implementingMHAMask(inferred). - dtype (
DType): Element type of the query tensor (inferred). - output_type (
DType): Element type of the output tensor, which may differ fromdtypeon the native FP8 path (inferred). - q_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the query tensor (inferred). - output_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the output tensor (inferred). - valid_length_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the valid-length tensor (inferred). - config (
MHAConfig[dtype]): Tile/pipeline configuration; defaults are derived from the query layout's last two dimensions. - ragged (
Bool):Truefor ragged-batch (variable-length) inputs (defaults toFalse). - sink (
Bool):Trueto enable attention-sink mode where the first tokens always attend (defaults toFalse). - decoding_warp_split_k (
Bool):Trueto enable warp-level split-K for decode (defaults toFalse). - naive_kernel (
Bool):Trueto force the fallback naive attention kernel (defaults toFalse). - cross_p (
Bool): Cross-stage P TMEM placement in the SM100 FA4 kernel, defaulting to the measured choice; read only by the 2Q route.
Args:
- output (
TileTensor[output_type, output_tt_layout, Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Mutable destinationTileTensorfor the attention output. - q (
TileTensor[dtype, q_tt_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): QueryTileTensorwith BSHD layout. - k (
cache_t): Key operand backed by a KV cache. - v (
cache_t): Value operand backed by a KV cache. - mask_functor (
mask_t): Mask instance used to apply the attention mask. - valid_length (
TileTensor[.uint32, valid_length_tt_layout, Engine=valid_length.Engine, linear_idx_type=valid_length.linear_idx_type]): Per-sequence valid lengths for masking padded batches. - scale (
Float32): Softmax temperature scale applied to Q·Kᵀ. - ctx (
DeviceContext): GPU device context for kernel dispatch. - q_max_seq_len (
Optional[Int]): Maximum query sequence length in the batch;Noneinfers it from the KV cache. - kv_input_row_offsets (
OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Row offsets for ragged KV inputs;Nonefor self-attention. - num_partitions (
Optional[Int]): Override the number of split-K partitions;Noneselects automatically. - sink_weights (
OptionalReg[TileTensor[dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Optional sink-token weight tensor for attention sinks. - decode_dispatch_metadata (
OptionalReg[MHADecodeDispatchMetadata]): Pre-computed decode dispatch metadata;Nonerecomputes it.
def flash_attention[mask_t: MHAMask, dtype: DType, k_type: DType, v_type: DType, output_type: DType, q_tt_layout: TensorLayout, k_tt_layout: TensorLayout, v_tt_layout: TensorLayout, output_tt_layout: TensorLayout, //, config: MHAConfig[dtype] = MHAConfig(q_tt_layout.static_shape[Int(2)], q_tt_layout.static_shape[Int(3)], Optional(None), Optional(None), Optional(None), Optional(None), Optional(None), Int(4), Int(1), FlashAttentionAlgorithm(Int(-1)), TensorMapSwizzle.SWIZZLE_128B, Int(-1)), decoding_warp_split_k: Bool = False, _use_valid_length: Bool = False, _padded_ndbuffer: Bool = False, naive_kernel: Bool = False, sink: Bool = False, cross_p: Bool = True](output: TileTensor[output_type, output_tt_layout, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[dtype, q_tt_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: TileTensor[k_type, k_tt_layout, Engine=k.Engine, linear_idx_type=k.linear_idx_type], v: TileTensor[v_type, v_tt_layout, Engine=v.Engine, linear_idx_type=v.linear_idx_type], mask_functor: mask_t, scale: Float32, ctx: DeviceContext, num_partitions: Optional[Int] = None, valid_length: OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None, sink_weights: OptionalReg[TileTensor[dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None)
Runs flash attention with dense tensor operands.
Delegates native tensor operands to the shared dense implementation.
Parameters:
- mask_t (
MHAMask): Attention mask type implementingMHAMask(inferred). - dtype (
DType): Element type of the query tensor (inferred). - k_type (
DType): Element type of the key tensor (inferred). - v_type (
DType): Element type of the value tensor (inferred). - output_type (
DType): Element type of the output tensor, which may differ fromdtype(inferred). - q_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the query tensor (inferred). - k_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the key tensor (inferred). - v_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the value tensor (inferred). - output_tt_layout (
TensorLayout): Compile-timeTensorLayoutof the output tensor (inferred). - config (
MHAConfig[dtype]): Tile/pipeline configuration; defaults are derived from the query layout's last two dimensions. - decoding_warp_split_k (
Bool):Trueto enable warp-level split-K for decode (defaults toFalse). - _use_valid_length (
Bool):Trueto mask output with per-sequence lengths (defaults toFalse). - _padded_ndbuffer (
Bool):Truewhen the NBuffer holds padded dense inputs (defaults toFalse). - naive_kernel (
Bool):Trueto force the fallback naive attention kernel (defaults toFalse). - sink (
Bool):Trueto enable attention-sink mode where the first tokens always attend (defaults toFalse). - cross_p (
Bool): Cross-stage P TMEM placement in the SM100 FA4 kernel, defaulting to the measured choice; read only by the 2Q route.
Args:
- output (
TileTensor[output_type, output_tt_layout, Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Mutable destinationTileTensorfor the attention output. - q (
TileTensor[dtype, q_tt_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): QueryTileTensor. - k (
TileTensor[k_type, k_tt_layout, Engine=k.Engine, linear_idx_type=k.linear_idx_type]): KeyTileTensor. - v (
TileTensor[v_type, v_tt_layout, Engine=v.Engine, linear_idx_type=v.linear_idx_type]): ValueTileTensor. - mask_functor (
mask_t): Mask instance used to apply the attention mask. - scale (
Float32): Softmax temperature scale applied to Q·Kᵀ. - ctx (
DeviceContext): GPU device context for kernel dispatch. - num_partitions (
Optional[Int]): Override the number of split-K partitions;Noneselects automatically. - valid_length (
OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Per-sequence valid lengths for masking padded batches. - sink_weights (
OptionalReg[TileTensor[dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Optional sink-token weight tensor for attention sinks.