IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

flash_attention

def flash_attention[dtype: DType, q_layout: TensorLayout, //, config: MHAConfig[dtype] = MHAConfig(q_layout.static_shape[Int(2)], q_layout.static_shape[Int(3)], Optional(None), Optional(None), Optional(None), Optional(None), Optional(None), Int(4), Int(1), FlashAttentionAlgorithm(Int(-1)), TensorMapSwizzle.SWIZZLE_128B, Int(-1)), decoding_warp_split_k: Bool = False, naive_kernel: Bool = False, sink: Bool = False](output: TileTensor[Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[dtype, q_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: TileTensor[Engine=k.Engine, linear_idx_type=k.linear_idx_type], v: TileTensor[Engine=v.Engine, linear_idx_type=v.linear_idx_type], mask: TileTensor[Engine=mask.Engine, linear_idx_type=mask.linear_idx_type], scale: Float32, context: DeviceContext, num_partitions: Optional[Int] = None, sink_weights: OptionalReg[TileTensor[dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None)

Run flash attention with a dense mask tensor on the current device.

Wraps the mask tensor in a MaterializedMask and delegates to the shared dense implementation. Selects the flash-attention algorithm variant (FA2 / FA3 / naive) based on config.algorithm and the detected GPU.

Parameters:

  • ​dtype (DType): Element type of the query tensor.
  • ​q_layout (TensorLayout): Compile-time layout of the query tensor.
  • ​config (MHAConfig[dtype]): Tile/pipeline configuration; defaults are derived from the query layout's last two dimensions.
  • ​decoding_warp_split_k (Bool): Enable warp-level split-K for decode.
  • ​naive_kernel (Bool): Force the fallback naive attention kernel.
  • ​sink (Bool): Enable attention-sink mode (first tokens always attend).

Args:

def flash_attention[cache_t: KVCacheT, mask_t: MHAMask, dtype: DType, output_type: DType, q_tt_layout: TensorLayout, output_tt_layout: TensorLayout, valid_length_tt_layout: TensorLayout, //, config: MHAConfig[dtype] = MHAConfig(q_tt_layout.static_shape[Int((add q_tt_layout.rank, -2))], q_tt_layout.static_shape[Int((add q_tt_layout.rank, -1))], Optional(None), Optional(None), Optional(None), Optional(None), Optional(None), Int(4), Int(1), FlashAttentionAlgorithm(Int(-1)), TensorMapSwizzle.SWIZZLE_128B, Int(-1)), ragged: Bool = False, sink: Bool = False, decoding_warp_split_k: Bool = False, naive_kernel: Bool = False, cross_p: Bool = True](output: TileTensor[output_type, output_tt_layout, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[dtype, q_tt_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: cache_t, v: cache_t, mask_functor: mask_t, valid_length: TileTensor[.uint32, valid_length_tt_layout, Engine=valid_length.Engine, linear_idx_type=valid_length.linear_idx_type], scale: Float32, ctx: DeviceContext, q_max_seq_len: Optional[Int] = None, kv_input_row_offsets: OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None, num_partitions: Optional[Int] = None, sink_weights: OptionalReg[TileTensor[dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None, decode_dispatch_metadata: OptionalReg[MHADecodeDispatchMetadata] = None)

Runs flash attention with KV-cache operands.

Delegates native tensor operands to the shared KV-cache implementation.

Parameters:

  • ​cache_t (KVCacheT): KV-cache type backing the key and value tensors (inferred).
  • ​mask_t (MHAMask): Attention mask type implementing MHAMask (inferred).
  • ​dtype (DType): Element type of the query tensor (inferred).
  • ​output_type (DType): Element type of the output tensor, which may differ from dtype on the native FP8 path (inferred).
  • ​q_tt_layout (TensorLayout): Compile-time TensorLayout of the query tensor (inferred).
  • ​output_tt_layout (TensorLayout): Compile-time TensorLayout of the output tensor (inferred).
  • ​valid_length_tt_layout (TensorLayout): Compile-time TensorLayout of the valid-length tensor (inferred).
  • ​config (MHAConfig[dtype]): Tile/pipeline configuration; defaults are derived from the query layout's last two dimensions.
  • ​ragged (Bool): True for ragged-batch (variable-length) inputs (defaults to False).
  • ​sink (Bool): True to enable attention-sink mode where the first tokens always attend (defaults to False).
  • ​decoding_warp_split_k (Bool): True to enable warp-level split-K for decode (defaults to False).
  • ​naive_kernel (Bool): True to force the fallback naive attention kernel (defaults to False).
  • ​cross_p (Bool): Cross-stage P TMEM placement in the SM100 FA4 kernel, defaulting to the measured choice; read only by the 2Q route.

Args:

def flash_attention[mask_t: MHAMask, dtype: DType, k_type: DType, v_type: DType, output_type: DType, q_tt_layout: TensorLayout, k_tt_layout: TensorLayout, v_tt_layout: TensorLayout, output_tt_layout: TensorLayout, //, config: MHAConfig[dtype] = MHAConfig(q_tt_layout.static_shape[Int(2)], q_tt_layout.static_shape[Int(3)], Optional(None), Optional(None), Optional(None), Optional(None), Optional(None), Int(4), Int(1), FlashAttentionAlgorithm(Int(-1)), TensorMapSwizzle.SWIZZLE_128B, Int(-1)), decoding_warp_split_k: Bool = False, _use_valid_length: Bool = False, _padded_ndbuffer: Bool = False, naive_kernel: Bool = False, sink: Bool = False, cross_p: Bool = True](output: TileTensor[output_type, output_tt_layout, Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[dtype, q_tt_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: TileTensor[k_type, k_tt_layout, Engine=k.Engine, linear_idx_type=k.linear_idx_type], v: TileTensor[v_type, v_tt_layout, Engine=v.Engine, linear_idx_type=v.linear_idx_type], mask_functor: mask_t, scale: Float32, ctx: DeviceContext, num_partitions: Optional[Int] = None, valid_length: OptionalReg[TileTensor[.uint32, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None, sink_weights: OptionalReg[TileTensor[dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None)

Runs flash attention with dense tensor operands.

Delegates native tensor operands to the shared dense implementation.

Parameters:

  • ​mask_t (MHAMask): Attention mask type implementing MHAMask (inferred).
  • ​dtype (DType): Element type of the query tensor (inferred).
  • ​k_type (DType): Element type of the key tensor (inferred).
  • ​v_type (DType): Element type of the value tensor (inferred).
  • ​output_type (DType): Element type of the output tensor, which may differ from dtype (inferred).
  • ​q_tt_layout (TensorLayout): Compile-time TensorLayout of the query tensor (inferred).
  • ​k_tt_layout (TensorLayout): Compile-time TensorLayout of the key tensor (inferred).
  • ​v_tt_layout (TensorLayout): Compile-time TensorLayout of the value tensor (inferred).
  • ​output_tt_layout (TensorLayout): Compile-time TensorLayout of the output tensor (inferred).
  • ​config (MHAConfig[dtype]): Tile/pipeline configuration; defaults are derived from the query layout's last two dimensions.
  • ​decoding_warp_split_k (Bool): True to enable warp-level split-K for decode (defaults to False).
  • ​_use_valid_length (Bool): True to mask output with per-sequence lengths (defaults to False).
  • ​_padded_ndbuffer (Bool): True when the NBuffer holds padded dense inputs (defaults to False).
  • ​naive_kernel (Bool): True to force the fallback naive attention kernel (defaults to False).
  • ​sink (Bool): True to enable attention-sink mode where the first tokens always attend (defaults to False).
  • ​cross_p (Bool): Cross-stage P TMEM placement in the SM100 FA4 kernel, defaulting to the measured choice; read only by the 2Q route.

Args:

Was this page helpful?