IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

flash_attention

def flash_attention[dtype: DType, rank: Int, mask_rank: Int, q_origin: ImmOrigin, output_origin: MutOrigin, InputKFnType: def[simd_width: Int](Coord[**?]) -> SIMD[dtype, simd_width], InputVFnType: def[simd_width: Int](Coord[**?]) -> SIMD[dtype, simd_width], InputMaskFnType: def[simd_width: Int](Coord[**?]) -> SIMD[dtype, simd_width], //](input_k_fn: InputKFnType, input_v_fn: InputVFnType, input_mask_fn: InputMaskFnType, q: TileTensor[dtype, q_origin, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k_shape: IndexList[rank], v_shape: IndexList[rank], mask_shape: IndexList[mask_rank], output: TileTensor[dtype, output_origin, Engine=output.Engine, linear_idx_type=output.linear_idx_type], scale: Float32, sink_weights: OptionalReg[TileTensor[dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None, ctx: Optional[DeviceContext[CompilationTarget()]] = None)

Computes scaled dot-product flash attention on CPU for the given query, key, value, and mask accessors.

Parameters:

  • ​dtype (DType): The element type of the query, key, value, and output tensors (inferred).
  • ​rank (Int): The number of dimensions in the query, key, value, and output tensors, either 3 or 4 (inferred).
  • ​mask_rank (Int): The number of dimensions in the attention mask tensor (inferred).
  • ​q_origin (ImmOrigin): The memory origin of the read-only query tensor (inferred).
  • ​output_origin (MutOrigin): The memory origin of the writable output tensor (inferred).
  • ​InputKFnType (def[simd_width: Int](Coord[**?]) -> SIMD[dtype, simd_width]): The type of input_k_fn (inferred).
  • ​InputVFnType (def[simd_width: Int](Coord[**?]) -> SIMD[dtype, simd_width]): The type of input_v_fn (inferred).
  • ​InputMaskFnType (def[simd_width: Int](Coord[**?]) -> SIMD[dtype, simd_width]): The type of input_mask_fn (inferred).

Args:

def flash_attention[dtype: DType, rank: Int, q_origin: ImmOrigin, output_origin: MutOrigin, InputKFnType: def[simd_width: Int](Coord[**?]) -> SIMD[dtype, simd_width], InputVFnType: def[simd_width: Int](Coord[**?]) -> SIMD[dtype, simd_width], mask_t: MHAMask, //](input_k_fn: InputKFnType, input_v_fn: InputVFnType, q: TileTensor[dtype, q_origin, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k_shape: IndexList[rank], v_shape: IndexList[rank], output: TileTensor[dtype, output_origin, Engine=output.Engine, linear_idx_type=output.linear_idx_type], mask: mask_t, scale: Float32, ctx: Optional[DeviceContext[CompilationTarget()]] = None)

Computes scaled dot-product flash attention on CPU under an MHAMask.

Parameters:

  • ​dtype (DType): The element type of the query, key, value, and output tensors (inferred).
  • ​rank (Int): The number of dimensions in the query, key, value, and output tensors, either 3 or 4 (inferred).
  • ​q_origin (ImmOrigin): The memory origin of the read-only query tensor (inferred).
  • ​output_origin (MutOrigin): The memory origin of the writable output tensor (inferred).
  • ​InputKFnType (def[simd_width: Int](Coord[**?]) -> SIMD[dtype, simd_width]): The type of input_k_fn (inferred).
  • ​InputVFnType (def[simd_width: Int](Coord[**?]) -> SIMD[dtype, simd_width]): The type of input_v_fn (inferred).
  • ​mask_t (MHAMask): The MHAMask type (inferred).

Args:

Was this page helpful?