For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
flash_attention
def flash_attention[dtype: DType, rank: Int, mask_rank: Int, q_origin: ImmOrigin, output_origin: MutOrigin, InputKFnType: def[simd_width: Int, rank: Int](IndexList[rank]) -> SIMD[dtype, simd_width], InputVFnType: def[simd_width: Int, rank: Int](IndexList[rank]) -> SIMD[dtype, simd_width], InputMaskFnType: def[simd_width: Int, mask_rank: Int](IndexList[mask_rank]) -> SIMD[dtype, simd_width], //](input_k_fn: InputKFnType, input_v_fn: InputVFnType, input_mask_fn: InputMaskFnType, q: TileTensor[dtype, q_origin, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k_shape: IndexList[rank], v_shape: IndexList[rank], mask_shape: IndexList[mask_rank], output: TileTensor[dtype, output_origin, Engine=output.Engine, linear_idx_type=output.linear_idx_type], scale: Float32, sink_weights: OptionalReg[TileTensor[dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]] = None, ctx: Optional[DeviceContextBase[CompilationTarget()]] = None)
Computes scaled dot-product flash attention on CPU for the given query, key, value, and mask accessors.
Parameters:
- dtype (
DType): The element type of the query, key, value, and output tensors (inferred). - rank (
Int): The number of dimensions in the query, key, value, and output tensors, either 3 or 4 (inferred). - mask_rank (
Int): The number of dimensions in the attention mask tensor (inferred). - q_origin (
ImmOrigin): The memory origin of the read-only query tensor (inferred). - output_origin (
MutOrigin): The memory origin of the writable output tensor (inferred). - InputKFnType (
def[simd_width: Int, rank: Int](IndexList[rank]) -> SIMD[dtype, simd_width]): The type ofinput_k_fn(inferred). - InputVFnType (
def[simd_width: Int, rank: Int](IndexList[rank]) -> SIMD[dtype, simd_width]): The type ofinput_v_fn(inferred). - InputMaskFnType (
def[simd_width: Int, mask_rank: Int](IndexList[mask_rank]) -> SIMD[dtype, simd_width]): The type ofinput_mask_fn(inferred).
Args:
- input_k_fn (
InputKFnType): Function loading aSIMDvector of key elements at a givenIndexListindex. - input_v_fn (
InputVFnType): Function loading aSIMDvector of value elements at a givenIndexListindex. - input_mask_fn (
InputMaskFnType): Function loading aSIMDvector of additive mask values at a givenIndexListindex. - q (
TileTensor[dtype, q_origin, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): Query tensor in BSHD or BSD layout. - k_shape (
IndexList[rank]): Shape of the key tensor. - v_shape (
IndexList[rank]): Shape of the value tensor. - mask_shape (
IndexList[mask_rank]): Shape of the attention mask tensor. - output (
TileTensor[dtype, output_origin, Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Output tensor to write the attention results into. - scale (
Float32): Scaling factor applied to the query-key dot products. - sink_weights (
OptionalReg[TileTensor[dtype, Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], ImmutAnyOrigin]]): Optional per-head attention sink weights. - ctx (
Optional[DeviceContextBase[CompilationTarget()]]): Optional device context for controlling parallelism.