IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

FlashAttentionCPU

struct FlashAttentionCPU

Registers the CPU kernel of the attention.mha.no_cache graph op with the graph compiler.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[MaskIns: TypeList[MaskIns.values], //, mask_str: StaticString, local_window_size: Int = Int(-1)](output: T, q: T, k: T, v: T, scale: Float32, mask_operands: TensorTuple[*MaskIns.values], ctx: DeviceContext)

attention.mha.no_cache on CPU: the GPU kernel's computation, with k and v read through loads so that their producers fuse into them.

Parameters:

  • ​MaskIns (TypeList[values]): The mask operand types.
  • ​mask_str (StaticString): Attention mask type string passed to dispatch_mask.
  • ​local_window_size (Int): Sliding window size for windowed attention (defaults to -1, meaning no window).

Args:

  • ​output (T): Output attention tensor with the same shape as q.
  • ​q (T): Query tensor in BSHD layout.
  • ​k (T): Key tensor in BSHD layout, or BShD for grouped attention.
  • ​v (T): Value tensor in BSHD layout, or BShD for grouped attention.
  • ​scale (Float32): Scaling factor applied to attention scores before softmax.
  • ​mask_operands (TensorTuple[*MaskIns.values]): The mask's operands; dispatch_mask lists what each mask reads.
  • ​ctx (DeviceContext): Device context for CPU execution.

Was this page helpful?