For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
FlashAttentionCPU
struct FlashAttentionCPU
Registers the CPU kernel of the attention.mha.no_cache graph op with the graph compiler.
Implemented traits
Methods
execute
static def execute[MaskIns: TypeList[MaskIns.values], //, mask_str: StaticString, local_window_size: Int = Int(-1)](output: T, q: T, k: T, v: T, scale: Float32, mask_operands: TensorTuple[*MaskIns.values], ctx: DeviceContext)
attention.mha.no_cache on CPU: the GPU kernel's computation, with k and v read through loads so that their producers fuse into them.
Parameters:
- MaskIns (
TypeList[values]): The mask operand types. - mask_str (
StaticString): Attention mask type string passed todispatch_mask. - local_window_size (
Int): Sliding window size for windowed attention (defaults to -1, meaning no window).
Args:
- output (
T): Output attention tensor with the same shape asq. - q (
T): Query tensor in BSHD layout. - k (
T): Key tensor in BSHD layout, or BShD for grouped attention. - v (
T): Value tensor in BSHD layout, or BShD for grouped attention. - scale (
Float32): Scaling factor applied to attention scores before softmax. - mask_operands (
TensorTuple[*MaskIns.values]): The mask's operands;dispatch_masklists what each mask reads. - ctx (
DeviceContext): Device context for CPU execution.