For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
RaggedFlashAttentionGPU
struct RaggedFlashAttentionGPU
Registers the attention.mha.ragged.no_cache graph op with the graph compiler.
Implemented traits
Methods
execute
static def execute[MaskIns: TypeList[MaskIns.values], //, mask_str: StaticString, local_window_size: Int = Int(-1)](output: T, q: T, k: T, v: T, input_row_offsets: T, q_max_seq_len: T, scale: Float32, mask_operands: TensorTuple[*MaskIns.values], ctx: DeviceContext)
attention.mha.ragged.no_cache computes flash attention for ragged inputs without KV cache.
The inputs q, k, v are in ragged format with shape [total_seq_len, num_heads, head_dim]. input_row_offsets indicates where each sequence starts and ends in the ragged tensors.
Parameters:
- MaskIns (
TypeList[values]): The mask operand types. - mask_str (
StaticString): Attention mask type string passed todispatch_maskto select the mask functor. - local_window_size (
Int): Sliding window size for windowed attention (defaults to -1, meaning no window).
Args:
- output (
T): Output attention tensor with the same shape asq. - q (
T): Query tensor in ragged format with shape [total_seq_len, num_heads, head_dim]. - k (
T): Key tensor in ragged format with shape [total_seq_len, num_heads, head_dim]. - v (
T): Value tensor in ragged format with shape [total_seq_len, num_heads, head_dim]. - input_row_offsets (
T): Row offsets [batch_size + 1] marking the start and end of each sequence in the ragged tensors. - q_max_seq_len (
T): Maximum query sequence length across the batch. - scale (
Float32): Scaling factor applied to attention scores before softmax. - mask_operands (
TensorTuple[*MaskIns.values]): The mask's operands;dispatch_masklists what each mask reads. - ctx (
DeviceContext): Device context for GPU execution.