For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
flash_attention_ragged
def flash_attention_ragged[mask_t: MHAMask, type: DType, q_layout: TensorLayout, //, config: MHAConfig[type] = MHAConfig(q_layout.static_shape[Int((add q_layout.rank, -2))], q_layout.static_shape[Int((add q_layout.rank, -1))], Optional(None), Optional(None), Optional(None), Optional(None), Optional(None), Int(4), Int(1), FlashAttentionAlgorithm(Int(-1)), TensorMapSwizzle.SWIZZLE_128B, Int(-1)), decoding_warp_split_k: Bool = False, naive_kernel: Bool = False](output: TileTensor[Engine=output.Engine, linear_idx_type=output.linear_idx_type], q: TileTensor[type, q_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type], k: TileTensor[Engine=k.Engine, linear_idx_type=k.linear_idx_type], v: TileTensor[Engine=v.Engine, linear_idx_type=v.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], max_prompt_len: TileTensor[.uint32, Engine=max_prompt_len.Engine, linear_idx_type=max_prompt_len.linear_idx_type], mask_functor: mask_t, scale: Float32, ctx: DeviceContext, num_partitions: Optional[Int] = None)
Run flash attention on ragged (variable-length) batch inputs.
Accepts Q/K/V as flat rank-3 tensors with shape
[total_seq_len, num_heads, head_dim] and a CSR-style
input_row_offsets tensor of length batch + 1 that encodes per-sequence
boundaries. Dispatches the same kernel paths as the dense overload but
wraps K/V in RaggedMHAOperand adapters.
Parameters:
- mask_t (
MHAMask): Attention mask type implementingMHAMask. - type (
DType): Element data type for Q/K/V and the output. - q_layout (
TensorLayout): Compile-time layout of the query tensor. - config (
MHAConfig[type]): Tile/pipeline configuration; defaults from query shape. - decoding_warp_split_k (
Bool): Enable warp-level split-K for decode. - naive_kernel (
Bool): Force the fallback naive attention kernel.
Args:
- output (
TileTensor[Engine=output.Engine, linear_idx_type=output.linear_idx_type]): Mutable output tensor, same shape as Q. - q (
TileTensor[type, q_layout, Engine=q.Engine, linear_idx_type=q.linear_idx_type]): Query tensor[total_seq_len, num_heads, head_dim]. - k (
TileTensor[Engine=k.Engine, linear_idx_type=k.linear_idx_type]): Key tensor[total_seq_len, kv_heads, head_dim]. - v (
TileTensor[Engine=v.Engine, linear_idx_type=v.linear_idx_type]): Value tensor[total_seq_len, kv_heads, head_dim]. - input_row_offsets (
TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type]): Contiguous scalar CSR row offsets[batch + 1]. - max_prompt_len (
TileTensor[.uint32, Engine=max_prompt_len.Engine, linear_idx_type=max_prompt_len.linear_idx_type]): Scalar tensor holding the maximum sequence length. - mask_functor (
mask_t): Mask instance. - scale (
Float32): Softmax temperature scale. - ctx (
DeviceContext): GPU device context. - num_partitions (
Optional[Int]): Override split-K partition count;Nonefor auto.
Raises:
If input_row_offsets does not have unit stride.