For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
MLAIndexerRaggedFloat8Paged
struct MLAIndexerRaggedFloat8Paged
Registers the attention.mla.indexer.ragged.float8.paged graph op with the graph compiler.
Implemented traits
Methods
execute
static def execute[*, num_heads: Int, depth: Int, k: Int, quantization_granularity: Int, mask_str: StaticString, kpool: Int = Int(1), deterministic: Bool = False](output_indices: T, q: T, qs: T, input_row_offsets: T, k_blocks: T, page_stride: T, k_cache_lengths: T, k_lookup_table: T, k_max_prompt_length: T, k_max_cache_length: T, k_scales: T, scales_page_stride: T, k_scales_lookup_table: T, layer_idx: UInt32, mixed_step_split: T, ctx: DeviceContext)
Compute FP8 attention scores and return top-k key indices per token.
This kernel is designed for Multi-head Latent Attention (MLA) architectures. It computes FP8 matmul between queries and cached keys (with scales), applies masking, and returns the indices of the top-k highest-scoring keys per token. Scores are aggregated (summed) across all attention heads.
Parameters:
- num_heads (
Int): Number of query attention heads (must be 128). - depth (
Int): Head dimension (must be 128). - k (
Int): Number of top indices to return per token. - quantization_granularity (
Int): Quantization granularity for the K cache. - mask_str (
StaticString): Mask type - either MaskName.NULL (no mask) or MaskName.CAUSAL. - kpool (
Int): Tokens per pooled cache row.1scores one row per token;k > 1scores one pooled key perkpoolconsecutive tokens, so the K cache holds pooled keys andkcounts pools. - deterministic (
Bool): Whether the top-k must place the same columns in the same slots on every launch (defaults toFalse). A consumer that sums over the slots in order, like the sparse MLA softmax, is then bit-reproducible too.
Args:
- output_indices (
T): Output tensor [total_seq_len, top_k] containing top-k key indices per token. Invalid positions (where there are fewer than top_k valid keys) are filled with -1. - q (
T): Query tensor [total_seq_len, num_heads, depth] in FP8. - qs (
T): Query scales [total_seq_len, num_heads] in float32. - input_row_offsets (
T): Ragged row offsets [batch_size + 1] for queries. - k_blocks (
T): Paged K cache blocks [num_blocks, 1, num_layers, page_size, num_heads, head_size] in FP8. - page_stride (
T): Page-to-page distance in elements; -1 when packed. - k_cache_lengths (
T): Cache lengths [batch_size] - number of cached tokens per sequence. - k_lookup_table (
T): Page lookup table [batch_size, pages_per_seq] mapping sequence pages to block indices. - k_max_prompt_length (
T): Max prompt (query) length scalar tensor [1]. - k_max_cache_length (
T): Max cache length scalar tensor [1]. - k_scales (
T): K scale blocks matching k_blocks shape with scale values. - scales_page_stride (
T): The same, for the scales leaf. - k_scales_lookup_table (
T): Page lookup table for the scale blocks [batch_size, pages_per_seq]. Equal tok_lookup_tablewhen the scales share the values' block-id space. - layer_idx (
UInt32): Layer index for retrieving the correct cache layer. - mixed_step_split (
T): The KV cache's split record; zeros keep the whole-batch route. - ctx (
DeviceContext): Device context for GPU execution.