For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
MLAIndexerRaggedFloat8PagedRemap
struct MLAIndexerRaggedFloat8PagedRemap
Registers the attention.mla.indexer.ragged.float8.paged.remap graph op.
attention.mla.indexer.ragged.float8.paged plus a second output: each selected
position's physical row in the MLA cache, exactly what
paged_sparse_kv_index_remap would compute from the logical output. A
sparse MLA op given it skips that remap.
Implemented traits
Methods
execute
static def execute[*, num_heads: Int, depth: Int, k: Int, quantization_granularity: Int, mask_str: StaticString, write_logical: Bool = False, deterministic: Bool = False](output_indices: T, physical_indices: T, q: T, qs: T, input_row_offsets: T, k_blocks: T, page_stride: T, k_cache_lengths: T, k_lookup_table: T, k_max_prompt_length: T, k_max_cache_length: T, k_scales: T, scales_page_stride: T, k_scales_lookup_table: T, layer_idx: UInt32, mla_kv_blocks: T, mla_lookup_table: T, mixed_step_split: T, ctx: DeviceContext)
Computes top-k key positions and their physical MLA cache rows.
Parameters:
- num_heads (
Int): Number of query attention heads. - depth (
Int): Head dimension. - k (
Int): Number of top indices to return per token. - quantization_granularity (
Int): Quantization granularity for the K cache. - mask_str (
StaticString): Mask type - either MaskName.NULL or MaskName.CAUSAL. - write_logical (
Bool): Whether to writeoutput_indices(defaults toFalse). WithFalseonly the physical rows are written, andoutput_indicesmay have no rows. - deterministic (
Bool): Whether the top-k must place the same columns in the same slots on every launch (defaults toFalse).
Args:
- output_indices (
T): Logical top-k key positions[total_seq_len, k],-1past a token's candidates. - physical_indices (
T): Physical MLA cache row of eachoutput_indicesslot,block_id * page_size + offset,-1where the position is-1or its page is unmapped. - q (
T): Query tensor [total_seq_len, num_heads, depth] in FP8. - qs (
T): Query scales [total_seq_len, num_heads] in float32. - input_row_offsets (
T): Ragged row offsets [batch_size + 1] for queries. - k_blocks (
T): Paged indexer K cache blocks in FP8. - page_stride (
T): Page-to-page distance in elements; -1 when packed. - k_cache_lengths (
T): Cache lengths [batch_size]. - k_lookup_table (
T): Indexer cache page lookup table. - k_max_prompt_length (
T): Max prompt (query) length scalar tensor [1]. - k_max_cache_length (
T): Max cache length scalar tensor [1]. - k_scales (
T): K scale blocks matching k_blocks shape with scale values. - scales_page_stride (
T): The same, for the scales leaf. - k_scales_lookup_table (
T): Page lookup table for the scale blocks. - layer_idx (
UInt32): Layer index for retrieving the correct cache layer. - mla_kv_blocks (
T): The MLA cache's blocks; their page count is the invalid block id and their page size the remap's. - mla_lookup_table (
T): The MLA cache's page lookup table[batch_size, pages_per_seq]. - mixed_step_split (
T): The KV cache's split record; zeros keep the whole-batch route. - ctx (
DeviceContext): Device context for GPU execution.