IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

MLAIndexerRaggedFloat8PagedRemapFiltered

struct MLAIndexerRaggedFloat8PagedRemapFiltered

Registers the attention.mla.indexer.ragged.float8.paged.remap.filtered graph op.

attention.mla.indexer.ragged.float8.paged.remap with the filtered top-k select (nn.topk_filtered.FilteredTopk): bounded rows that fill the GPU keep only the columns at or above a sampled bound before the exact select. Same operands, parameters and outputs. Hy4's indexer calls it at every step; other score windows keep the shared select.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[*, num_heads: Int, depth: Int, k: Int, quantization_granularity: Int, mask_str: StaticString, write_logical: Bool = False](output_indices: T, physical_indices: T, q: T, qs: T, input_row_offsets: T, k_blocks: T, page_stride: T, k_cache_lengths: T, k_lookup_table: T, k_max_prompt_length: T, k_max_cache_length: T, k_scales: T, scales_page_stride: T, k_scales_lookup_table: T, layer_idx: UInt32, mla_kv_blocks: T, mla_lookup_table: T, ctx: DeviceContext)

Computes top-k key positions and their physical MLA cache rows.

Parameters:

  • ​num_heads (Int): Number of query attention heads.
  • ​depth (Int): Head dimension.
  • ​k (Int): Number of top indices to return per token.
  • ​quantization_granularity (Int): Quantization granularity for the K cache.
  • ​mask_str (StaticString): Mask type - either MaskName.NULL or MaskName.CAUSAL.
  • ​write_logical (Bool): Whether to write output_indices (defaults to False). With False only the physical rows are written, and output_indices may have no rows.

Args:

  • ​output_indices (T): Logical top-k key positions [total_seq_len, k], -1 past a token's candidates.
  • ​physical_indices (T): Physical MLA cache row of each output_indices slot, block_id * page_size + offset, -1 where the position is -1 or its page is unmapped.
  • ​q (T): Query tensor [total_seq_len, num_heads, depth] in FP8.
  • ​qs (T): Query scales [total_seq_len, num_heads] in float32.
  • ​input_row_offsets (T): Ragged row offsets [batch_size + 1] for queries.
  • ​k_blocks (T): Paged indexer K cache blocks in FP8.
  • ​page_stride (T): Page-to-page distance in elements; -1 when packed.
  • ​k_cache_lengths (T): Cache lengths [batch_size].
  • ​k_lookup_table (T): Indexer cache page lookup table.
  • ​k_max_prompt_length (T): Max prompt (query) length scalar tensor [1].
  • ​k_max_cache_length (T): Max cache length scalar tensor [1].
  • ​k_scales (T): K scale blocks matching k_blocks shape with scale values.
  • ​scales_page_stride (T): The same, for the scales leaf.
  • ​k_scales_lookup_table (T): Page lookup table for the scale blocks.
  • ​layer_idx (UInt32): Layer index for retrieving the correct cache layer.
  • ​mla_kv_blocks (T): The MLA cache's blocks; their page count is the invalid block id and their page size the remap's.
  • ​mla_lookup_table (T): The MLA cache's page lookup table [batch_size, pages_per_seq].
  • ​ctx (DeviceContext): Device context for GPU execution.

Was this page helpful?