IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

MLAIndexerRaggedFloat8PagedRemapNVFP4

struct MLAIndexerRaggedFloat8PagedRemapNVFP4

Registers the attention.mla.indexer.ragged.float8.paged.remap.nvfp4 graph op.

attention.mla.indexer.ragged.float8.paged.remap for an NVFP4 MLA cache, whose scales are paged apart from its values. Each row of physical_indices is 2 * k wide: the selected positions' physical rows in the values, then their rows in the scale pool, resolved through the cache's scales lookup table. The sparse NVFP4 MLA kernels read the second half instead of deriving a scale row from a value row.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[*, num_heads: Int, depth: Int, k: Int, quantization_granularity: Int, mask_str: StaticString, write_logical: Bool = False](output_indices: T, physical_indices: T, q: T, qs: T, input_row_offsets: T, k_blocks: T, page_stride: T, k_cache_lengths: T, k_lookup_table: T, k_max_prompt_length: T, k_max_cache_length: T, k_scales: T, scales_page_stride: T, k_scales_lookup_table: T, layer_idx: UInt32, mla_kv_blocks: T, mla_lookup_table: T, mla_kv_scales: T, mla_scales_page_stride: T, mla_scales_lookup_table: T, mixed_step_split: T, ctx: DeviceContext)

Computes top-k key positions and their value and scale rows.

Parameters:

  • ​num_heads (Int): Number of query attention heads.
  • ​depth (Int): Head dimension.
  • ​k (Int): Number of top indices to return per token.
  • ​quantization_granularity (Int): Quantization granularity for the K cache.
  • ​mask_str (StaticString): Mask type - either MaskName.NULL or MaskName.CAUSAL.
  • ​write_logical (Bool): Whether to write output_indices (defaults to False). With False only physical_indices is written, and output_indices may have no rows.

Args:

  • ​output_indices (T): Logical top-k key positions [total_seq_len, k], -1 past a token's candidates.
  • ​physical_indices (T): [total_seq_len, 2 * k]. Columns [0, k) are each output_indices slot's physical MLA cache row, block_id * page_size + offset; columns [k, 2 * k) are its row in the MLA scale pool, scale_block_id * rows_per_block + offset. -1 where the position is -1 or its page is unmapped.
  • ​q (T): Query tensor [total_seq_len, num_heads, depth] in FP8.
  • ​qs (T): Query scales [total_seq_len, num_heads] in float32.
  • ​input_row_offsets (T): Ragged row offsets [batch_size + 1] for queries.
  • ​k_blocks (T): Paged indexer K cache blocks in FP8.
  • ​page_stride (T): Page-to-page distance in elements; -1 when packed.
  • ​k_cache_lengths (T): Cache lengths [batch_size].
  • ​k_lookup_table (T): Indexer cache page lookup table.
  • ​k_max_prompt_length (T): Max prompt (query) length scalar tensor [1].
  • ​k_max_cache_length (T): Max cache length scalar tensor [1].
  • ​k_scales (T): K scale blocks matching k_blocks shape with scale values.
  • ​scales_page_stride (T): The same, for the scales leaf.
  • ​k_scales_lookup_table (T): Page lookup table for the scale blocks.
  • ​layer_idx (UInt32): Layer index for retrieving the correct cache layer.
  • ​mla_kv_blocks (T): The MLA cache's blocks; their page count is the invalid block id and their page size the remap's.
  • ​mla_lookup_table (T): The MLA cache's page lookup table [batch_size, pages_per_seq].
  • ​mla_kv_scales (T): The MLA cache's scale blocks; their page count is the invalid scale block id.
  • ​mla_scales_page_stride (T): Page-to-page distance of mla_kv_scales in elements; -1 when packed.
  • ​mla_scales_lookup_table (T): The MLA cache's scales page lookup table [batch_size, pages_per_seq].
  • ​mixed_step_split (T): The KV cache's split record; zeros keep the whole-batch route.
  • ​ctx (DeviceContext): Device context for GPU execution.

Was this page helpful?