IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

MLAIndexerRaggedFloat8PagedRemapCP

struct MLAIndexerRaggedFloat8PagedRemapCP

Registers the attention.mla.indexer.ragged.float8.paged.remap.cp graph op.

attention.mla.indexer.ragged.float8.paged.remap for one rank of a decode context-parallel (DCP) group. The rank's caches hold only the positions it owns, at their local slots (kv_cache.context_parallel), behind the group's shared lookup tables and global cache lengths. The op selects the top-k of those local slots, returns their physical rows, and also returns their scores, which the group all-gathers for attention.mla.cp_merge_topk.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[*, num_heads: Int, depth: Int, k: Int, quantization_granularity: Int, mask_str: StaticString, dcp_size: Int, dcp_interleave: Int, write_logical: Bool = False](output_indices: T, physical_indices: T, scores: T, q: T, qs: T, input_row_offsets: T, k_blocks: T, page_stride: T, k_cache_lengths: T, k_lookup_table: T, k_max_prompt_length: T, k_max_cache_length: T, k_scales: T, scales_page_stride: T, k_scales_lookup_table: T, layer_idx: UInt32, mla_kv_blocks: T, mla_lookup_table: T, dcp_rank: T, ctx: DeviceContext)

Selects this rank's local top-k physical rows and their scores.

Parameters:

  • ​num_heads (Int): Number of query attention heads.
  • ​depth (Int): Head dimension.
  • ​k (Int): Number of local slots to select per token.
  • ​quantization_granularity (Int): Quantization granularity for the K cache.
  • ​mask_str (StaticString): Mask type - either MaskName.NULL or MaskName.CAUSAL.
  • ​dcp_size (Int): Ranks in the decode context-parallel group.
  • ​dcp_interleave (Int): Consecutive positions a rank owns in turn.
  • ​write_logical (Bool): Whether to write output_indices, the selected local slots (defaults to False, which allows no rows).

Args:

  • ​output_indices (T): Selected local slots [total_seq_len, k].
  • ​physical_indices (T): Physical MLA cache row of each selection [total_seq_len, k], -1 past a token's local candidates.
  • ​scores (T): Each selection's bf16 score [total_seq_len, k], -inf past a token's local candidates.
  • ​q (T): Query tensor [total_seq_len, num_heads, depth] in FP8.
  • ​qs (T): Query scales [total_seq_len, num_heads] in float32.
  • ​input_row_offsets (T): Ragged row offsets [batch_size + 1] for queries.
  • ​k_blocks (T): This rank's paged indexer K cache blocks in FP8.
  • ​page_stride (T): Page-to-page distance in elements; -1 when packed.
  • ​k_cache_lengths (T): Global cache lengths [batch_size].
  • ​k_lookup_table (T): Indexer cache page lookup table.
  • ​k_max_prompt_length (T): Max prompt (query) length scalar tensor [1].
  • ​k_max_cache_length (T): Global max cache length scalar tensor [1].
  • ​k_scales (T): K scale blocks matching k_blocks shape with scale values.
  • ​scales_page_stride (T): The same, for the scales leaf.
  • ​k_scales_lookup_table (T): Page lookup table for the scale blocks.
  • ​layer_idx (UInt32): Layer index for retrieving the correct cache layer.
  • ​mla_kv_blocks (T): This rank's MLA cache blocks.
  • ​mla_lookup_table (T): The MLA cache's page lookup table [batch_size, pages_per_seq].
  • ​dcp_rank (T): This rank in [0, dcp_size), int32 [1] on the device.
  • ​ctx (DeviceContext): Device context for GPU execution.

Was this page helpful?