IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

mla_indexer_ragged_float8_paged_cp

def mla_indexer_ragged_float8_paged_cp[oi_layout: TensorLayout, os_layout: TensorLayout, q_layout: TensorLayout, qs_layout: TensorLayout, iro_layout: TensorLayout, R: SparseIndexRemap, //, dtype: DType, KCollectionT: KVCollectionT, num_heads: Int, depth: Int, top_k: Int, mask_str: StaticString, cp_size: Int, cp_interleave: Int, write_logical: Bool = False, Topk: IndexerTopk = BlockSplitTopk](output_indices: MutTileTensor[.int32, oi_layout], output_scores: MutTileTensor[.bfloat16, os_layout], q: ImmTileTensor[dtype, q_layout], q_s: ImmTileTensor[.float32, qs_layout], input_row_offsets: ImmTileTensor[.uint32, iro_layout], k_collection: KCollectionT, layer_idx: UInt32, ctx: DeviceContext, remap: R, cp_rank: ImmPointer[Int32], scores_budget_bytes: Int = Int((mul get_defined_int[StringSpan("MLA_INDEX_SCORES_BUDGET_MB"), Int(512)](), 1048576)))

Selects one decode context-parallel rank's local top-k, with scores.

Under decode context parallelism (DCP) each rank of a group stores only the positions it owns (ContextParallelLayout[cp_size, cp_interleave]), at its local slots, addressed through the same lookup table as without CP, while cache_lengths stay global. This scores only the rank's local slots, bounding each query row by the number of slots below its global causal bound, and selects their top-k: the slots' physical rows through remap, and their scores, which the group all-gathers for cp_merge_topk. A token in the group's top-k is in its owner's local top-k, so that merge is exact up to ties.

Requires the SM100 FP8 scorer and kpool == 1. With cp_size == 1 it is mla_indexer_ragged_float8_paged plus the scores.

Parameters:

  • ​oi_layout (TensorLayout): Layout of the logical index output.
  • ​os_layout (TensorLayout): Layout of the score output.
  • ​q_layout (TensorLayout): Layout of the query tensor.
  • ​qs_layout (TensorLayout): Layout of the query scales.
  • ​iro_layout (TensorLayout): Layout of the ragged query row offsets.
  • ​R (SparseIndexRemap): Type of remap.
  • ​dtype (DType): Element type of the q query tensor, an FP8 dtype.
  • ​KCollectionT (KVCollectionT): Type of the KV collection holding this rank's cached K values and K scales.
  • ​num_heads (Int): Number of attention heads per token.
  • ​depth (Int): Per-head key dimension (head size) in elements.
  • ​top_k (Int): Number of local key slots to select per token.
  • ​mask_str (StaticString): Name of the mask, MaskName.NULL or MaskName.CAUSAL.
  • ​cp_size (Int): Ranks in the context-parallel group.
  • ​cp_interleave (Int): Consecutive positions a rank owns in turn.
  • ​write_logical (Bool): Whether to write output_indices, the selected local slots. False leaves it untouched, so it may be empty.
  • ​Topk (IndexerTopk): The top-k over the score rows.

Args:

  • ​output_indices (MutTileTensor[.int32, oi_layout]): Dense [total_seq_len, top_k] local slots, -1 past a token's local candidates.
  • ​output_scores (MutTileTensor[.bfloat16, os_layout]): Dense [total_seq_len, top_k] score of each selection in bfloat16, the score buffer's own values, -inf past a token's local candidates.
  • ​q (ImmTileTensor[dtype, q_layout]): Query tensor [total_seq_len, num_heads, head_dim] in FP8.
  • ​q_s (ImmTileTensor[.float32, qs_layout]): Query scales [total_seq_len, num_heads] in float32.
  • ​input_row_offsets (ImmTileTensor[.uint32, iro_layout]): Ragged row offsets for queries [batch_size + 1].
  • ​k_collection (KCollectionT): This rank's KV collection, with global cache lengths.
  • ​layer_idx (UInt32): Layer index for retrieving cache.
  • ​ctx (DeviceContext): Device context.
  • ​remap (R): Writes each selected slot's physical row, laid out like output_indices.
  • ​cp_rank (ImmPointer[Int32]): This rank in [0, cp_size), one int32 on the device.
  • ​scores_budget_bytes (Int): Peak bytes the transient score matrix may occupy.

Was this page helpful?