For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
mla_indexer_ragged_float8_paged_cp
def mla_indexer_ragged_float8_paged_cp[oi_layout: TensorLayout, os_layout: TensorLayout, q_layout: TensorLayout, qs_layout: TensorLayout, iro_layout: TensorLayout, R: SparseIndexRemap, //, dtype: DType, KCollectionT: KVCollectionT, num_heads: Int, depth: Int, top_k: Int, mask_str: StaticString, cp_size: Int, cp_interleave: Int, write_logical: Bool = False, Topk: IndexerTopk = BlockSplitTopk](output_indices: MutTileTensor[.int32, oi_layout], output_scores: MutTileTensor[.bfloat16, os_layout], q: ImmTileTensor[dtype, q_layout], q_s: ImmTileTensor[.float32, qs_layout], input_row_offsets: ImmTileTensor[.uint32, iro_layout], k_collection: KCollectionT, layer_idx: UInt32, ctx: DeviceContext, remap: R, cp_rank: ImmPointer[Int32], scores_budget_bytes: Int = Int((mul get_defined_int[StringSpan("MLA_INDEX_SCORES_BUDGET_MB"), Int(512)](), 1048576)))
Selects one decode context-parallel rank's local top-k, with scores.
Under decode context parallelism (DCP) each rank of a group stores only
the positions it owns (ContextParallelLayout[cp_size, cp_interleave]),
at its local slots, addressed through the same lookup table as without
CP, while cache_lengths stay global. This scores only the rank's local
slots, bounding each query row by the number of slots below its global
causal bound, and selects their top-k: the slots' physical rows through
remap, and their scores, which the group all-gathers for
cp_merge_topk. A token in the group's top-k is in its owner's local
top-k, so that merge is exact up to ties.
Requires the SM100 FP8 scorer and kpool == 1. With cp_size == 1 it is
mla_indexer_ragged_float8_paged plus the scores.
Parameters:
- oi_layout (
TensorLayout): Layout of the logical index output. - os_layout (
TensorLayout): Layout of the score output. - q_layout (
TensorLayout): Layout of the query tensor. - qs_layout (
TensorLayout): Layout of the query scales. - iro_layout (
TensorLayout): Layout of the ragged query row offsets. - R (
SparseIndexRemap): Type ofremap. - dtype (
DType): Element type of theqquery tensor, an FP8 dtype. - KCollectionT (
KVCollectionT): Type of the KV collection holding this rank's cached K values and K scales. - num_heads (
Int): Number of attention heads per token. - depth (
Int): Per-head key dimension (head size) in elements. - top_k (
Int): Number of local key slots to select per token. - mask_str (
StaticString): Name of the mask,MaskName.NULLorMaskName.CAUSAL. - cp_size (
Int): Ranks in the context-parallel group. - cp_interleave (
Int): Consecutive positions a rank owns in turn. - write_logical (
Bool): Whether to writeoutput_indices, the selected local slots.Falseleaves it untouched, so it may be empty. - Topk (
IndexerTopk): The top-k over the score rows.
Args:
- output_indices (
MutTileTensor[.int32, oi_layout]): Dense[total_seq_len, top_k]local slots,-1past a token's local candidates. - output_scores (
MutTileTensor[.bfloat16, os_layout]): Dense[total_seq_len, top_k]score of each selection in bfloat16, the score buffer's own values,-infpast a token's local candidates. - q (
ImmTileTensor[dtype, q_layout]): Query tensor[total_seq_len, num_heads, head_dim]in FP8. - q_s (
ImmTileTensor[.float32, qs_layout]): Query scales[total_seq_len, num_heads]in float32. - input_row_offsets (
ImmTileTensor[.uint32, iro_layout]): Ragged row offsets for queries[batch_size + 1]. - k_collection (
KCollectionT): This rank's KV collection, with global cache lengths. - layer_idx (
UInt32): Layer index for retrieving cache. - ctx (
DeviceContext): Device context. - remap (
R): Writes each selected slot's physical row, laid out likeoutput_indices. - cp_rank (
ImmPointer[Int32]): This rank in[0, cp_size), oneint32on the device. - scores_budget_bytes (
Int): Peak bytes the transient score matrix may occupy.