For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
MLAIndexerRaggedFloat8PagedRemapCP
struct MLAIndexerRaggedFloat8PagedRemapCP
Registers the attention.mla.indexer.ragged.float8.paged.remap.cp graph op.
attention.mla.indexer.ragged.float8.paged.remap for one rank of a decode
context-parallel (DCP) group. The rank's caches hold only the positions it
owns, at their local slots (kv_cache.context_parallel), behind the
group's shared lookup tables and global cache lengths. The op selects the
top-k of those local slots, returns their physical rows, and also returns
their scores, which the group all-gathers for attention.mla.cp_merge_topk.
Implemented traits
Methods
execute
static def execute[*, num_heads: Int, depth: Int, k: Int, quantization_granularity: Int, mask_str: StaticString, dcp_size: Int, dcp_interleave: Int, write_logical: Bool = False](output_indices: T, physical_indices: T, scores: T, q: T, qs: T, input_row_offsets: T, k_blocks: T, page_stride: T, k_cache_lengths: T, k_lookup_table: T, k_max_prompt_length: T, k_max_cache_length: T, k_scales: T, scales_page_stride: T, k_scales_lookup_table: T, layer_idx: UInt32, mla_kv_blocks: T, mla_lookup_table: T, dcp_rank: T, ctx: DeviceContext)
Selects this rank's local top-k physical rows and their scores.
Parameters:
- num_heads (
Int): Number of query attention heads. - depth (
Int): Head dimension. - k (
Int): Number of local slots to select per token. - quantization_granularity (
Int): Quantization granularity for the K cache. - mask_str (
StaticString): Mask type - either MaskName.NULL or MaskName.CAUSAL. - dcp_size (
Int): Ranks in the decode context-parallel group. - dcp_interleave (
Int): Consecutive positions a rank owns in turn. - write_logical (
Bool): Whether to writeoutput_indices, the selected local slots (defaults toFalse, which allows no rows).
Args:
- output_indices (
T): Selected local slots[total_seq_len, k]. - physical_indices (
T): Physical MLA cache row of each selection[total_seq_len, k],-1past a token's local candidates. - scores (
T): Each selection's bf16 score[total_seq_len, k],-infpast a token's local candidates. - q (
T): Query tensor [total_seq_len, num_heads, depth] in FP8. - qs (
T): Query scales [total_seq_len, num_heads] in float32. - input_row_offsets (
T): Ragged row offsets [batch_size + 1] for queries. - k_blocks (
T): This rank's paged indexer K cache blocks in FP8. - page_stride (
T): Page-to-page distance in elements; -1 when packed. - k_cache_lengths (
T): Global cache lengths [batch_size]. - k_lookup_table (
T): Indexer cache page lookup table. - k_max_prompt_length (
T): Max prompt (query) length scalar tensor [1]. - k_max_cache_length (
T): Global max cache length scalar tensor [1]. - k_scales (
T): K scale blocks matching k_blocks shape with scale values. - scales_page_stride (
T): The same, for the scales leaf. - k_scales_lookup_table (
T): Page lookup table for the scale blocks. - layer_idx (
UInt32): Layer index for retrieving the correct cache layer. - mla_kv_blocks (
T): This rank's MLA cache blocks. - mla_lookup_table (
T): The MLA cache's page lookup table[batch_size, pages_per_seq]. - dcp_rank (
T): This rank in[0, dcp_size), int32[1]on the device. - ctx (
DeviceContext): Device context for GPU execution.