For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
MLAIndexerRaggedFloat8PagedRemapCPNVFP4
struct MLAIndexerRaggedFloat8PagedRemapCPNVFP4
Registers the attention.mla.indexer.ragged.float8.paged.remap.cp.nvfp4 graph op.
attention.mla.indexer.ragged.float8.paged.remap.cp for an NVFP4 MLA
cache, whose scales are paged apart from its values. As in
attention.mla.indexer.ragged.float8.paged.remap.nvfp4, each row of
physical_indices is 2 * k wide: the selected local slots' physical
rows in the values, then their rows in the scale pool, both resolved
through the group's shared lookup tables at the rank's local slot. Feed
the rows and the scores to attention.mla.cp_merge_topk with
halves = 2.
Implemented traits
Methods
execute
static def execute[*, num_heads: Int, depth: Int, k: Int, quantization_granularity: Int, mask_str: StaticString, dcp_size: Int, dcp_interleave: Int, write_logical: Bool = False, deterministic: Bool = False](output_indices: T, physical_indices: T, scores: T, q: T, qs: T, input_row_offsets: T, k_blocks: T, page_stride: T, k_cache_lengths: T, k_lookup_table: T, k_max_prompt_length: T, k_max_cache_length: T, k_scales: T, scales_page_stride: T, k_scales_lookup_table: T, layer_idx: UInt32, mla_kv_blocks: T, mla_lookup_table: T, mla_kv_scales: T, mla_scales_page_stride: T, mla_scales_lookup_table: T, dcp_rank: T, ctx: DeviceContext)
Selects this rank's local top-k value and scale rows, with scores.
Parameters:
- num_heads (
Int): Number of query attention heads. - depth (
Int): Head dimension. - k (
Int): Number of local slots to select per token. - quantization_granularity (
Int): Quantization granularity for the K cache. - mask_str (
StaticString): Mask type - either MaskName.NULL or MaskName.CAUSAL. - dcp_size (
Int): Ranks in the decode context-parallel group. - dcp_interleave (
Int): Consecutive positions a rank owns in turn. - write_logical (
Bool): Whether to writeoutput_indices, the selected local slots (defaults toFalse, which allows no rows). - deterministic (
Bool): Whether the top-k must place the same columns in the same slots on every launch (defaults toFalse).
Args:
- output_indices (
T): Selected local slots[total_seq_len, k]. - physical_indices (
T):[total_seq_len, 2 * k]. Columns[0, k)are each selection's physical MLA cache row, columns[k, 2 * k)its row in the MLA scale pool;-1past a token's local candidates or where a page is unmapped. - scores (
T): Each selection's bf16 score[total_seq_len, k],-infpast a token's local candidates. - q (
T): Query tensor [total_seq_len, num_heads, depth] in FP8. - qs (
T): Query scales [total_seq_len, num_heads] in float32. - input_row_offsets (
T): Ragged row offsets [batch_size + 1] for queries. - k_blocks (
T): This rank's paged indexer K cache blocks in FP8. - page_stride (
T): Page-to-page distance in elements; -1 when packed. - k_cache_lengths (
T): Global cache lengths [batch_size]. - k_lookup_table (
T): Indexer cache page lookup table. - k_max_prompt_length (
T): Max prompt (query) length scalar tensor [1]. - k_max_cache_length (
T): Global max cache length scalar tensor [1]. - k_scales (
T): K scale blocks matching k_blocks shape with scale values. - scales_page_stride (
T): The same, for the scales leaf. - k_scales_lookup_table (
T): Page lookup table for the scale blocks. - layer_idx (
UInt32): Layer index for retrieving the correct cache layer. - mla_kv_blocks (
T): This rank's MLA cache blocks; their page count is the invalid block id and their page size the remap's. - mla_lookup_table (
T): The MLA cache's page lookup table[batch_size, pages_per_seq]. - mla_kv_scales (
T): This rank's MLA scale blocks; their page count is the invalid scale block id. - mla_scales_page_stride (
T): Page-to-page distance ofmla_kv_scalesin elements; -1 when packed. - mla_scales_lookup_table (
T): The MLA cache's scales page lookup table[batch_size, pages_per_seq]. - dcp_rank (
T): This rank in[0, dcp_size), int32[1]on the device. - ctx (
DeviceContext): Device context for GPU execution.