For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
MLAIndexerRaggedFloat8PagedRemapNVFP4
struct MLAIndexerRaggedFloat8PagedRemapNVFP4
Registers the attention.mla.indexer.ragged.float8.paged.remap.nvfp4 graph op.
attention.mla.indexer.ragged.float8.paged.remap for an NVFP4 MLA cache, whose
scales are paged apart from its values. Each row of physical_indices is
2 * k wide: the selected positions' physical rows in the values, then
their rows in the scale pool, resolved through the cache's scales lookup
table. The sparse NVFP4 MLA kernels read the second half instead of
deriving a scale row from a value row.
Implemented traits
Methods
execute
static def execute[*, num_heads: Int, depth: Int, k: Int, quantization_granularity: Int, mask_str: StaticString, write_logical: Bool = False](output_indices: T, physical_indices: T, q: T, qs: T, input_row_offsets: T, k_blocks: T, page_stride: T, k_cache_lengths: T, k_lookup_table: T, k_max_prompt_length: T, k_max_cache_length: T, k_scales: T, scales_page_stride: T, k_scales_lookup_table: T, layer_idx: UInt32, mla_kv_blocks: T, mla_lookup_table: T, mla_kv_scales: T, mla_scales_page_stride: T, mla_scales_lookup_table: T, mixed_step_split: T, ctx: DeviceContext)
Computes top-k key positions and their value and scale rows.
Parameters:
- num_heads (
Int): Number of query attention heads. - depth (
Int): Head dimension. - k (
Int): Number of top indices to return per token. - quantization_granularity (
Int): Quantization granularity for the K cache. - mask_str (
StaticString): Mask type - either MaskName.NULL or MaskName.CAUSAL. - write_logical (
Bool): Whether to writeoutput_indices(defaults toFalse). WithFalseonlyphysical_indicesis written, andoutput_indicesmay have no rows.
Args:
- output_indices (
T): Logical top-k key positions[total_seq_len, k],-1past a token's candidates. - physical_indices (
T):[total_seq_len, 2 * k]. Columns[0, k)are eachoutput_indicesslot's physical MLA cache row,block_id * page_size + offset; columns[k, 2 * k)are its row in the MLA scale pool,scale_block_id * rows_per_block + offset.-1where the position is-1or its page is unmapped. - q (
T): Query tensor [total_seq_len, num_heads, depth] in FP8. - qs (
T): Query scales [total_seq_len, num_heads] in float32. - input_row_offsets (
T): Ragged row offsets [batch_size + 1] for queries. - k_blocks (
T): Paged indexer K cache blocks in FP8. - page_stride (
T): Page-to-page distance in elements; -1 when packed. - k_cache_lengths (
T): Cache lengths [batch_size]. - k_lookup_table (
T): Indexer cache page lookup table. - k_max_prompt_length (
T): Max prompt (query) length scalar tensor [1]. - k_max_cache_length (
T): Max cache length scalar tensor [1]. - k_scales (
T): K scale blocks matching k_blocks shape with scale values. - scales_page_stride (
T): The same, for the scales leaf. - k_scales_lookup_table (
T): Page lookup table for the scale blocks. - layer_idx (
UInt32): Layer index for retrieving the correct cache layer. - mla_kv_blocks (
T): The MLA cache's blocks; their page count is the invalid block id and their page size the remap's. - mla_lookup_table (
T): The MLA cache's page lookup table[batch_size, pages_per_seq]. - mla_kv_scales (
T): The MLA cache's scale blocks; their page count is the invalid scale block id. - mla_scales_page_stride (
T): Page-to-page distance ofmla_kv_scalesin elements; -1 when packed. - mla_scales_lookup_table (
T): The MLA cache's scales page lookup table[batch_size, pages_per_seq]. - mixed_step_split (
T): The KV cache's split record; zeros keep the whole-batch route. - ctx (
DeviceContext): Device context for GPU execution.