For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
MLAIndexerRaggedFloat8PagedRemap
struct MLAIndexerRaggedFloat8PagedRemap
Registers the mo.mla.indexer.ragged.float8.paged.remap graph op.
mo.mla.indexer.ragged.float8.paged plus a second output: each selected
position's physical row in the MLA cache, exactly what
paged_sparse_kv_index_remap would compute from the logical output. A
sparse MLA op given it skips that remap.
Implemented traits
Methods
execute
static def execute[*, num_heads: Int, depth: Int, k: Int, quantization_granularity: Int, mask_str: StringSpan[ImmStaticOrigin], write_logical: Bool = False](output_indices: ManagedTensorSlice[IOSpec[_, _].Output, static_spec=output_indices.static_spec], physical_indices: ManagedTensorSlice[IOSpec[_, _].Output, static_spec=physical_indices.static_spec], q: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=q.static_spec], qs: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=qs.static_spec], input_row_offsets: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=input_row_offsets.static_spec], k_blocks: ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=k_blocks.static_spec], page_stride: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=page_stride.static_spec], k_cache_lengths: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=k_cache_lengths.static_spec], k_lookup_table: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=k_lookup_table.static_spec], k_max_prompt_length: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=k_max_prompt_length.static_spec], k_max_cache_length: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=k_max_cache_length.static_spec], k_scales: ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=k_scales.static_spec], scales_page_stride: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=scales_page_stride.static_spec], k_scales_lookup_table: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=k_scales_lookup_table.static_spec], layer_idx: UInt32, mla_kv_blocks: ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=mla_kv_blocks.static_spec], mla_lookup_table: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=mla_lookup_table.static_spec], ctx: DeviceContextBase)
Computes top-k key positions and their physical MLA cache rows.
Parameters:
- num_heads (
Int): Number of query attention heads. - depth (
Int): Head dimension. - k (
Int): Number of top indices to return per token. - quantization_granularity (
Int): Quantization granularity for the K cache. - mask_str (
StringSpan[ImmStaticOrigin]): Mask type - either MaskName.NULL or MaskName.CAUSAL. - write_logical (
Bool): Whether to writeoutput_indices(defaults toFalse). WithFalseonly the physical rows are written, andoutput_indicesmay have no rows.
Args:
- output_indices (
ManagedTensorSlice[IOSpec[_, _].Output, static_spec=output_indices.static_spec]): Logical top-k key positions[total_seq_len, k],-1past a token's candidates. - physical_indices (
ManagedTensorSlice[IOSpec[_, _].Output, static_spec=physical_indices.static_spec]): Physical MLA cache row of eachoutput_indicesslot,block_id * page_size + offset,-1where the position is-1or its page is unmapped. - q (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=q.static_spec]): Query tensor [total_seq_len, num_heads, depth] in FP8. - qs (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=qs.static_spec]): Query scales [total_seq_len, num_heads] in float32. - input_row_offsets (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=input_row_offsets.static_spec]): Ragged row offsets [batch_size + 1] for queries. - k_blocks (
ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=k_blocks.static_spec]): Paged indexer K cache blocks in FP8. - page_stride (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=page_stride.static_spec]): Page-to-page distance in elements; -1 when packed. - k_cache_lengths (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=k_cache_lengths.static_spec]): Cache lengths [batch_size]. - k_lookup_table (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=k_lookup_table.static_spec]): Indexer cache page lookup table. - k_max_prompt_length (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=k_max_prompt_length.static_spec]): Max prompt (query) length scalar tensor [1]. - k_max_cache_length (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=k_max_cache_length.static_spec]): Max cache length scalar tensor [1]. - k_scales (
ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=k_scales.static_spec]): K scale blocks matching k_blocks shape with scale values. - scales_page_stride (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=scales_page_stride.static_spec]): The same, for the scales leaf. - k_scales_lookup_table (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=k_scales_lookup_table.static_spec]): Page lookup table for the scale blocks. - layer_idx (
UInt32): Layer index for retrieving the correct cache layer. - mla_kv_blocks (
ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=mla_kv_blocks.static_spec]): The MLA cache's blocks; their page count is the invalid block id and their page size the remap's. - mla_lookup_table (
ManagedTensorSlice[IOSpec[_, _].Input, static_spec=mla_lookup_table.static_spec]): The MLA cache's page lookup table[batch_size, pages_per_seq]. - ctx (
DeviceContextBase): Device context for GPU execution.