IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

mla_indexer_ragged_mxfp4_paged

def mla_indexer_ragged_mxfp4_paged[oi_layout: TensorLayout, q_layout: TensorLayout, qsf_layout: TensorLayout, w_layout: TensorLayout, iro_layout: TensorLayout, //, num_heads: Int, top_k: Int, mask_str: StaticString, page_size: Int, scores_dtype: DType = .bfloat16, kpool: Int = Int(1), Topk: IndexerTopk = BlockSplitTopk](output_indices: MutTileTensor[.int32, oi_layout], q: ImmTileTensor[.uint8, q_layout], q_sf: ImmTileTensor[.uint32, qsf_layout], weights: ImmTileTensor[.float32, w_layout], input_row_offsets: ImmTileTensor[.uint32, iro_layout], k_cache: MXFP4IndexKCache[page_size], ctx: DeviceContext, scores_budget_bytes: Int = Int((mul get_defined_int[StringSpan("MLA_INDEX_SCORES_BUDGET_MB"), Int(512)](), 1048576)))

Computes the MXFP4 indexer's top-k key positions, without a remap.

See the overload taking remap; this one writes the logical indices only.

Parameters:

  • ​oi_layout (TensorLayout): Layout of the top-k index output.
  • ​q_layout (TensorLayout): Layout of the query tensor.
  • ​qsf_layout (TensorLayout): Layout of the query block scales.
  • ​w_layout (TensorLayout): Layout of the per-head weights.
  • ​iro_layout (TensorLayout): Layout of the ragged query row offsets.
  • ​num_heads (Int): Index heads per token.
  • ​top_k (Int): Key positions to select per token.
  • ​mask_str (StaticString): MaskName.NULL or MaskName.CAUSAL.
  • ​page_size (Int): Keys per cache page.
  • ​scores_dtype (DType): Element type of the transient score matrix.
  • ​kpool (Int): Tokens per cache row, the compression ratio.
  • ​Topk (IndexerTopk): The top-k over the score rows.

Args:

def mla_indexer_ragged_mxfp4_paged[oi_layout: TensorLayout, q_layout: TensorLayout, qsf_layout: TensorLayout, w_layout: TensorLayout, iro_layout: TensorLayout, R: SparseIndexRemap, //, num_heads: Int, top_k: Int, mask_str: StaticString, page_size: Int, scores_dtype: DType = .bfloat16, kpool: Int = Int(1), write_logical: Bool = False, Topk: IndexerTopk = BlockSplitTopk](output_indices: MutTileTensor[.int32, oi_layout], q: ImmTileTensor[.uint8, q_layout], q_sf: ImmTileTensor[.uint32, qsf_layout], weights: ImmTileTensor[.float32, w_layout], input_row_offsets: ImmTileTensor[.uint32, iro_layout], k_cache: MXFP4IndexKCache[page_size], ctx: DeviceContext, remap: R, scores_budget_bytes: Int = Int((mul get_defined_int[StringSpan("MLA_INDEX_SCORES_BUDGET_MB"), Int(512)](), 1048576)))

Computes the MXFP4 indexer's top-k key positions over packed pages.

The lightning indexer with MXFP4 queries and keys:

score[t, s] = sum_h relu(q[t, h] . k[s]) * weights[t, h]

where q[t, h] and k[s] are 128 E2M1 values, each run of 32 scaled by its UE8M0 exponent, and weights already folds the softmax and head scales. Scores land in a transient matrix and the bounded top-k selects from it, as mla_indexer_ragged_float8_paged does.

Parameters:

  • ​oi_layout (TensorLayout): Layout of the top-k index output.
  • ​q_layout (TensorLayout): Layout of the query tensor.
  • ​qsf_layout (TensorLayout): Layout of the query block scales.
  • ​w_layout (TensorLayout): Layout of the per-head weights.
  • ​iro_layout (TensorLayout): Layout of the ragged query row offsets.
  • ​R (SparseIndexRemap): Type of remap.
  • ​num_heads (Int): Index heads per token; 32.
  • ​top_k (Int): Key positions to select per token.
  • ​mask_str (StaticString): MaskName.NULL or MaskName.CAUSAL.
  • ​page_size (Int): Keys per cache page; a multiple of 128 or a power-of-two fraction of it down to 8.
  • ​scores_dtype (DType): Element type of the transient score matrix.
  • ​kpool (Int): Tokens per cache row, the compression ratio. A key row covers kpool consecutive tokens, so indices count rows.
  • ​write_logical (Bool): Whether to write output_indices.
  • ​Topk (IndexerTopk): The top-k over the score rows.

Args:

def mla_indexer_ragged_mxfp4_paged[oi_layout: TensorLayout, q_layout: TensorLayout, qsf_layout: TensorLayout, w_layout: TensorLayout, iro_layout: TensorLayout, R: SparseIndexRemap, H: IndexerScoreHook, //, num_heads: Int, top_k: Int, mask_str: StaticString, page_size: Int, scores_dtype: DType = .bfloat16, kpool: Int = Int(1), write_logical: Bool = False, Topk: IndexerTopk = BlockSplitTopk](output_indices: MutTileTensor[.int32, oi_layout], q: ImmTileTensor[.uint8, q_layout], q_sf: ImmTileTensor[.uint32, qsf_layout], weights: ImmTileTensor[.float32, w_layout], input_row_offsets: ImmTileTensor[.uint32, iro_layout], k_cache: MXFP4IndexKCache[page_size], ctx: DeviceContext, remap: R, score_hook: H, scores_budget_bytes: Int = Int((mul get_defined_int[StringSpan("MLA_INDEX_SCORES_BUDGET_MB"), Int(512)](), 1048576)))

Computes the MXFP4 indexer's top-k key positions with a score hook.

The hook sees each score in the scorer's epilogue as it is stored for the top-k, so it can read the scores or drop keys from the selection. The op scores at most score_hook.window_rows() rows per window and calls the hook's after_window once each window is scored.

Parameters:

  • ​oi_layout (TensorLayout): Layout of the top-k index output.
  • ​q_layout (TensorLayout): Layout of the query tensor.
  • ​qsf_layout (TensorLayout): Layout of the query block scales.
  • ​w_layout (TensorLayout): Layout of the per-head weights.
  • ​iro_layout (TensorLayout): Layout of the ragged query row offsets.
  • ​R (SparseIndexRemap): Type of remap.
  • ​H (IndexerScoreHook): Type of score_hook.
  • ​num_heads (Int): Index heads per token; 32.
  • ​top_k (Int): Key positions to select per token.
  • ​mask_str (StaticString): MaskName.NULL or MaskName.CAUSAL.
  • ​page_size (Int): Keys per cache page.
  • ​scores_dtype (DType): Element type of the transient score matrix.
  • ​kpool (Int): Tokens per cache row, the compression ratio.
  • ​write_logical (Bool): Whether to write output_indices.
  • ​Topk (IndexerTopk): The top-k over the score rows.

Args:

Was this page helpful?