For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
mla_indexer_ragged_mxfp4_paged
def mla_indexer_ragged_mxfp4_paged[oi_layout: TensorLayout, q_layout: TensorLayout, qsf_layout: TensorLayout, w_layout: TensorLayout, iro_layout: TensorLayout, //, num_heads: Int, top_k: Int, mask_str: StaticString, page_size: Int, scores_dtype: DType = .bfloat16, kpool: Int = Int(1), Topk: IndexerTopk = BlockSplitTopk](output_indices: MutTileTensor[.int32, oi_layout], q: ImmTileTensor[.uint8, q_layout], q_sf: ImmTileTensor[.uint32, qsf_layout], weights: ImmTileTensor[.float32, w_layout], input_row_offsets: ImmTileTensor[.uint32, iro_layout], k_cache: MXFP4IndexKCache[page_size], ctx: DeviceContext, scores_budget_bytes: Int = Int((mul get_defined_int[StringSpan("MLA_INDEX_SCORES_BUDGET_MB"), Int(512)](), 1048576)))
Computes the MXFP4 indexer's top-k key positions, without a remap.
See the overload taking remap; this one writes the logical indices
only.
Parameters:
- oi_layout (
TensorLayout): Layout of the top-k index output. - q_layout (
TensorLayout): Layout of the query tensor. - qsf_layout (
TensorLayout): Layout of the query block scales. - w_layout (
TensorLayout): Layout of the per-head weights. - iro_layout (
TensorLayout): Layout of the ragged query row offsets. - num_heads (
Int): Index heads per token. - top_k (
Int): Key positions to select per token. - mask_str (
StaticString):MaskName.NULLorMaskName.CAUSAL. - page_size (
Int): Keys per cache page. - scores_dtype (
DType): Element type of the transient score matrix. - kpool (
Int): Tokens per cache row, the compression ratio. - Topk (
IndexerTopk): The top-k over the score rows.
Args:
- output_indices (
MutTileTensor[.int32, oi_layout]): Dense[total_seq_len, top_k]top-k output. - q (
ImmTileTensor[.uint8, q_layout]): Packed E2M1 queries,[total_seq_len, num_heads, 64]. - q_sf (
ImmTileTensor[.uint32, qsf_layout]): Query block scales,[total_seq_len, num_heads]. - weights (
ImmTileTensor[.float32, w_layout]): Per-head weights,[total_seq_len, num_heads]. - input_row_offsets (
ImmTileTensor[.uint32, iro_layout]): Ragged row offsets for queries[batch_size + 1]. - k_cache (
MXFP4IndexKCache[page_size]): The index keys, frommxfp4_index_k_cache. - ctx (
DeviceContext): Device context. - scores_budget_bytes (
Int): Peak bytes the transient score matrix may occupy.
def mla_indexer_ragged_mxfp4_paged[oi_layout: TensorLayout, q_layout: TensorLayout, qsf_layout: TensorLayout, w_layout: TensorLayout, iro_layout: TensorLayout, R: SparseIndexRemap, //, num_heads: Int, top_k: Int, mask_str: StaticString, page_size: Int, scores_dtype: DType = .bfloat16, kpool: Int = Int(1), write_logical: Bool = False, Topk: IndexerTopk = BlockSplitTopk](output_indices: MutTileTensor[.int32, oi_layout], q: ImmTileTensor[.uint8, q_layout], q_sf: ImmTileTensor[.uint32, qsf_layout], weights: ImmTileTensor[.float32, w_layout], input_row_offsets: ImmTileTensor[.uint32, iro_layout], k_cache: MXFP4IndexKCache[page_size], ctx: DeviceContext, remap: R, scores_budget_bytes: Int = Int((mul get_defined_int[StringSpan("MLA_INDEX_SCORES_BUDGET_MB"), Int(512)](), 1048576)))
Computes the MXFP4 indexer's top-k key positions over packed pages.
The lightning indexer with MXFP4 queries and keys:
score[t, s] = sum_h relu(q[t, h] . k[s]) * weights[t, h]where q[t, h] and k[s] are 128 E2M1 values, each run of 32 scaled by
its UE8M0 exponent, and weights already folds the softmax and head
scales. Scores land in a transient matrix and the bounded top-k selects
from it, as mla_indexer_ragged_float8_paged does.
Parameters:
- oi_layout (
TensorLayout): Layout of the top-k index output. - q_layout (
TensorLayout): Layout of the query tensor. - qsf_layout (
TensorLayout): Layout of the query block scales. - w_layout (
TensorLayout): Layout of the per-head weights. - iro_layout (
TensorLayout): Layout of the ragged query row offsets. - R (
SparseIndexRemap): Type ofremap. - num_heads (
Int): Index heads per token; 32. - top_k (
Int): Key positions to select per token. - mask_str (
StaticString):MaskName.NULLorMaskName.CAUSAL. - page_size (
Int): Keys per cache page; a multiple of 128 or a power-of-two fraction of it down to 8. - scores_dtype (
DType): Element type of the transient score matrix. - kpool (
Int): Tokens per cache row, the compression ratio. A key row coverskpoolconsecutive tokens, so indices count rows. - write_logical (
Bool): Whether to writeoutput_indices. - Topk (
IndexerTopk): The top-k over the score rows.
Args:
- output_indices (
MutTileTensor[.int32, oi_layout]): Dense[total_seq_len, top_k]top-k output;-1fills a row with fewer live keys thantop_k. - q (
ImmTileTensor[.uint8, q_layout]): Packed E2M1 queries,[total_seq_len, num_heads, 64], two channels a byte with the even one in the low nibble. - q_sf (
ImmTileTensor[.uint32, qsf_layout]): Query block scales,[total_seq_len, num_heads]: each word is four UE8M0 bytes, channels 32j..32j+31 in byte j. - weights (
ImmTileTensor[.float32, w_layout]): Per-head weights,[total_seq_len, num_heads]. - input_row_offsets (
ImmTileTensor[.uint32, iro_layout]): Ragged row offsets for queries[batch_size + 1]. - k_cache (
MXFP4IndexKCache[page_size]): The index keys, frommxfp4_index_k_cache. - ctx (
DeviceContext): Device context. - remap (
R): Second output beside each logical index; seemla_indexer_ragged_float8_paged. - scores_budget_bytes (
Int): Peak bytes the transient score matrix may occupy.
def mla_indexer_ragged_mxfp4_paged[oi_layout: TensorLayout, q_layout: TensorLayout, qsf_layout: TensorLayout, w_layout: TensorLayout, iro_layout: TensorLayout, R: SparseIndexRemap, H: IndexerScoreHook, //, num_heads: Int, top_k: Int, mask_str: StaticString, page_size: Int, scores_dtype: DType = .bfloat16, kpool: Int = Int(1), write_logical: Bool = False, Topk: IndexerTopk = BlockSplitTopk](output_indices: MutTileTensor[.int32, oi_layout], q: ImmTileTensor[.uint8, q_layout], q_sf: ImmTileTensor[.uint32, qsf_layout], weights: ImmTileTensor[.float32, w_layout], input_row_offsets: ImmTileTensor[.uint32, iro_layout], k_cache: MXFP4IndexKCache[page_size], ctx: DeviceContext, remap: R, score_hook: H, scores_budget_bytes: Int = Int((mul get_defined_int[StringSpan("MLA_INDEX_SCORES_BUDGET_MB"), Int(512)](), 1048576)))
Computes the MXFP4 indexer's top-k key positions with a score hook.
The hook sees each score in the scorer's epilogue as it is stored for the
top-k, so it can read the scores or drop keys from the selection. The op
scores at most score_hook.window_rows() rows per window and calls the
hook's after_window once each window is scored.
Parameters:
- oi_layout (
TensorLayout): Layout of the top-k index output. - q_layout (
TensorLayout): Layout of the query tensor. - qsf_layout (
TensorLayout): Layout of the query block scales. - w_layout (
TensorLayout): Layout of the per-head weights. - iro_layout (
TensorLayout): Layout of the ragged query row offsets. - R (
SparseIndexRemap): Type ofremap. - H (
IndexerScoreHook): Type ofscore_hook. - num_heads (
Int): Index heads per token; 32. - top_k (
Int): Key positions to select per token. - mask_str (
StaticString):MaskName.NULLorMaskName.CAUSAL. - page_size (
Int): Keys per cache page. - scores_dtype (
DType): Element type of the transient score matrix. - kpool (
Int): Tokens per cache row, the compression ratio. - write_logical (
Bool): Whether to writeoutput_indices. - Topk (
IndexerTopk): The top-k over the score rows.
Args:
- output_indices (
MutTileTensor[.int32, oi_layout]): Dense[total_seq_len, top_k]top-k output. - q (
ImmTileTensor[.uint8, q_layout]): Packed E2M1 queries,[total_seq_len, num_heads, 64]. - q_sf (
ImmTileTensor[.uint32, qsf_layout]): Query block scales,[total_seq_len, num_heads]. - weights (
ImmTileTensor[.float32, w_layout]): Per-head weights,[total_seq_len, num_heads]. - input_row_offsets (
ImmTileTensor[.uint32, iro_layout]): Ragged row offsets for queries[batch_size + 1]. - k_cache (
MXFP4IndexKCache[page_size]): The index keys, frommxfp4_index_k_cache. - ctx (
DeviceContext): Device context. - remap (
R): Second output beside each logical index. - score_hook (
H): Runs on every score before the top-k. - scores_budget_bytes (
Int): Peak bytes the transient score matrix may occupy.