For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
mla_index_fp8
MLA FP8 index kernel for computing attention scores with paged KV cache.
comptime values
MXFP4IndexKCache
comptime MXFP4IndexKCache = PagedKVCache[.uint8, KVCacheStaticParams(Int(1), Int(64), True, Int(0), Int(0)), page_size, MutUntrackedOrigin, DefaultEngine, ImmutAnyOrigin, DefaultEngine, ImmutAnyOrigin, DefaultEngine, MutUntrackedOrigin, scale_dtype_=DType.uint32, quantization_granularity_=Int(64)]
A paged MXFP4 index-key cache in indexer_k_store's page layout.
A page holds page_size keys: their E2M1 values, then their 32-bit scale
words. The values are the cache's blocks and the words its scales, both
strided by the page pitch, so the FP8 indexer's paged operands and TMA tiles
read it unchanged.
Structs
-
BlockSplitTopk:persistent_topk_block_split, the indexers' default top-k. -
BlockTopKMaskScoreHook: Drops every key whose block ofblock_sizekeys a row does not keep. -
BlockTopKScoreHook: Writes the ids of each row'stop_kbest blocks ofblock_sizekeys.
Traits
-
IndexerTopk: Selects each indexer score row's top-k key positions.
Functions
-
mla_indexer_ragged_float8_paged: Computes the FP8 indexer's top-k key positions, without a remap. -
mla_indexer_ragged_float8_paged_cp: Selects one decode context-parallel rank's local top-k, with scores. -
mla_indexer_ragged_mxfp4_paged: Computes the MXFP4 indexer's top-k key positions, without a remap. -
mxfp4_index_k_cache: Views packed MXFP4 index-key pages as a paged cache.