IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

mla_index_fp8

MLA FP8 index kernel for computing attention scores with paged KV cache.

comptime values​

MXFP4IndexKCache​

comptime MXFP4IndexKCache = PagedKVCache[.uint8, KVCacheStaticParams(Int(1), Int(64), True, Int(0), Int(0)), page_size, MutUntrackedOrigin, DefaultEngine, ImmutAnyOrigin, DefaultEngine, ImmutAnyOrigin, DefaultEngine, MutUntrackedOrigin, scale_dtype_=DType.uint32, quantization_granularity_=Int(64)]

A paged MXFP4 index-key cache in indexer_k_store's page layout.

A page holds page_size keys: their E2M1 values, then their 32-bit scale words. The values are the cache's blocks and the words its scales, both strided by the page pitch, so the FP8 indexer's paged operands and TMA tiles read it unchanged.

Structs​

Traits​

  • ​IndexerTopk: Selects each indexer score row's top-k key positions.

Functions​

Was this page helpful?