IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

mla_nvfp4_kv_to_fp8

Sparse MLA prefill over an NVFP4 cache through a once-converted FP8 copy.

Target family: Blackwell sm_100 (the conversion kernels are plain SIMT).

mla_prefill_sparse_qkv_fp8 on an NVFP4 cache makes every CTA (one query token) expand its own top_k K rows into an FP8 tile in shared memory, so a row selected by n queries is expanded n times. At 8 heads per GPU that costs ~36% over the FP8-cache kernel. When the batch's context is small next to its query count, it is cheaper to expand every context row once, into an FP8 scratch, and run the unchanged FP8 kernel on that.

A row is 352 B = 256 B packed e2m1 latent + 64 B FP8 rope + (in the separate scale pool) 32 B E4M3 scales, one per 16 latent elements. The FP8 row is 576 B. The expansion is the helper the NVFP4 kernel uses (nvfp4_dequant_group), so the result is bit-identical to that kernel's.

The scratch holds whole pages: batch entry b's page p goes to scratch page b * pages_per_seq + p. Converting whole pages keeps a block that two entries share (a cached prefix) complete under either entry's copy. A table from physical block to scratch page then rewrites the indexer's value rows (block * page_size + offset) into scratch rows; the scale half of the index row is not read, since the conversion takes each row's scales through the cache's own scale page table.

Structs​

Functions​

Was this page helpful?