For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
mla_nvfp4_kv_to_fp8
Sparse MLA prefill over an NVFP4 cache through a once-converted FP8 copy.
Target family: Blackwell sm_100 (the conversion kernels are plain SIMT).
mla_prefill_sparse_qkv_fp8 on an NVFP4 cache makes every CTA (one query
token) expand its own top_k K rows into an FP8 tile in shared memory, so a
row selected by n queries is expanded n times. At 8 heads per GPU that
costs ~36% over the FP8-cache kernel. When the batch's context is small next
to its query count, it is cheaper to expand every context row once, into an FP8
scratch, and run the unchanged FP8 kernel on that.
A row is 352 B = 256 B packed e2m1 latent + 64 B FP8 rope + (in the separate
scale pool) 32 B E4M3 scales, one per 16 latent elements. The FP8 row is
576 B. The expansion is the helper the NVFP4 kernel uses
(nvfp4_dequant_group), so the result is bit-identical to that kernel's.
The scratch holds whole pages: batch entry b's page p goes to scratch page
b * pages_per_seq + p. Converting whole pages keeps a block that two entries
share (a cached prefix) complete under either entry's copy. A table from
physical block to scratch page then rewrites the indexer's value rows
(block * page_size + offset) into scratch rows; the scale half of the index
row is not read, since the conversion takes each row's scales through the
cache's own scale page table.
Structs
-
NVFP4PrefillWorkspace: Owns reusable conversion storage allocated outside graph recording. -
NVFP4PrefillWorkspaceView: Borrows conversion storage; its owner must outlive every graph replay.
Functions
-
mla_prefill_sparse_nvfp4_via_fp8: Runs sparse attention on an FP8 copy of the batch's context. -
nvfp4_prefill_converts_once: Whether converting the batch's context once beats the NVFP4 kernel. -
nvfp4_prefill_scratch_rows: Rows of the FP8 scratch a batch converts into. -
nvfp4_prefill_workspace: Allocates conversion storage before capturing a helper invocation.