IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

mla_sparse_prefill_on_decode_kernel

def mla_sparse_prefill_on_decode_kernel[q_dtype: DType, cache_dtype: DType, num_heads: Int, fp8_q: Bool = True, fold_shared_index: Bool = False, has_extra_k: Bool = False, cache_is_nvfp4: Bool = False]() -> Bool

Returns whether sparse prefill of this shape runs the decode kernel.

FP8 and packed NVFP4 caches do at 8 to 64 heads (a multiple of 8) and at 128, unless the prefill folds a shared index or appends extra keys. A packed NVFP4 cache whose context fits nvfp4_prefill_converts_once converts once and runs the FP8 copy; otherwise the kernel gathers the packed value and scale rows directly. Graph dispatch and sparse-index remap preparation share this predicate. Bit-exact FP8/NVFP4 comparisons require matching key tiles, CTA grouping, and split configuration.

Parameters:

  • ​q_dtype (DType): Element type of the query tensor.
  • ​cache_dtype (DType): Element type of the latent KV cache.
  • ​num_heads (Int): Number of query attention heads.
  • ​fp8_q (Bool): Whether the decode branch stages Q in the FP8 cache dtype (defaults to True); see mla_decode_branch_fp8.
  • ​fold_shared_index (Bool): Whether the decode branch folds a shared top-k row, which prefill cannot take (defaults to False).
  • ​has_extra_k (Bool): Whether an extra-KV segment is supplied, which the masked-tile skip does not handle (defaults to False).
  • ​cache_is_nvfp4 (Bool): Whether cache_dtype is a packed NVFP4 cache (uint8 with E4M3 scales), which the dtype alone cannot say (defaults to False). Callers that hold the cache type pass _mla_nvfp4_cache; the remap helper, which only needs the answer for logical indices that NVFP4 ops never supply, keeps the default.

Returns:

Bool: True when the sparse decode kernel serves this shape's prefill.

Was this page helpful?