For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
mla_decode_max_seq_len
def mla_decode_max_seq_len[dtype: DType, num_heads: Int, sparse_mla: Bool, mask_places_query_span: Bool = False]() -> Int
Max query tokens (S) the MLA decode branch can fold for this config.
On AMD the S>1 fold is FP8-only AND num_heads<=AMD_MLA_DECODE_FOLD_MAX_NUM_HEADS
only (the larger-head dispatch arm is S=1-only), so a BF16 cache or num_heads>16
routes S>1 to MLA prefill; NVIDIA folds up to MLA_DECODE_MAX_SEQ_LEN for both.
Mirrors the host-side fold gate in AMDMLA.decode_dispatch so the router
never hands the gate an S>1 batch it would reject.
A short prompt chunk (up to MLA_DECODE_FOLD_MAX_SEQ_LEN query tokens) also
runs the absorbed decode when this head count has a head-aligned M-tile for
it, because MLA prefill decompresses the whole cached context per chunk. A
mask that places the query span (see mla_mask_places_query_span) cannot
take that padded width and stays on the exact-width MTP fold.
Sparse MLA on AMD is a different composition -- one gather slot per row of
q, so the batch geometry is only correct at S=1 -- and it has no fold at
any head count. Its S>1 work goes to the AMD sparse prefill arm, which
gathers per query token and so serves any S.
Parameters:
- βdtype (
DType): Element type of the query tensor. On AMD the S>1 fold is FP8-only, so non-FP8 routes S>1 to MLA prefill. - βnum_heads (
Int): Number of query attention heads. On AMD the S>1 fold requiresnum_heads <= AMD_MLA_DECODE_FOLD_MAX_NUM_HEADS. - βsparse_mla (
Bool): Whether top-k sparse attention is in use. On AMD this forces S>1 to prefill regardless of dtype and head count. - βmask_places_query_span (
Bool): Whether the batch's mask places the query span from the comptime query width, which excludes the padded short-chunk buckets (defaults toFalse).
Returns: