IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

mla_decode_max_seq_len

def mla_decode_max_seq_len[dtype: DType, num_heads: Int, sparse_mla: Bool]() -> Int

Max query tokens (S) the MLA decode branch can fold for this config.

On AMD the S>1 fold is FP8-only AND num_heads<=AMD_MLA_DECODE_FOLD_MAX_NUM_HEADS only (the larger-head dispatch arm is S=1-only), so a BF16 cache or num_heads>16 routes S>1 to MLA prefill; NVIDIA folds up to MLA_DECODE_MAX_SEQ_LEN for both. Mirrors the host-side fold gate in flare_mla_decoding_dispatch so the router never hands the gate an S>1 batch it would reject.

Sparse MLA on AMD is a different composition -- one gather slot per row of q, so the batch geometry is only correct at S=1 -- and it has no fold at any head count. Its S>1 work goes to the AMD sparse prefill arm, which gathers per query token and so serves any S.

Parameters:

  • โ€‹dtype (DType): Element type of the query tensor. On AMD the S>1 fold is FP8-only, so non-FP8 routes S>1 to MLA prefill.
  • โ€‹num_heads (Int): Number of query attention heads. On AMD the S>1 fold requires num_heads <= AMD_MLA_DECODE_FOLD_MAX_NUM_HEADS.
  • โ€‹sparse_mla (Bool): Whether top-k sparse attention is in use. On AMD this forces S>1 to prefill regardless of dtype and head count.

Returns:

Int

Was this page helpful?