IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

amd_mla_decode_dispatch_scalars

def amd_mla_decode_dispatch_scalars(batch_size: Int, max_cache_valid_length: Int, q_max_seq_len: Int, num_heads: Int, is_fp8_kv: Bool, ctx: DeviceContext, sparse_indices_stride: Int = Int(0)) -> Tuple[Int, Int, Int]

Returns the AMD MLA decode dispatch metadata.

Returns (batch_size, q_max_seq_len, num_partitions). The AMD MLA decode kernel bakes its split-K grid from mha_decoding_num_partitions(..., is_mla=True) at dispatch time and never reads num_partitions from the scalar buffer, so the value serves only as the device-graph capture key (the decode graph op forwards it as the launch's num_partitions override, which the kernel's own ladder would answer identically). To keep that key equal to the baked grid, dense MLA passes the raw max_cache_valid_length and num_heads (kv_heads == 1 for MLA), with no q_max_seq_len length adjustment: the query width enters only through the fold's M-tile count (amd_mla_decode_fold_num_m_tiles) and the single-tile S>1 one-wave target, both keyed on the width the launch folds -- a short prompt chunk's bucket ceiling (amd_mla_decode_fold_seq_len_bucket), not its raw S.

No AMD sparse op (sparse_indices_stride > 0) reads the value either. The FP8-weight sparse ops size their split with amd_sparse_mla_inplace_decode_num_partitions of their own token count, and the BF16-weight arm with amd_sparse_mla_bf16_num_partitions, so the key depends on batch_size and q_max_seq_len alone. It matches the BF16 arm on a BF16 cache and, at one query token with an FP8 cache, the FP8-weight ops.

Args:

  • ​batch_size (Int): Current decode batch size.
  • ​max_cache_valid_length (Int): Max valid KV-cache length across the batch.
  • ​q_max_seq_len (Int): Max query sequence length (1 for decode).
  • ​num_heads (Int): Number of Q attention heads.
  • ​is_fp8_kv (Bool): Whether the KV cache is FP8.
  • ​ctx (DeviceContext): Device context used to read the CU count.
  • ​sparse_indices_stride (Int): Selected rows per token for sparse MLA, the sparse index row stride; 0 for dense MLA.

Returns:

Tuple[Int, Int, Int]: The packed (batch_size, q_max_seq_len, num_partitions) metadata.

Was this page helpful?