For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
amd_mla_decode_dispatch_scalars
def amd_mla_decode_dispatch_scalars(batch_size: Int, max_cache_valid_length: Int, q_max_seq_len: Int, num_heads: Int, is_fp8_kv: Bool, ctx: DeviceContext, sparse_indices_stride: Int = Int(0)) -> Tuple[Int, Int, Int]
Returns the AMD MLA decode dispatch metadata.
Returns (batch_size, q_max_seq_len, num_partitions). The AMD MLA decode
kernel bakes its split-K grid from mha_decoding_num_partitions(..., is_mla=True) at dispatch time and never reads num_partitions from the
scalar buffer, so the value serves only as the device-graph capture key (the
decode graph op forwards it as the launch's
num_partitions override, which the kernel's own ladder would answer
identically). To keep that key equal to the baked grid, dense MLA passes the
raw max_cache_valid_length and num_heads (kv_heads == 1 for MLA), with
no q_max_seq_len length adjustment: the query width enters only through
the fold's M-tile count (amd_mla_decode_fold_num_m_tiles) and the
single-tile S>1 one-wave target, both keyed on the width the launch folds --
a short prompt chunk's bucket ceiling
(amd_mla_decode_fold_seq_len_bucket), not its raw S.
No AMD sparse op (sparse_indices_stride > 0) reads the value either.
The FP8-weight sparse ops size their split with
amd_sparse_mla_inplace_decode_num_partitions of their own token count,
and the BF16-weight arm with amd_sparse_mla_bf16_num_partitions, so the
key depends on batch_size and q_max_seq_len alone. It matches the
BF16 arm on a BF16 cache and, at one query token with an FP8 cache, the
FP8-weight ops.
Args:
- batch_size (
Int): Current decode batch size. - max_cache_valid_length (
Int): Max valid KV-cache length across the batch. - q_max_seq_len (
Int): Max query sequence length (1 for decode). - num_heads (
Int): Number of Q attention heads. - is_fp8_kv (
Bool): Whether the KV cache is FP8. - ctx (
DeviceContext): Device context used to read the CU count. - sparse_indices_stride (
Int): Selected rows per token for sparse MLA, the sparse index row stride; 0 for dense MLA.
Returns:
Tuple[Int, Int, Int]: The packed (batch_size, q_max_seq_len, num_partitions) metadata.