For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
mha_decode_partition_heuristic
Split-K partition-count heuristics for MHA and MLA decode kernels.
Exposes per-backend functions that decide how many partitions the decode split-K loop should run across, plus num_keys-independent upper bounds used to launch graph-stable decode grids whose extra partitions early-return in the kernel. CUDA uses a simple SM-fill target clamped to the single-warp reducer limit; HIP (MI355X) uses a wave-aligned formula tuned for both MHA and MLA decode shapes.
comptime values
AMD_MLA_BF16_DECODE_BLOCK_M
comptime AMD_MLA_BF16_DECODE_BLOCK_M = 16
AMD_MLA_DECODE_FOLD_MAX_NUM_HEADS
comptime AMD_MLA_DECODE_FOLD_MAX_NUM_HEADS = 16
AMD_MLA_DECODE_FOLD_MAX_SEQ_LEN
comptime AMD_MLA_DECODE_FOLD_MAX_SEQ_LEN = 128
AMD_MLA_DECODE_MTP_SEQ_LEN
comptime AMD_MLA_DECODE_MTP_SEQ_LEN = 8
MAX_HIP_PARTITIONS
comptime MAX_HIP_PARTITIONS = 256
MLA_DECODE_KEY_TILE_ROWS
comptime MLA_DECODE_KEY_TILE_ROWS = 128
Functions
-
amd_mla_decode_dispatch_scalars: Returns the AMD MLA decode dispatch metadata. -
amd_mla_decode_fold_m_tile_rows: Returns the rows per grid.y M-tile the AMD MLA decode token-fold launch uses. -
amd_mla_decode_fold_num_m_tiles: Number of grid.y M-tiles the AMD MLA decode token-fold launch spans. -
amd_mla_decode_fold_seq_len_bucket: Returns the query width a short prompt chunk's launch pads its rows up to. -
amd_sparse_mla_bf16_num_partitions: Returns the split-K partition count for the AMD BF16 sparse MLA arm. -
amd_sparse_mla_decode_num_partitions: Returns the split-K partition count for AMD sparse MLA decode. -
amd_sparse_mla_inplace_decode_num_partitions: Returns the split-K partition count for AMD sparse MLA decode that reads the selected rows in place (the FP8 cache arm). -
cuda_mha_decoding_max_num_partitions: Returns the num_keys-independent upper bound on CUDA decode split-K partitions. -
cuda_mha_decoding_num_partitions: Returns the CUDA decode split-K partition count for the given shape. -
hip_mha_decoding_num_partitions: Wave-aligned split-K target for MI355X MHA + MLA decode. -
mha_decoding_max_num_partitions: Returns the num_keys-independent upper bound onmha_decoding_num_partitions. -
mha_decoding_num_partitions: Dispatches to the backend-specific decode split-K partition heuristic.