IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

hip_mha_decoding_num_partitions

def hip_mha_decoding_num_partitions(batch_size: Int, num_keys: Int, heads_per_group: Int, sm_count: Int, is_mla: Bool = False, block_m: Int = Int(32), num_m_tiles: Int = Int(1), q_seq_len: Int = Int(1)) -> Int

Wave-aligned split-K target for MI355X MHA + MLA decode.

Two regimes, distinguished by whether the kernel packs queries by BM (MLA) or spawns one CTA per kv-head (MHA):

  • MHA-style (heads_per_group < BM): the call comes from get_mha_decoding_num_partitions passing heads_per_group = num_heads // group = kv_num_heads, which is typically small (≤ 8). Each (kv_head, batch) is its own CTA in grid_y, so actual_ctas_per_partition = heads_per_group × batch_size. When work_items = heads_per_group × batch_size ≥ sm_count, one partition already fills the GPU; use few partitions, just enough to amortize key reads. Derived from the original heuristic's HIGH_OCC branch, with two changes: its cap narrows to 32 at exactly 16 kv heads from batch 8 up, and the 16-kv-head key-stream rule below pre-empts the branch entirely once a full unsplit wave and 8 pages are reached.

  • MLA-style (heads_per_group ≥ BM): the call comes from mla.mojo passing heads_per_group = num_heads (≥ BM=32 for h ∈ {32, 64, 128}). MLA packs BM queries into one CTA, so actual_ctas_per_partition = ceildiv(num_heads, BM) × batch_size. Even when work_items looks large (e.g. bs=8 h=64 → 512), actual CTAs are only 16; needs many partitions. Apply the 2-wave wave-aligned formula: one_wave = sm_count // ctas_per_partition two_wave = 2 × sm_count // ctas_per_partition work_floor = ceildiv(pages, MAX_PAGES_PER_SPLIT) np_target = clamp(work_floor, one_wave, two_wave)

    EXCEPTION (MLA num_heads <= 16, e.g. Kimi-K2.5 TP=4, and single-kv-head MHA): the one_wave floor underfills. With num_blocks_y=1, ctas_per_partition = batch_size, so one wave (np = sm/bs) gives each CU exactly one CTA: no second block to overlap HBM-read latency. These shapes are latency-bound, so target two full waves instead: np_target = min(two_wave, pages) Measured on MI355: two-wave np is 5-10% faster than one-wave across bs=4 (32K-128K) and bs=8/16 short context; past two waves regresses on reduce cost. MLA bs=1 is unchanged (two_wave=512 clamps to the 256 cap); single-kv-head MHA does move, since its finer split floor doubles pages.

    Multi-tile fold (num_m_tiles > 1, the AMD MLA M > 64 two-tile fold): each (batch, partition) already spans num_m_tiles CTAs, and the split-K workspace round trip (per-partition partials + stats, then the reduce read) scales with np, so past one full wave the extra partitions cost more reduce work than the latency they hide. Target ONE wave of the true CTA count instead, rounded UP to the next power-of-two rung (the mid-point buckets rag the per-partition key stream against the 128-key BN tiles): np_target = min(next_power_of_two(one_wave), pages) Measured on MI355 (Kimi K3 verify, H=12, fp8 KV, decode+reduce): one wave beats two by 12% at bs=4/32K (np=32 56.9us vs np=64 64.7us) and the single-tile two-wave default (np=128, 84.1us) by 32%; bs=2/32K np=64 vs 128: -24%; bs=3/32K np=64 vs 128: -7%; bs=1/120K np=128 vs 256: -15.5%; np=256 at bs=4/32K regresses +36%.

    Single-tile fold at heads_per_group <= 8 with M = heads * S > 16 (DSpark drafter, H=8 S=7): the fold packs the whole query into one M tile, so ctas_per_partition = batch_size and one wave fills the device on the batch axis alone. Same rule and same reason as the two-tile fold -- a second wave doubles np, and the split-K workspace round trip then costs more than the stalls it hides. Measured on MI355 (drafter, S=7, b4, fp8 KV, nk 87040, decode+reduce): np=64 84.1us against the two-wave np=128 99.3us, and half a wave (np=32) 136.6us. Held to <= 8 heads so the H=12 S=8 verify shape keeps its two-tile rule, and to M > 16 because M <= 16 runs the legacy BM=16 launch, which is latency-bound per CTA like plain H=16 S=1 decode.

    This target deliberately supersedes work_floor (the 1280-key per-partition cap the heads >= 32 branch clamps to): among the measured shapes that cap binds only at 120K context x batch >= 2, and there the curve still falls with np — bs=4/120K: np=32 169.0us < np=64 176.1us < np=128 206.9us < np=192 210.5us (6 reps ABBA) — so the cap's np=96 would sit ~10% above the measured optimum. Once each partition's K-loop runs long, the split-K reduce round trip outweighs the shorter key streams the cap would buy.

Phase 0 sweep (PARTITIONING_PLAN.md) validated MLA-style at h=64: bs=1 ctx=131K → np=128 (capped, fills GPU at 1-wave + cap) bs=2 ctx=65K → np=64 (one_wave=64 dominates) bs=2 ctx=80K → np=64 (one_wave=64 dominates; work_floor 64 capped) bs=2 ctx=131K → np=128 (work_floor=103 → bucket to 128) bs=8 ctx=80K → np=32 (work_floor 64 capped by two_wave=32) bs=8 ctx=131K → np=32 (work_floor 103 capped by two_wave=32) bs=16 ctx=131K → np=16 (work_floor 103 capped by two_wave=16)

AMD reducer constraint: mla_splitk_reduce supports MAX_PARTITIONS up to parts_per_lane × WARP_SIZE; the 256-partition specialization (parts_per_lane=4) lifts the MLA-style cap to 256. Only nk >= 64K (pages >= 256) actually reaches np=256; smaller nk is page-limited.

Tunables (MLA-style): block_m = 32 (default MLA decode block-M) SPLIT_PAGE_SIZE = 256 (min keys per partition; 128 for single-kv-head MHA) MAX_PAGES_PER_SPLIT = 5 (= 1280 keys per partition cap) MAX_HIP_PARTITIONS = 256 (reducer's MAX_PARTITIONS limit; the MHA-style branch above stays pinned ≤64)

Args:

  • ​batch_size (Int): Number of decode requests in the batch.
  • ​num_keys (Int): Number of key cache entries to scan.
  • ​heads_per_group (Int): Not a group size: the kv-head count for MHA (num_heads // group), num_heads for MLA.
  • ​sm_count (Int): Device multiprocessor count used to size the wave-fill target.
  • ​is_mla (Bool): Whether the caller is the MLA decode path (defaults to False).
  • ​block_m (Int): Number of query rows packed into each MLA workgroup. Read for MLA only; MHA callers keep the default.
  • ​num_m_tiles (Int): M-tiles (grid.y blocks) each (batch, partition) spans, from amd_mla_decode_fold_num_m_tiles (2 for the AMD MLA decode two-tile S>1 fold, more for a padded short-chunk fold); 1 for every other caller. Multiplies ctas_per_partition, which otherwise counts only the ceildiv(heads_per_group, block_m) packing.
  • ​q_seq_len (Int): Query rows (S) folded into the MMA M dimension, i.e. the width the launch folds (a short prompt chunk's bucket ceiling, see amd_mla_decode_fold_seq_len_bucket), not its raw S; 1 for plain decode and for every MHA caller. Selects the one-wave target for the single-tile MLA fold above 16 folded rows.

Returns:

Int

Was this page helpful?