IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

mha_split_kv_decode_num_partitions

def mha_split_kv_decode_num_partitions[sm_count: Int](batch_size: Int, num_keys: Int, kv_heads_per_device: Int, sliding_window: Int = Int(0)) -> Int

Returns the split-K partition count for decode and speculative verify over a split KV cache (key wider than value), the count an engine puts in the dispatch metadata.

The split cache's 1Q route honors the metadata count exactly (mha_sm100_dispatch's split_kv arm), so this is the production policy for that route, and engines that key graph captures on the count port it. It is not _sk_ws_partitions or the WS-G ceiling: those size routes the split cache never takes.

One partition per SPLIT_KV_KEYS_PER_PARTITION visible keys, no more than fill one wave of SMs with one CTA per (request, KV head), snapped DOWN to a splitk_p_ladder rung (each rung has its own unrolled fa4_splitk_combine; any other count takes its runtime-bound fallback) and capped at the combine's P_MAX. A windowed group sees at most its window, so a window under 512 keys always runs one partition.

Parameters:

  • ​sm_count (Int): The device's SM count, the ladder's top rung.

Args:

  • ​batch_size (Int): Requests in the batch.
  • ​num_keys (Int): The batch's largest cache length, in keys.
  • ​kv_heads_per_device (Int): KV heads on one device.
  • ​sliding_window (Int): The group's attention window, or 0 for none.

Returns:

Int: A partition count in [1, min(sm_count, P_MAX)].

Was this page helpful?