For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
mha_split_kv_decode_num_partitions
def mha_split_kv_decode_num_partitions[sm_count: Int](batch_size: Int, num_keys: Int, kv_heads_per_device: Int, sliding_window: Int = Int(0)) -> Int
Returns the split-K partition count for decode and speculative verify over a split KV cache (key wider than value), the count an engine puts in the dispatch metadata.
The split cache's 1Q route honors the metadata count exactly
(mha_sm100_dispatch's split_kv arm), so this is the production policy
for that route, and engines that key graph captures on the count port it.
It is not _sk_ws_partitions or the WS-G ceiling: those size routes the
split cache never takes.
One partition per SPLIT_KV_KEYS_PER_PARTITION visible keys, no more than
fill one wave of SMs with one CTA per (request, KV head), snapped DOWN to a
splitk_p_ladder rung (each rung has its own unrolled
fa4_splitk_combine; any other count takes its runtime-bound fallback)
and capped at the combine's P_MAX. A windowed group sees at most its
window, so a window under 512 keys always runs one partition.
Parameters:
- sm_count (
Int): The device's SM count, the ladder's top rung.
Args:
- batch_size (
Int): Requests in the batch. - num_keys (
Int): The batch's largest cache length, in keys. - kv_heads_per_device (
Int): KV heads on one device. - sliding_window (
Int): The group's attention window, or 0 for none.
Returns:
Int: A partition count in [1, min(sm_count, P_MAX)].