IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

cp_merge_topk

Merges a decode context-parallel group's local indexer top-k selections.

Under decode context parallelism (DCP) each rank of a group selects the top-k of its own key slots (mla_indexer_ragged_float8_paged_cp), and the group all-gathers the selections' scores, rank-major: rank r's k scores are columns [r * k, (r + 1) * k) of a [T, dcp * k] matrix. The group's top-k is the top-k of those dcp * k candidates, exactly up to ties, because a token in the group's top-k is in its owner's local top-k. Every rank computes the same global choice from the same matrix, then keeps the rows of its own candidates that made it.

Per row, a two-pass radix select over the bf16 scores' order-preserving 16-bit keys finds the k-th largest score t and how many candidates scoring exactly t the top-k takes; those ties go to the lowest columns, so the choice is deterministic and agreed on by every rank without communication.

Over an NVFP4 latent cache each selection carries a second row, its row in the cache's scale pool, which the indexer writes top_k columns after the value row (PagedKVScaleRemap). With halves = 2 a row of local_rows and out_rows is 2 * top_k wide, and the merge moves each kept selection's scale row with its value row, so both halves come out compacted the same way.

Functions​

Was this page helpful?