IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

topk_cluster

Cluster-cooperative top-k for the sparse-attention indexer.

Port of vLLM's cooperative_topk structure (csrc/cooperative_topk.cuh): one query row per thread-block cluster of CS CTAs, 1024 threads each, combined through distributed shared memory. A decode batch whose rows under-fill the GPU gets CS times the CTAs on the same rows, which is the whole point -- at 24 rows a B200 leaves 124 of 148 SMs idle while each row's scan runs serially.

Where this deviates from vLLM it is because MAX's contract is different, not because the port is incomplete:

  • No global tie workspace and no truncated tie list. Like vLLM, round 0 bins a coarse monotone key (_ctopk_key0, 16 bins per octave as vLLM's fp16-rounded bin) and only the threshold bin's candidates are refined, on rank 0; but the candidates gather into rank 0's shared memory over DSMEM, and a bin too dense for that list falls back to refining over the whole slice rather than dropping candidates. Either way the score half of the key resolves fully, so the threshold is an exact score and the plateau at it may be cut first-come -- the ordered=False, deterministic=False contract the GLM indexer ships.
  • Scores are read at in_dtype, f32 or bf16, and widened exactly on load (_widen_scores' contract); bf16 halves the score-buffer traffic and the staging.
  • The streaming flavor can stage its slices with the 1D cp.async.bulk copy through a two-slot mbarrier pipeline (vLLM's tma_stream_pass), on SM100+ where that copy compiles and behind an off-by-default switch: the plain-load streaming measured faster on B200, so the pipeline is opt-in (use_tma, pinned by tests) rather than the route. A row stride the copy cannot serve (not a whole 16 B unit per row) keeps the plain loads, so the correct-at-any-alignment contract holds everywhere.
  • Rows carry a live-column bound (RowBound) instead of a lengths tensor, and every output slot goes through a TopkSink.

As in vLLM, a row short enough for one CTA's registers (_ctopk_solo_cap) is selected by rank 0 alone, vLLM's histogram_4096_topk, while the other ranks exit before any cluster synchronization.

One kernel body, comptime on CS and the score dtype. The launch is grid = (rows, CS), block = 1024, cluster = (1, CS, 1); block_idx.x is the row and block_rank_in_cluster() the rank. A row stride rank 0 covers puts every row on the mid-length path, so it launches grid = (rows, 1) without a cluster, each CTA its own rank 0, and only the mid-length path's shared memory.

Functions​

Was this page helpful?