For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
topk_cluster
Cluster-cooperative top-k for the sparse-attention indexer.
Port of vLLM's cooperative_topk structure (csrc/cooperative_topk.cuh): one
query row per thread-block cluster of CS CTAs, 1024 threads each, combined
through distributed shared memory. A decode batch whose rows under-fill the
GPU gets CS times the CTAs on the same rows, which is the whole point -- at
24 rows a B200 leaves 124 of 148 SMs idle while each row's scan runs serially.
Where this deviates from vLLM it is because MAX's contract is different, not because the port is incomplete:
- No global tie workspace and no truncated tie list. Like vLLM, round 0
bins a coarse monotone key (
_ctopk_key0, 16 bins per octave as vLLM's fp16-rounded bin) and only the threshold bin's candidates are refined, on rank 0; but the candidates gather into rank 0's shared memory over DSMEM, and a bin too dense for that list falls back to refining over the whole slice rather than dropping candidates. Either way the score half of the key resolves fully, so the threshold is an exact score and the plateau at it may be cut first-come -- theordered=False, deterministic=Falsecontract the GLM indexer ships. - Scores are read at
in_dtype, f32 or bf16, and widened exactly on load (_widen_scores' contract); bf16 halves the score-buffer traffic and the staging. - The streaming flavor can stage its slices with the 1D
cp.async.bulkcopy through a two-slot mbarrier pipeline (vLLM'stma_stream_pass), on SM100+ where that copy compiles and behind an off-by-default switch: the plain-load streaming measured faster on B200, so the pipeline is opt-in (use_tma, pinned by tests) rather than the route. A row stride the copy cannot serve (not a whole 16 B unit per row) keeps the plain loads, so the correct-at-any-alignment contract holds everywhere. - Rows carry a live-column bound (
RowBound) instead of a lengths tensor, and every output slot goes through aTopkSink.
As in vLLM, a row short enough for one CTA's registers (_ctopk_solo_cap)
is selected by rank 0 alone, vLLM's histogram_4096_topk, while the other
ranks exit before any cluster synchronization.
One kernel body, comptime on CS and the score dtype. The launch is
grid = (rows, CS), block = 1024, cluster = (1, CS, 1); block_idx.x is
the row and block_rank_in_cluster() the rank. A row stride rank 0 covers
puts every row on the mid-length path, so it launches grid = (rows, 1)
without a cluster, each CTA its own rank 0, and only the mid-length path's
shared memory.
Functions
-
topk_cluster_split: Launch the cluster-cooperative top-k, one row percluster_sizeCTAs.