For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
topk_cluster_det
Deterministic cluster-cooperative top-k over bf16 scores.
The ordered=False, deterministic=True contract of
persistent_topk_block_split, for rows that under-fill the GPU. It writes the
same output as the ranked streaming select, slot for slot: the K largest
(score, column) keys, descending score and ascending column within a score,
then -1 past a row's live columns. That select spends one SM on a row, and
its counting rank, which is what makes its output reproducible, degrades on
the bf16 score plateaus an indexer row is made of. This kernel spreads a row
over a cluster of CS CTAs and orders the selection with a comparison sort,
whose cost does not depend on the scores.
Per row, every rank owns a contiguous, 16 B-aligned slice of [0, count):
- Each rank histograms the top 12 bits of its scores'
phi16(the order-preserving bf16 bijection_phi_ofdefines). Every rank reads the peers' histograms over DSMEM, first as per-warp group totals and then the split group's bins, and finds the same split bin. - A 4-bit round over the split bin's columns resolves the exact threshold
key
t, theK-th largestphi16. Each rank publishes its 16 fine bins and its count above the split bin, which is also what tells every rank how many columns each rank holds above and att. - Every column above
tis selected, and of the columns attthe lowest-numbered ones that completeK, which the ranks below take first. A block scan places each selected column at its positiongin column order. Nothing is placed by arrival, so the set andgare functions of the input alone. - The key
(phi16 << 16) | (0xFFFF - g)orders a selection exactly as the ranked select does, since ascendinggis ascending column. Rankrreceives the keys of positions[r L, (r + 1) L),L = 2048 / CS, and sorts them; then every rank reads the other sorted chunks and places each of its keys at its own index plus the count of larger keys in every other chunk, and writes it there.
A row with at most K live columns selects all of them, already in column
order, so it starts at step 4.
Every count is a histogram sum or a scan, and every slot is written by exactly one thread at a position those counts determine, so one input always gives one output, whatever order the CTAs and warps run in.
Every DSMEM producer and consumer are separated by a full cluster_sync(),
whose arrive releases and whose wait acquires at cluster scope: the
semantics _ctopk_cluster_sync_cta_release in topk_cluster.mojo documents
from the PTX ISA. compute-sanitizer's racecheck still reports "potential RAW"
hazards on one rank's stores into a peer's pairs and the peer's reads after
that barrier; it reports the same class on the shipped non-deterministic
cluster kernel at 16 ranks (a 128k-key row), and nothing at one rank, so
these are read as the tool not modelling cluster barriers.
Functionsโ
- โ
topk_cluster_det: Launches the deterministic cluster top-k, one row percluster_sizeCTAs. - โ
topk_cluster_det_cluster_size: The widest cluster that keeps every row resident in one wave, or 1.