IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

topk_cluster_det

Deterministic cluster-cooperative top-k over bf16 scores.

The ordered=False, deterministic=True contract of persistent_topk_block_split, for rows that under-fill the GPU. It writes the same output as the ranked streaming select, slot for slot: the K largest (score, column) keys, descending score and ascending column within a score, then -1 past a row's live columns. That select spends one SM on a row, and its counting rank, which is what makes its output reproducible, degrades on the bf16 score plateaus an indexer row is made of. This kernel spreads a row over a cluster of CS CTAs and orders the selection with a comparison sort, whose cost does not depend on the scores.

Per row, every rank owns a contiguous, 16 B-aligned slice of [0, count):

  1. Each rank histograms the top 12 bits of its scores' phi16 (the order-preserving bf16 bijection _phi_of defines). Every rank reads the peers' histograms over DSMEM, first as per-warp group totals and then the split group's bins, and finds the same split bin.
  2. A 4-bit round over the split bin's columns resolves the exact threshold key t, the K-th largest phi16. Each rank publishes its 16 fine bins and its count above the split bin, which is also what tells every rank how many columns each rank holds above and at t.
  3. Every column above t is selected, and of the columns at t the lowest-numbered ones that complete K, which the ranks below take first. A block scan places each selected column at its position g in column order. Nothing is placed by arrival, so the set and g are functions of the input alone.
  4. The key (phi16 << 16) | (0xFFFF - g) orders a selection exactly as the ranked select does, since ascending g is ascending column. Rank r receives the keys of positions [r L, (r + 1) L), L = 2048 / CS, and sorts them; then every rank reads the other sorted chunks and places each of its keys at its own index plus the count of larger keys in every other chunk, and writes it there.

A row with at most K live columns selects all of them, already in column order, so it starts at step 4.

Every count is a histogram sum or a scan, and every slot is written by exactly one thread at a position those counts determine, so one input always gives one output, whatever order the CTAs and warps run in.

Every DSMEM producer and consumer are separated by a full cluster_sync(), whose arrive releases and whose wait acquires at cluster scope: the semantics _ctopk_cluster_sync_cta_release in topk_cluster.mojo documents from the PTX ISA. compute-sanitizer's racecheck still reports "potential RAW" hazards on one rank's stores into a peer's pairs and the peer's reads after that barrier; it reports the same class on the shipped non-deterministic cluster kernel at 16 ranks (a 128k-key row), and nothing at one rank, so these are read as the tool not modelling cluster barriers.

Functionsโ€‹

Was this page helpful?