For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
persistent_topk_block_split
def persistent_topk_block_split[in_dtype: DType, //, ordered: Bool = False, deterministic: Bool = False, sig_bits: Int = _hsel_sig_bits[in_dtype](), phi_dtype: DType = _hsel_phi_dtype[in_dtype](), scan_items: Int = _hsel_prefetch_scan_items[in_dtype, phi_dtype](), RB: RowBound = RowBoundsPtr](ctx: DeviceContext, in_scores: ImmPointer[Scalar[in_dtype], ImmutAnyOrigin], out_idxs: MutPointer[Int32, MutAnyOrigin], N: Int, K: Int, total_seq_len: Int, row_bounds: Optional[RB] = None)
Launch bitonic top-k, splitting the N dimension when rows under-fill GPU.
Same contract and output as persistent_topk_block. When the row count is
small relative to the SM count and N spans many tiles (the long-context
decode regime — a handful of blocks would otherwise each fold the whole row
serially), the streaming fold is split across rows * S blocks (phase 1),
each producing a sorted top-_TILE partial, then merged per row (phase 2).
All other shapes fall back to persistent_topk_block unchanged.
On SM90+ targets (SM12x excluded), up to as many rows as clusters of two
keep resident in one wave (_ctopk_max_rows) instead take the
cluster-cooperative kernel
(nn/attention/gpu/nvidia/topk_cluster.mojo), which spreads each row over
a thread-block cluster — and only under the
ordered=False, deterministic=False contract, which is all that kernel
promises. Under ordered=False, deterministic=True, bounded bf16 rows
past the widest resident row, few enough that each gets a cluster of at
least two CTAs, take nn/attention/gpu/nvidia/topk_cluster_det.mojo
instead, which writes the same output as the ranked select it replaces,
slot for slot. Every other contract and shape keeps the kernels this
docstring otherwise describes.
On AMD gfx950, bounded rows of the same contract that under-fill the GPU and are
wider than the resident select take the row-split select
(_histsel_split_topk): two launches, each row cut into slices selected on
their own blocks, then merged. It returns the same set as the single-block
select, ties included, in a different (deterministic) slot order.
The score-dtype and pointer-alignment contract is persistent_topk_block's,
unchanged.
Parameters:
- in_dtype (
DType): Element type of the score buffer, f32 or bf16 (inferred). - ordered (
Bool): Whether slotqmust hold theq-th largest score.Trueis the strong contract: descending score, ties by ascending column.Falsepromises theKcolumns in an unspecified order, and leaves whether that order and set repeat from run to run todeterministic. Only shapes that have a cheaper path take one; the rest stay ordered, which satisfies the weaker promise too. - deterministic (
Bool): Whether one input must always give one output: the sameKcolumns in the same slots on every run (defaults toFalse). WithTruethe set is the ordered contract's, since the tie-break lives in the key the select already compares.Falsedrops the guarantee: slots are filled in the order threads arrive, and when more columns tie at the threshold than there are slots left, which of them survive is decided by the order the block's warps reach a cursor. In exchange the tail needs no ranking pass and few rows may take the cluster-cooperative kernel. Only meaningful together withordered=False; the ordered path is deterministic by construction. - sig_bits (
Int): How many of a key half's register a score can occupy, which is what sets the radix select's round count. Defaults to_hsel_sig_bits[in_dtype]()-- 16 at bf16 and 32 at f32 -- so a caller never needs to pass it. Forcing 32 at bf16 asks for the f32 round schedule on a bf16 buffer, which is a slower way to the same answer and exists so the two schedules can be compared inside one process. It reaches the schedule only, never the choice of kernel, and is irrelevant belowPERSISTENT_TOPK_MAX_N, where no radix select runs. - phi_dtype (
DType): The width the in-registerphipayload is carried at, which is what sets how many columns a scan group or a resident row costs. Defaults to_hsel_phi_dtype[in_dtype](), which isuint32at every score dtype, so a caller never needs to pass it. Forcinguint16at bf16 gives the narrow payload; it must be at least as wide as the score and at leastsig_bitswide. Likesig_bitsit reaches the kernels only, never the dispatch, so both arms run the same instantiation family on the same grid. - scan_items (
Int): Columns a thread carries per scan step on the arms that prefetch. Defaults to_hsel_prefetch_scan_items[in_dtype, phi_dtype]()-- 16 where the payload is narrow enough to hold them, 8 otherwise, which at the defaultphi_dtypeis always 8. Must be a multiple of_elems_per_16b[in_dtype](), or a thread's group base lands mid-vector for some threads and not others. The arms that do not prefetch always take the default width; they hold no group across the loop, so a wider one would cost registers for nothing. - RB (
RowBound): Type ofrow_bounds; seeRowBound.
Args:
- ctx (
DeviceContext): Device context. - in_scores (
ImmPointer[Scalar[in_dtype], ImmutAnyOrigin]): Flat score buffer[total_seq_len × N]row-major. - out_idxs (
MutPointer[Int32, MutAnyOrigin]): Output buffer[total_seq_len × K]row-major (int32). - N (
Int): Score columns per token (the row stride). - K (
Int): Top-k count per token, at most PERSISTENT_TOPK_MAX_N. It may exceedN; slots past a row's live columns are-1. - total_seq_len (
Int): Number of rows. - row_bounds (
Optional[RB]): Optional per-row live-column counts; seepersistent_topk_block. Rowrscans only[0, row_bounds(r))of itsN-wide stripe, so under capture-frozen metadata (whereNis a worst-case bound) the scan cost tracks each row's real length. Supported on every path this launcher selects forN > 2048(the histogram-select family) and on the 2048 single-block path.