IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

persistent_topk_block_split

def persistent_topk_block_split[in_dtype: DType, //, ordered: Bool = False, deterministic: Bool = False, sig_bits: Int = _hsel_sig_bits[in_dtype](), phi_dtype: DType = _hsel_phi_dtype[in_dtype](), scan_items: Int = _hsel_prefetch_scan_items[in_dtype, phi_dtype](), RB: RowBound = RowBoundsPtr](ctx: DeviceContext, in_scores: ImmPointer[Scalar[in_dtype], ImmutAnyOrigin], out_idxs: MutPointer[Int32, MutAnyOrigin], N: Int, K: Int, total_seq_len: Int, row_bounds: Optional[RB] = None)

Launch bitonic top-k, splitting the N dimension when rows under-fill GPU.

Same contract and output as persistent_topk_block. When the row count is small relative to the SM count and N spans many tiles (the long-context decode regime — a handful of blocks would otherwise each fold the whole row serially), the streaming fold is split across rows * S blocks (phase 1), each producing a sorted top-_TILE partial, then merged per row (phase 2). All other shapes fall back to persistent_topk_block unchanged.

On SM90+ targets (SM12x excluded), up to as many rows as clusters of two keep resident in one wave (_ctopk_max_rows) instead take the cluster-cooperative kernel (nn/attention/gpu/nvidia/topk_cluster.mojo), which spreads each row over a thread-block cluster — and only under the ordered=False, deterministic=False contract, which is all that kernel promises. Under ordered=False, deterministic=True, bounded bf16 rows past the widest resident row, few enough that each gets a cluster of at least two CTAs, take nn/attention/gpu/nvidia/topk_cluster_det.mojo instead, which writes the same output as the ranked select it replaces, slot for slot. Every other contract and shape keeps the kernels this docstring otherwise describes.

On AMD gfx950, bounded rows of the same contract that under-fill the GPU and are wider than the resident select take the row-split select (_histsel_split_topk): two launches, each row cut into slices selected on their own blocks, then merged. It returns the same set as the single-block select, ties included, in a different (deterministic) slot order.

The score-dtype and pointer-alignment contract is persistent_topk_block's, unchanged.

Parameters:

  • ​in_dtype (DType): Element type of the score buffer, f32 or bf16 (inferred).
  • ​ordered (Bool): Whether slot q must hold the q-th largest score. True is the strong contract: descending score, ties by ascending column. False promises the K columns in an unspecified order, and leaves whether that order and set repeat from run to run to deterministic. Only shapes that have a cheaper path take one; the rest stay ordered, which satisfies the weaker promise too.
  • ​deterministic (Bool): Whether one input must always give one output: the same K columns in the same slots on every run (defaults to False). With True the set is the ordered contract's, since the tie-break lives in the key the select already compares. False drops the guarantee: slots are filled in the order threads arrive, and when more columns tie at the threshold than there are slots left, which of them survive is decided by the order the block's warps reach a cursor. In exchange the tail needs no ranking pass and few rows may take the cluster-cooperative kernel. Only meaningful together with ordered=False; the ordered path is deterministic by construction.
  • ​sig_bits (Int): How many of a key half's register a score can occupy, which is what sets the radix select's round count. Defaults to _hsel_sig_bits[in_dtype]() -- 16 at bf16 and 32 at f32 -- so a caller never needs to pass it. Forcing 32 at bf16 asks for the f32 round schedule on a bf16 buffer, which is a slower way to the same answer and exists so the two schedules can be compared inside one process. It reaches the schedule only, never the choice of kernel, and is irrelevant below PERSISTENT_TOPK_MAX_N, where no radix select runs.
  • ​phi_dtype (DType): The width the in-register phi payload is carried at, which is what sets how many columns a scan group or a resident row costs. Defaults to _hsel_phi_dtype[in_dtype](), which is uint32 at every score dtype, so a caller never needs to pass it. Forcing uint16 at bf16 gives the narrow payload; it must be at least as wide as the score and at least sig_bits wide. Like sig_bits it reaches the kernels only, never the dispatch, so both arms run the same instantiation family on the same grid.
  • ​scan_items (Int): Columns a thread carries per scan step on the arms that prefetch. Defaults to _hsel_prefetch_scan_items[in_dtype, phi_dtype]() -- 16 where the payload is narrow enough to hold them, 8 otherwise, which at the default phi_dtype is always 8. Must be a multiple of _elems_per_16b[in_dtype](), or a thread's group base lands mid-vector for some threads and not others. The arms that do not prefetch always take the default width; they hold no group across the loop, so a wider one would cost registers for nothing.
  • ​RB (RowBound): Type of row_bounds; see RowBound.

Args:

  • ​ctx (DeviceContext): Device context.
  • ​in_scores (ImmPointer[Scalar[in_dtype], ImmutAnyOrigin]): Flat score buffer [total_seq_len × N] row-major.
  • ​out_idxs (MutPointer[Int32, MutAnyOrigin]): Output buffer [total_seq_len × K] row-major (int32).
  • ​N (Int): Score columns per token (the row stride).
  • ​K (Int): Top-k count per token, at most PERSISTENT_TOPK_MAX_N. It may exceed N; slots past a row's live columns are -1.
  • ​total_seq_len (Int): Number of rows.
  • ​row_bounds (Optional[RB]): Optional per-row live-column counts; see persistent_topk_block. Row r scans only [0, row_bounds(r)) of its N-wide stripe, so under capture-frozen metadata (where N is a worst-case bound) the scan cost tracks each row's real length. Supported on every path this launcher selects for N > 2048 (the histogram-select family) and on the 2048 single-block path.

Was this page helpful?