For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
topk_cluster_split
def topk_cluster_split[in_dtype: DType, //, RB: RowBound = RowBoundsPtr](ctx: DeviceContext, in_scores: Pointer[Scalar[in_dtype], ImmutAnyOrigin], out_idxs: Pointer[Int32, MutAnyOrigin], N: Int, K: Int, total_seq_len: Int, row_bounds: Optional[RB], cluster_size: Int, use_tma: Bool = False)
Launch the cluster-cooperative top-k, one row per cluster_size CTAs.
The ordered=False, deterministic=False contract of
persistent_topk_block_split, and only that contract: the selection is
exact on the strictly-above columns, the threshold plateau is cut
first-come, and the arrangement is unspecified. The launcher
persistent_topk_block_split routes here under its cluster gate; this
entry point also exists so tests and benchmarks can pin cluster_size
and reach the kernel at shapes the dispatch would not choose.
The score-dtype and pointer-alignment contract is
persistent_topk_block's, unchanged: 16 B-aligned buffer, row stride a
multiple of _elems_per_16b[in_dtype]() (correct but slower otherwise --
an unpadded stride always takes the plain-load streaming kernel; a padded
one can take the cp.async.bulk streaming kernel on SM100+, opt-in via
use_tma).
Parameters:
- in_dtype (
DType): Element type of the score buffer, f32 or bf16 (inferred). - RB (
RowBound): Type ofrow_bounds; seeRowBound.
Args:
- ctx (
DeviceContext): Device context. - in_scores (
Pointer[Scalar[in_dtype], ImmutAnyOrigin]): Flat score buffer[total_seq_len x N]row-major. - out_idxs (
Pointer[Int32, MutAnyOrigin]): Output buffer[total_seq_len x K]row-major (int32). - N (
Int): Score columns per token (the row stride). - K (
Int): Top-k count per token, at mostPERSISTENT_TOPK_MAX_N. - total_seq_len (
Int): Number of rows; one cluster ofcluster_sizeCTAs each. - row_bounds (
Optional[RB]): Optional per-row live-column counts; seepersistent_topk_block. - cluster_size (
Int): CTAs per row: 2, 3, 4, 5, 6, 8, 10 or 16. A size past 8 is nonportable; the runtime allows it on every function it loads. - use_tma (
Bool): Whether the streaming flavor stages its slices withcp.async.bulk(SM100+, padded row stride only; clamped to the plain-load kernel otherwise). Defaults to off: the plain-load streaming measures faster on B200 (the plan doc's M3 table), so only tests pinning the flavor opt in.
Raises:
If K exceeds the selection width or cluster_size is not a
supported size. A K of at most 0 selects nothing and launches
nothing.