IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

topk_cluster_split

def topk_cluster_split[in_dtype: DType, //, RB: RowBound = RowBoundsPtr](ctx: DeviceContext, in_scores: Pointer[Scalar[in_dtype], ImmutAnyOrigin], out_idxs: Pointer[Int32, MutAnyOrigin], N: Int, K: Int, total_seq_len: Int, row_bounds: Optional[RB], cluster_size: Int, use_tma: Bool = False)

Launch the cluster-cooperative top-k, one row per cluster_size CTAs.

The ordered=False, deterministic=False contract of persistent_topk_block_split, and only that contract: the selection is exact on the strictly-above columns, the threshold plateau is cut first-come, and the arrangement is unspecified. The launcher persistent_topk_block_split routes here under its cluster gate; this entry point also exists so tests and benchmarks can pin cluster_size and reach the kernel at shapes the dispatch would not choose.

The score-dtype and pointer-alignment contract is persistent_topk_block's, unchanged: 16 B-aligned buffer, row stride a multiple of _elems_per_16b[in_dtype]() (correct but slower otherwise -- an unpadded stride always takes the plain-load streaming kernel; a padded one can take the cp.async.bulk streaming kernel on SM100+, opt-in via use_tma).

Parameters:

  • ​in_dtype (DType): Element type of the score buffer, f32 or bf16 (inferred).
  • ​RB (RowBound): Type of row_bounds; see RowBound.

Args:

  • ​ctx (DeviceContext): Device context.
  • ​in_scores (Pointer[Scalar[in_dtype], ImmutAnyOrigin]): Flat score buffer [total_seq_len x N] row-major.
  • ​out_idxs (Pointer[Int32, MutAnyOrigin]): Output buffer [total_seq_len x K] row-major (int32).
  • ​N (Int): Score columns per token (the row stride).
  • ​K (Int): Top-k count per token, at most PERSISTENT_TOPK_MAX_N.
  • ​total_seq_len (Int): Number of rows; one cluster of cluster_size CTAs each.
  • ​row_bounds (Optional[RB]): Optional per-row live-column counts; see persistent_topk_block.
  • ​cluster_size (Int): CTAs per row: 2, 3, 4, 5, 6, 8, 10 or 16. A size past 8 is nonportable; the runtime allows it on every function it loads.
  • ​use_tma (Bool): Whether the streaming flavor stages its slices with cp.async.bulk (SM100+, padded row stride only; clamped to the plain-load kernel otherwise). Defaults to off: the plain-load streaming measures faster on B200 (the plan doc's M3 table), so only tests pinning the flavor opt in.

Raises:

If K exceeds the selection width or cluster_size is not a supported size. A K of at most 0 selects nothing and launches nothing.

Was this page helpful?