IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

TopKTopPMaskedProbsKernel

def TopKTopPMaskedProbsKernel[block_size: Int, vec_size: Int, dtype: DType, LogitsLayoutType: TensorLayout, logits_origin: ImmOrigin, ProbsLayoutType: TensorLayout, coop_size: Int = Int(1), *, LogitsEngine: TensorEngine, ProbsEngine: TensorEngine](logits: TileTensor[dtype, LogitsLayoutType, logits_origin, Engine=LogitsEngine], probs: TileTensor[.float32, ProbsLayoutType, MutAnyOrigin, Engine=ProbsEngine], top_k_arr: Optional[Pointer[Int64, ImmutAnyOrigin]], top_k_val: Int32, top_p_arr: Optional[Pointer[Float32, ImmutAnyOrigin]], top_p_val: Float32, temperature: Optional[Pointer[Float32, ImmutAnyOrigin]], d: Int32, coop_ws: Optional[Pointer[Int32, MutAnyOrigin]], hist_ws: Optional[Pointer[UInt64, MutAnyOrigin]])

Writes each row's top-k/top-p masked softmax.

A row whose total mass is not finite (a NaN logit the row max ignores, so only that weight goes NaN) has no distribution to report and comes out all-zero -- the same row shape the emit_dist sampler zeroes for the draft side.

A row with k == 1 gets a one-hot at its first maximum (lowest index, NaN never wins), whatever its temperature, top-p or mass, rather than splitting the mass across tied maxima; that is the token ops.argmax and a k == 1 draw pick. So a k == 1 row without a finite mass still names a token instead of coming out all-zero: column 0 of an all -inf row, the column of a +inf, the finite maximum beside a lone NaN.

With coop_size above one, coop_size blocks share a row: each owns a contiguous slice and combines whole-row statistics through coop_ws. The launch must keep every block in a group resident.

Parameters:

  • ​block_size (Int): Number of threads per block.
  • ​vec_size (Int): Number of elements each thread loads per access.
  • ​dtype (DType): Element type of logits.
  • ​LogitsLayoutType (TensorLayout): Memory layout of the logits tile.
  • ​logits_origin (ImmOrigin): Origin tag for the immutable logits tile.
  • ​ProbsLayoutType (TensorLayout): Memory layout of the probs tile.
  • ​coop_size (Int): Blocks sharing each row; 1 keeps the whole row in one block and compiles the cross-block traffic away.
  • ​LogitsEngine (TensorEngine): Storage engine backing the logits tile.
  • ​ProbsEngine (TensorEngine): Storage engine backing the probs tile.

Args:

Was this page helpful?