For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
TopKTopPMaskedProbsKernel
def TopKTopPMaskedProbsKernel[block_size: Int, vec_size: Int, dtype: DType, LogitsLayoutType: TensorLayout, logits_origin: ImmOrigin, ProbsLayoutType: TensorLayout, coop_size: Int = Int(1), *, LogitsEngine: TensorEngine, ProbsEngine: TensorEngine](logits: TileTensor[dtype, LogitsLayoutType, logits_origin, Engine=LogitsEngine], probs: TileTensor[.float32, ProbsLayoutType, MutAnyOrigin, Engine=ProbsEngine], top_k_arr: Optional[Pointer[Int64, ImmutAnyOrigin]], top_k_val: Int32, top_p_arr: Optional[Pointer[Float32, ImmutAnyOrigin]], top_p_val: Float32, temperature: Optional[Pointer[Float32, ImmutAnyOrigin]], d: Int32, coop_ws: Optional[Pointer[Int32, MutAnyOrigin]], hist_ws: Optional[Pointer[UInt64, MutAnyOrigin]])
Writes each row's top-k/top-p masked softmax.
A row whose total mass is not finite (a NaN logit the row max ignores, so
only that weight goes NaN) has no distribution to report and comes out
all-zero -- the same row shape the emit_dist sampler zeroes for the
draft side.
A row with k == 1 gets a one-hot at its first maximum (lowest index, NaN
never wins), whatever its temperature, top-p or mass, rather than
splitting the mass across tied maxima; that is the token ops.argmax
and a k == 1 draw pick. So a k == 1 row without a finite mass still names
a token instead of coming out all-zero: column 0 of an all -inf row,
the column of a +inf, the finite maximum beside a lone NaN.
With coop_size above one, coop_size blocks share a row: each owns a
contiguous slice and combines whole-row statistics through coop_ws. The
launch must keep every block in a group resident.
Parameters:
- block_size (
Int): Number of threads per block. - vec_size (
Int): Number of elements each thread loads per access. - dtype (
DType): Element type oflogits. - LogitsLayoutType (
TensorLayout): Memory layout of thelogitstile. - logits_origin (
ImmOrigin): Origin tag for the immutablelogitstile. - ProbsLayoutType (
TensorLayout): Memory layout of theprobstile. - coop_size (
Int): Blocks sharing each row; 1 keeps the whole row in one block and compiles the cross-block traffic away. - LogitsEngine (
TensorEngine): Storage engine backing thelogitstile. - ProbsEngine (
TensorEngine): Storage engine backing theprobstile.
Args:
- logits (
TileTensor[dtype, LogitsLayoutType, logits_origin, Engine=LogitsEngine]): Input logits [batch_size, d]. - probs (
TileTensor[.float32, ProbsLayoutType, MutAnyOrigin, Engine=ProbsEngine]): Output masked renormalized distribution [batch_size, d]. - top_k_arr (
Optional[Pointer[Int64, ImmutAnyOrigin]]): Optional per-row top-k [batch_size]. - top_k_val (
Int32): Default top-k iftop_k_arris null. - top_p_arr (
Optional[Pointer[Float32, ImmutAnyOrigin]]): Optional per-row top-p [batch_size]. - top_p_val (
Float32): Default top-p iftop_p_arris null. - temperature (
Optional[Pointer[Float32, ImmutAnyOrigin]]): Optional per-row temperature [batch_size]. - d (
Int32): Vocabulary size. - coop_ws (
Optional[Pointer[Int32, MutAnyOrigin]]): Zeroed cross-block workspace; required whencoop_sizeexceeds one, ignored otherwise. - hist_ws (
Optional[Pointer[UInt64, MutAnyOrigin]]): Cross-block cutoff histogram workspace ofcutoff_hist_words(batch_size, coop_size)words; required whencoop_sizeexceeds one, ignored otherwise.