For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
top_p_sampling
def top_p_sampling[dtype: DType, out_idx_type: DType, //, _test_sort: Bool = False](top_ps: TileTensor[dtype, Engine=top_ps.Engine, address_space=top_ps.address_space, linear_idx_type=top_ps.linear_idx_type], input_logits: TileTensor[dtype, Engine=input_logits.Engine, address_space=input_logits.address_space, linear_idx_type=input_logits.linear_idx_type], out_token_ids: TileTensor[out_idx_type, Engine=out_token_ids.Engine, address_space=out_token_ids.address_space, linear_idx_type=out_token_ids.linear_idx_type], temperature: Scalar[dtype] = 1)
Naive CPU implementation of Top-P sampling for token selection. This function applies temperature scaling, softmax, a merge sort, and then samples tokens based on the cumulative probability mass (Top-P).
Parameters:
- dtype (
DType): Element type ofinput_logits,top_ps, andtemperature(inferred). - out_idx_type (
DType): Element type of theout_token_idstensor (inferred). - _test_sort (
Bool): When true, copies the sorted probabilities back intoinput_logitsto verify descending order (defaults to false).
Args:
- top_ps (
TileTensor[dtype, Engine=top_ps.Engine, address_space=top_ps.address_space, linear_idx_type=top_ps.linear_idx_type]): Per-batch cumulative probability mass thresholds in the range (0, 1]. - input_logits (
TileTensor[dtype, Engine=input_logits.Engine, address_space=input_logits.address_space, linear_idx_type=input_logits.linear_idx_type]): Rank-2 logits tensor of shape (batch, vocab) read for temperature scaling and softmax. - out_token_ids (
TileTensor[out_idx_type, Engine=out_token_ids.Engine, address_space=out_token_ids.address_space, linear_idx_type=out_token_ids.linear_idx_type]): Rank-2 output tensor receiving the sampled token index at column 0 of each batch row. - temperature (
Scalar[dtype]): Positive scalar divisor applied to logits before the softmax (defaults to 1).