For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
min_p_sampling_gpu
def min_p_sampling_gpu[dtype: DType, out_idx_type: DType, //, _test_sort: Bool = False, use_max_threshold: Bool = False](ctx: DeviceContext, min_ps: TileTensor[dtype, Engine=min_ps.Engine, linear_idx_type=min_ps.linear_idx_type], input_logits: TileTensor[dtype, Engine=input_logits.Engine, linear_idx_type=input_logits.linear_idx_type], out_token_ids: TileTensor[out_idx_type, Engine=out_token_ids.Engine, linear_idx_type=out_token_ids.linear_idx_type], temperature: Scalar[dtype] = 1)
GPU implementation of Min-P sampling for token selection. This function applies temperature scaling and softmax, and then samples tokens based on the calculated probability threshold (Min-P).
Parameters:
- dtype (
DType): DType - The data type of the input logits, min_ps, and temperature. - out_idx_type (
DType): DType - The data type for output token indices. - _test_sort (
Bool): Bool - For internal testing purposes to check if the sorted probs are in descending order. Requires the sort-based path. - use_max_threshold (
Bool): Bool - Whether to sample without ordering: reduce the row maximum, form the threshold from it, mask, renormalize, and draw. False (default) keeps the path that radix sorts the probabilities first. Both draw from the same set of tokens.
Args:
- ctx (
DeviceContext): DeviceContext The context for GPU execution. - min_ps (
TileTensor[dtype, Engine=min_ps.Engine, linear_idx_type=min_ps.linear_idx_type]): Per-row min-p coefficients in [0, 1]; the threshold ismin_p * max_probability. - input_logits (
TileTensor[dtype, Engine=input_logits.Engine, linear_idx_type=input_logits.linear_idx_type]): Input logits tensor of shape [batch_size, vocab_size]. - out_token_ids (
TileTensor[out_idx_type, Engine=out_token_ids.Engine, linear_idx_type=out_token_ids.linear_idx_type]): Output buffer for sampled token indices of shape [batch_size, 1]. - temperature (
Scalar[dtype]): Temperature for softmax scaling of logits (default=1.0).