IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

min_p_sampling_gpu

def min_p_sampling_gpu[dtype: DType, out_idx_type: DType, //, _test_sort: Bool = False, use_max_threshold: Bool = False](ctx: DeviceContext, min_ps: TileTensor[dtype, Engine=min_ps.Engine, linear_idx_type=min_ps.linear_idx_type], input_logits: TileTensor[dtype, Engine=input_logits.Engine, linear_idx_type=input_logits.linear_idx_type], out_token_ids: TileTensor[out_idx_type, Engine=out_token_ids.Engine, linear_idx_type=out_token_ids.linear_idx_type], temperature: Scalar[dtype] = 1)

GPU implementation of Min-P sampling for token selection. This function applies temperature scaling and softmax, and then samples tokens based on the calculated probability threshold (Min-P).

Parameters:

  • ​dtype (DType): DType - The data type of the input logits, min_ps, and temperature.
  • ​out_idx_type (DType): DType - The data type for output token indices.
  • ​_test_sort (Bool): Bool - For internal testing purposes to check if the sorted probs are in descending order. Requires the sort-based path.
  • ​use_max_threshold (Bool): Bool - Whether to sample without ordering: reduce the row maximum, form the threshold from it, mask, renormalize, and draw. False (default) keeps the path that radix sorts the probabilities first. Both draw from the same set of tokens.

Args:

Was this page helpful?