For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
topp_minp_sampling_kernel
def topp_minp_sampling_kernel[dtype: DType, out_idx_type: DType, is_top_p: Bool](p_thresholds_: MutPointer[Scalar[dtype], MutUntrackedOrigin], sorted_probs_: MutPointer[Scalar[dtype], MutUntrackedOrigin], sorted_ids_: MutPointer[Scalar[out_idx_type], MutUntrackedOrigin], out_token_ids: MutPointer[Scalar[out_idx_type], MutUntrackedOrigin], skip_sort: MutPointer[Scalar[DType.bool], MutUntrackedOrigin], vocab_size: Int32)
Top P-Min P sampling kernel.
Parameters:
- dtype (
DType): DType - scalar values dtype. - out_idx_type (
DType): DType - output index type. - is_top_p (
Bool): Bool - Whether to use Top-P (True) or Min-P (False) sampling.
Args:
- p_thresholds_ (
MutPointer[Scalar[dtype], MutUntrackedOrigin]): Top p or min-p calculated thresholds for each batch. - sorted_probs_ (
MutPointer[Scalar[dtype], MutUntrackedOrigin]): Sorted probabilities in descending order. - sorted_ids_ (
MutPointer[Scalar[out_idx_type], MutUntrackedOrigin]): Sorted token ids in descending order. - out_token_ids (
MutPointer[Scalar[out_idx_type], MutUntrackedOrigin]): Output token ids. - skip_sort (
MutPointer[Scalar[DType.bool], MutUntrackedOrigin]): Whether sorting was skipped for this batch. - vocab_size (
Int32): Number of tokens in the vocabulary per batch, used to iterate over the sorted probability and id buffers.