For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
topk_per_row_gpu
def topk_per_row_gpu[dtype: DType, //](ctx: DeviceContext, scores: ImmPointer[Scalar[dtype], ImmutAnyOrigin], k: ImmPointer[Int64, ImmutAnyOrigin], out_vals: MutPointer[Scalar[dtype], MutAnyOrigin], out_idxs: MutPointer[Int64, MutAnyOrigin], rows: Int, n: Int, max_k: Int)
Selects the first k[r] picks of each row's top-max_k.
Row r gets the k[r] largest scores in descending order as
(value, column) pairs, with equal scores in any order. Every other slot
of its max_k holds (-inf, -1).
Parameters:
- dtype (
DType): Score and value dtype, float32 or bfloat16.
Args:
- ctx (
DeviceContext): Device context. - scores (
ImmPointer[Scalar[dtype], ImmutAnyOrigin]): Flat[rows, n]scores, 16 byte aligned. - k (
ImmPointer[Int64, ImmutAnyOrigin]):[rows]picks per row, each in[0, max_k]. - out_vals (
MutPointer[Scalar[dtype], MutAnyOrigin]): Flat[rows, max_k]output values. - out_idxs (
MutPointer[Int64, MutAnyOrigin]): Flat[rows, max_k]output columns. - rows (
Int): Number of rows. - n (
Int): Scores per row. - max_k (
Int): Output slots per row, in[1, TOPK_PER_ROW_MAX_K].