For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
kpool_expand_topk_kernel
def kpool_expand_topk_kernel[OutLayoutType: TensorLayout, out_origin: MutOrigin, PoolLayoutType: TensorLayout, pool_origin: ImmOrigin, IROLayoutType: TensorLayout, iro_origin: ImmOrigin, CacheLenLayoutType: TensorLayout, OutEngine: TensorEngine, PoolEngine: TensorEngine, IROEngine: TensorEngine, CacheLenEngine: TensorEngine, kpool: Int, pool_topk: Int, always_select_tail: Bool](out_indices: TileTensor[.int32, OutLayoutType, out_origin, Engine=OutEngine], pool_ids: TileTensor[.int32, PoolLayoutType, pool_origin, Engine=PoolEngine], input_row_offsets: TileTensor[.uint32, IROLayoutType, iro_origin, Engine=IROEngine], cache_lengths: TileTensor[.uint32, CacheLenLayoutType, ImmutAnyOrigin, Engine=CacheLenEngine], total_seq_len: Int32)
Turns selected pool ids back into the token positions they cover.
The indexer selects pools; attention reads tokens. Each selected pool
expands to the kpool consecutive positions it covers.
With always_select_tail the row also carries up to kpool - 1 of the
query's most recent positions, the ones no complete pool covers yet. Their
location comes from the query's visible count, so it tracks the pool
currently being filled.
Each row is packed: the selected pools first, in selection order, then the
tail, then -1 in every remaining column. Unselected slots never become
clamped valid positions, which would point attention at a token the
indexer did not choose. Packing keeps every valid entry ahead of the
padding, so a row has at most visible leading valid entries; the sparse
decode bounds its scan by the context length and would otherwise skip a
tail placed in the last columns.
Launch with KPOOL_EXPAND_BLOCK_SIZE threads per block, one block per
token row.
Parameters:
- OutLayoutType (
TensorLayout): Layout ofout_indices. - out_origin (
MutOrigin): Origin ofout_indices. - PoolLayoutType (
TensorLayout): Layout ofpool_ids. - pool_origin (
ImmOrigin): Origin ofpool_ids. - IROLayoutType (
TensorLayout): Layout ofinput_row_offsets. - iro_origin (
ImmOrigin): Origin ofinput_row_offsets. - CacheLenLayoutType (
TensorLayout): Layout ofcache_lengths. - OutEngine (
TensorEngine): Engine ofout_indices. - PoolEngine (
TensorEngine): Engine ofpool_ids. - IROEngine (
TensorEngine): Engine ofinput_row_offsets. - CacheLenEngine (
TensorEngine): Engine ofcache_lengths. - kpool (
Int): Tokens per pool. - pool_topk (
Int): Selected pools per token,index_topk // kpool. - always_select_tail (
Bool): Whether to append the incomplete trailing pool.
Args:
- out_indices (
TileTensor[.int32, OutLayoutType, out_origin, Engine=OutEngine]): Output[total_seq_len, pool_topk * kpool + tail], wheretailiskpool - 1whenalways_select_tailand 0 otherwise. - pool_ids (
TileTensor[.int32, PoolLayoutType, pool_origin, Engine=PoolEngine]): Selected pool ids,[total_seq_len, pool_topk],-1where fewer thanpool_topkpools were available. - input_row_offsets (
TileTensor[.uint32, IROLayoutType, iro_origin, Engine=IROEngine]): Token row offsets per request,[batch_size + 1]. - cache_lengths (
TileTensor[.uint32, CacheLenLayoutType, ImmutAnyOrigin, Engine=CacheLenEngine]): Cached-prefix length per request,[batch_size]. - total_seq_len (
Int32): Number of token rows.