IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

kpool_expand_topk_kernel

def kpool_expand_topk_kernel[OutLayoutType: TensorLayout, out_origin: MutOrigin, PoolLayoutType: TensorLayout, pool_origin: ImmOrigin, IROLayoutType: TensorLayout, iro_origin: ImmOrigin, CacheLenLayoutType: TensorLayout, OutEngine: TensorEngine, PoolEngine: TensorEngine, IROEngine: TensorEngine, CacheLenEngine: TensorEngine, kpool: Int, pool_topk: Int, always_select_tail: Bool](out_indices: TileTensor[.int32, OutLayoutType, out_origin, Engine=OutEngine], pool_ids: TileTensor[.int32, PoolLayoutType, pool_origin, Engine=PoolEngine], input_row_offsets: TileTensor[.uint32, IROLayoutType, iro_origin, Engine=IROEngine], cache_lengths: TileTensor[.uint32, CacheLenLayoutType, ImmutAnyOrigin, Engine=CacheLenEngine], total_seq_len: Int32)

Turns selected pool ids back into the token positions they cover.

The indexer selects pools; attention reads tokens. Each selected pool expands to the kpool consecutive positions it covers.

With always_select_tail the row also carries up to kpool - 1 of the query's most recent positions, the ones no complete pool covers yet. Their location comes from the query's visible count, so it tracks the pool currently being filled.

Each row is packed: the selected pools first, in selection order, then the tail, then -1 in every remaining column. Unselected slots never become clamped valid positions, which would point attention at a token the indexer did not choose. Packing keeps every valid entry ahead of the padding, so a row has at most visible leading valid entries; the sparse decode bounds its scan by the context length and would otherwise skip a tail placed in the last columns.

Launch with KPOOL_EXPAND_BLOCK_SIZE threads per block, one block per token row.

Parameters:

  • ​OutLayoutType (TensorLayout): Layout of out_indices.
  • ​out_origin (MutOrigin): Origin of out_indices.
  • ​PoolLayoutType (TensorLayout): Layout of pool_ids.
  • ​pool_origin (ImmOrigin): Origin of pool_ids.
  • ​IROLayoutType (TensorLayout): Layout of input_row_offsets.
  • ​iro_origin (ImmOrigin): Origin of input_row_offsets.
  • ​CacheLenLayoutType (TensorLayout): Layout of cache_lengths.
  • ​OutEngine (TensorEngine): Engine of out_indices.
  • ​PoolEngine (TensorEngine): Engine of pool_ids.
  • ​IROEngine (TensorEngine): Engine of input_row_offsets.
  • ​CacheLenEngine (TensorEngine): Engine of cache_lengths.
  • ​kpool (Int): Tokens per pool.
  • ​pool_topk (Int): Selected pools per token, index_topk // kpool.
  • ​always_select_tail (Bool): Whether to append the incomplete trailing pool.

Args:

Was this page helpful?