IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

kpool_append_ragged_paged

def kpool_append_ragged_paged[dtype: DType, cache_t: KVCacheT, //, head_dim: Int, kpool: Int](tail: TileTensor[dtype, Engine=tail.Engine, linear_idx_type=tail.linear_idx_type], k: TileTensor[dtype, Engine=k.Engine, linear_idx_type=k.linear_idx_type], gate: TileTensor[dtype, Engine=gate.Engine, linear_idx_type=gate.linear_idx_type], ape: TileTensor[.float32, Engine=ape.Engine, linear_idx_type=ape.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], cache_lengths: TileTensor[.uint32, Engine=cache_lengths.Engine, linear_idx_type=cache_lengths.linear_idx_type], slot_idx: TileTensor[.uint32, Engine=slot_idx.Engine, linear_idx_type=slot_idx.linear_idx_type], k_cache: cache_t, scale_ub: Float32, ctx: DeviceContext)

Pools, quantizes and caches every pool this call completes, in one grid.

Equivalent, byte for byte, to kpool_ring_close_kernel and kpool_compress_kernel followed by a per-pool dynamic FP8 quantization (quantize_dynamic_scaled_fp8, one group per pooled key) and the paged cache stores of the values and their scale. Pool P of a request lands at cache positions [P * kpool, (P + 1) * kpool), head_dim // kpool values per position, with the pool's scale replicated into each position.

Block r < batch closes request r's pending tail-ring pool, if this call finishes it; every later block builds one pool entirely from this call's tokens. The tail ring is only read: kpool_seed_tail_kernel must run after this grid.

Parameters:

  • ​dtype (DType): Element type of tail, k and gate (inferred).
  • ​cache_t (KVCacheT): The pooled key cache of one layer (inferred), FP8 with one scale per position.
  • ​head_dim (Int): Channels per key; also the block width.
  • ​kpool (Int): Tokens per pool.

Args:

Was this page helpful?