For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
kpool_append_ragged_paged
def kpool_append_ragged_paged[dtype: DType, cache_t: KVCacheT, //, head_dim: Int, kpool: Int](tail: TileTensor[dtype, Engine=tail.Engine, linear_idx_type=tail.linear_idx_type], k: TileTensor[dtype, Engine=k.Engine, linear_idx_type=k.linear_idx_type], gate: TileTensor[dtype, Engine=gate.Engine, linear_idx_type=gate.linear_idx_type], ape: TileTensor[.float32, Engine=ape.Engine, linear_idx_type=ape.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], cache_lengths: TileTensor[.uint32, Engine=cache_lengths.Engine, linear_idx_type=cache_lengths.linear_idx_type], slot_idx: TileTensor[.uint32, Engine=slot_idx.Engine, linear_idx_type=slot_idx.linear_idx_type], k_cache: cache_t, scale_ub: Float32, ctx: DeviceContext)
Pools, quantizes and caches every pool this call completes, in one grid.
Equivalent, byte for byte, to kpool_ring_close_kernel and
kpool_compress_kernel followed by a per-pool dynamic FP8 quantization
(quantize_dynamic_scaled_fp8, one group per pooled key) and the paged
cache stores of the values and their scale. Pool P of a request lands at
cache positions [P * kpool, (P + 1) * kpool), head_dim // kpool values
per position, with the pool's scale replicated into each position.
Block r < batch closes request r's pending tail-ring pool, if this call
finishes it; every later block builds one pool entirely from this call's
tokens. The tail ring is only read: kpool_seed_tail_kernel must run
after this grid.
Parameters:
- dtype (
DType): Element type oftail,kandgate(inferred). - cache_t (
KVCacheT): The pooled key cache of one layer (inferred), FP8 with one scale per position. - head_dim (
Int): Channels per key; also the block width. - kpool (
Int): Tokens per pool.
Args:
- tail (
TileTensor[dtype, Engine=tail.Engine, linear_idx_type=tail.linear_idx_type]): Per-slot ring,[max_slots, 2, kpool, head_dim]. - k (
TileTensor[dtype, Engine=k.Engine, linear_idx_type=k.linear_idx_type]): Layer-normed indexer keys,[total_tokens, head_dim]. - gate (
TileTensor[dtype, Engine=gate.Engine, linear_idx_type=gate.linear_idx_type]): Per-token gate scores,[total_tokens, head_dim]. - ape (
TileTensor[.float32, Engine=ape.Engine, linear_idx_type=ape.linear_idx_type]): Within-pool position embedding,[kpool, head_dim], f32. - input_row_offsets (
TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type]): Token row offsets per request,[batch + 1]. - cache_lengths (
TileTensor[.uint32, Engine=cache_lengths.Engine, linear_idx_type=cache_lengths.linear_idx_type]): Cached-prefix length per request,[batch]. - slot_idx (
TileTensor[.uint32, Engine=slot_idx.Engine, linear_idx_type=slot_idx.linear_idx_type]): Ring slot per request,[batch]. - k_cache (
cache_t): The layer's pooled key cache. - scale_ub (
Float32): Upper bound on a pool's max-abs before the scale derives. - ctx (
DeviceContext): Device context to launch on.