For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
compress_windows_ragged_paged
def compress_windows_ragged_paged[dtype: DType, cache_t: KVCacheT, //, rotate: Bool](output: TileTensor[dtype, Engine=output.Engine, linear_idx_type=output.linear_idx_type], kv: TileTensor[.float32, Engine=kv.Engine, linear_idx_type=kv.linear_idx_type], score: TileTensor[.float32, Engine=score.Engine, linear_idx_type=score.linear_idx_type], ape: TileTensor[.float32, Engine=ape.Engine, linear_idx_type=ape.linear_idx_type], norm_weight: TileTensor[dtype, Engine=norm_weight.Engine, linear_idx_type=norm_weight.linear_idx_type], freqs: TileTensor[.float32, Engine=freqs.Engine, linear_idx_type=freqs.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], window_offsets: TileTensor[.uint32, Engine=window_offsets.Engine, linear_idx_type=window_offsets.linear_idx_type], starts: TileTensor[.int32, Engine=starts.Engine, linear_idx_type=starts.linear_idx_type], kv_cache: cache_t, score_cache: cache_t, epsilon: Float32, ctx: DeviceContext)
Pools every window of a ragged chunk into its compressed entry.
Parameters:
- dtype (
DType): Element type of the entries and the norm weight (inferred). - cache_t (
KVCacheT): The state leaf's cache type (inferred): one head ofcoff * head_dimfloat32 channels, paged by position. - rotate (
Bool): The indexer's entries: Hadamard-rotated and snapped to FP4, rather than FP8 on the non-rotary channels.
Args:
- output (
TileTensor[dtype, Engine=output.Engine, linear_idx_type=output.linear_idx_type]):[rows, head_dim]entries.rowsbounds the window count; the rows past it are left unwritten. - kv (
TileTensor[.float32, Engine=kv.Engine, linear_idx_type=kv.linear_idx_type]):[total_rows, coff * head_dim]the chunk's rawwkvrows. - score (
TileTensor[.float32, Engine=score.Engine, linear_idx_type=score.linear_idx_type]): Same shape, the chunk's rawwgaterows. - ape (
TileTensor[.float32, Engine=ape.Engine, linear_idx_type=ape.linear_idx_type]):[ratio, coff * head_dim]per-slot gate bias. - norm_weight (
TileTensor[dtype, Engine=norm_weight.Engine, linear_idx_type=norm_weight.linear_idx_type]):[head_dim]RMSNorm weight. - freqs (
TileTensor[.float32, Engine=freqs.Engine, linear_idx_type=freqs.linear_idx_type]):[positions, rope_dim // 2, 2]rotary cos/sin table. - input_row_offsets (
TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type]):[batch + 1]row offsets ofkvandscore. - window_offsets (
TileTensor[.uint32, Engine=window_offsets.Engine, linear_idx_type=window_offsets.linear_idx_type]):[batch + 1]output row offsets, requestbowningceil(len_b / ratio)rows. - starts (
TileTensor[.int32, Engine=starts.Engine, linear_idx_type=starts.linear_idx_type]):[batch]tokens each request had cached before the chunk. - kv_cache (
cache_t): The state leaf's K (wkv) at this layer. - score_cache (
cache_t): The state leaf's V (wgate) at this layer. - epsilon (
Float32): RMSNorm epsilon. - ctx (
DeviceContext): Device context used to enqueue the kernel.