IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

compress_windows_ragged_paged

def compress_windows_ragged_paged[dtype: DType, cache_t: KVCacheT, //, rotate: Bool](output: TileTensor[dtype, Engine=output.Engine, linear_idx_type=output.linear_idx_type], kv: TileTensor[.float32, Engine=kv.Engine, linear_idx_type=kv.linear_idx_type], score: TileTensor[.float32, Engine=score.Engine, linear_idx_type=score.linear_idx_type], ape: TileTensor[.float32, Engine=ape.Engine, linear_idx_type=ape.linear_idx_type], norm_weight: TileTensor[dtype, Engine=norm_weight.Engine, linear_idx_type=norm_weight.linear_idx_type], freqs: TileTensor[.float32, Engine=freqs.Engine, linear_idx_type=freqs.linear_idx_type], input_row_offsets: TileTensor[.uint32, Engine=input_row_offsets.Engine, linear_idx_type=input_row_offsets.linear_idx_type], window_offsets: TileTensor[.uint32, Engine=window_offsets.Engine, linear_idx_type=window_offsets.linear_idx_type], starts: TileTensor[.int32, Engine=starts.Engine, linear_idx_type=starts.linear_idx_type], kv_cache: cache_t, score_cache: cache_t, epsilon: Float32, ctx: DeviceContextBase)

Pools every window of a ragged chunk into its compressed entry.

Parameters:

  • ​dtype (DType): Element type of the entries and the norm weight (inferred).
  • ​cache_t (KVCacheT): The state leaf's cache type (inferred): one head of coff * head_dim float32 channels, paged by position.
  • ​rotate (Bool): The indexer's entries: Hadamard-rotated and snapped to FP4, rather than FP8 on the non-rotary channels.

Args:

Was this page helpful?