IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

chunked_qkv_rope_store_kernel

def chunked_qkv_rope_store_kernel[qk_head_dim: Int, v_head_dim: Int, chunk_padding: Int = Int(0)](qkv: TileTensor[.bfloat16, Engine=qkv.Engine, address_space=qkv.address_space, linear_idx_type=qkv.linear_idx_type], row_offsets: TileTensor[.uint32, Engine=row_offsets.Engine, address_space=row_offsets.address_space, linear_idx_type=row_offsets.linear_idx_type], freqs: TileTensor[.bfloat16, Engine=freqs.Engine, address_space=freqs.address_space, linear_idx_type=freqs.linear_idx_type], k_cache: PagedKVCache[scale_dtype_=k_cache.scale_dtype_, quantization_granularity_=k_cache.quantization_granularity_], v_cache: PagedKVCache[k_cache.dtype_, k_cache.kv_params_, k_cache.page_size, k_cache.blocks_origin, k_cache.blocks_engine, k_cache.cache_lengths_origin, k_cache.cache_lengths_engine_, k_cache.lookup_table_origin, k_cache.lookup_table_engine, k_cache.scales_origin, k_cache.scales_engine, scale_dtype_=k_cache.scale_dtype_, quantization_granularity_=k_cache.quantization_granularity_], q_output: TileTensor[.bfloat16, Engine=q_output.Engine, address_space=q_output.address_space, linear_idx_type=q_output.linear_idx_type], value_scale: Float32)

Rotates Q/K and writes scaled V from contiguous per-chunk projections.

Query output rows at or past the final ragged offset are left unwritten.

Constraints:

Projections, RoPE coefficients, queries, and cache must use BF16. The cache must be unquantized and non-MLA. Projections and queries must have rank 3, offsets rank 1, and coefficients rank 2. Chunk, projection width, query head, cache head, and rotary extents must be static. Chunk and head counts must be positive, and both query and KV head counts must be divisible by chunks. Each chunk must contain its Q, K, and V head rows, and query output width must match the cache key width. Q/K and V widths must be positive and fit the cache head; rotary width must be positive and fit the Q/K head. Q/K, V, cache head, and rotary widths must be multiples of 8. A split cache's key and value widths must equal the Q/K and V widths.

Parameters:

  • ​qk_head_dim (Int): Unpadded Q/K head width.
  • ​v_head_dim (Int): Unpadded V head width.
  • ​chunk_padding (Int): Trailing projection rows to ignore in each chunk.

Args:

Was this page helpful?