For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
rope_qkv_store_kernel
def rope_qkv_store_kernel[freq_dtype: DType](q: TileTensor[.bfloat16, Engine=q.Engine, address_space=q.address_space, linear_idx_type=q.linear_idx_type], k: TileTensor[.bfloat16, Engine=k.Engine, address_space=k.address_space, linear_idx_type=k.linear_idx_type], v: TileTensor[.bfloat16, Engine=v.Engine, address_space=v.address_space, linear_idx_type=v.linear_idx_type], row_offsets: TileTensor[.uint32, Engine=row_offsets.Engine, address_space=row_offsets.address_space, linear_idx_type=row_offsets.linear_idx_type], freqs: TileTensor[freq_dtype, Engine=freqs.Engine, address_space=freqs.address_space, linear_idx_type=freqs.linear_idx_type], k_cache: PagedKVCache[scale_dtype_=k_cache.scale_dtype_, quantization_granularity_=k_cache.quantization_granularity_], v_cache: PagedKVCache[k_cache.dtype_, k_cache.kv_params_, k_cache.page_size, k_cache.blocks_origin, k_cache.blocks_engine, k_cache.cache_lengths_origin, k_cache.cache_lengths_engine_, k_cache.lookup_table_origin, k_cache.lookup_table_engine, k_cache.scales_origin, k_cache.scales_engine, scale_dtype_=k_cache.scale_dtype_, quantization_granularity_=k_cache.quantization_granularity_], q_output: TileTensor[.bfloat16, Engine=q_output.Engine, address_space=q_output.address_space, linear_idx_type=q_output.linear_idx_type], value_scale: Float32)
Rotates Q/K and writes scaled V from head-major projections.
Query output rows at or past the final ragged offset are left unwritten.
Constraints:
The cache must be unquantized, non-MLA, and BF16, with a row for the key and a row for the value: a cache whose rows pack a key and a narrower value is unsupported. Q, K and V must be BF16 rank 3, row offsets rank 1, coefficients rank 2 and the query output rank 3. Head counts, head width, rotary width and the cache extents must be static, and the token extents of Q, K and V must match. Head and rotary widths must be multiples of 8, head counts positive, and rotary width must be positive and fit the head.
Parameters:
- freq_dtype (
DType): Element type of the RoPE coefficients (inferred).
Args:
- q (
TileTensor[.bfloat16, Engine=q.Engine, address_space=q.address_space, linear_idx_type=q.linear_idx_type]): BF16 queries [tokens, query_heads, head_size]. - k (
TileTensor[.bfloat16, Engine=k.Engine, address_space=k.address_space, linear_idx_type=k.linear_idx_type]): BF16 unrotated keys [tokens, kv_heads, head_size]. - v (
TileTensor[.bfloat16, Engine=v.Engine, address_space=v.address_space, linear_idx_type=v.linear_idx_type]): BF16 values [tokens, kv_heads, head_size], unscaled. - row_offsets (
TileTensor[.uint32, Engine=row_offsets.Engine, address_space=row_offsets.address_space, linear_idx_type=row_offsets.linear_idx_type]): Ragged offsets [batch_size + 1]. - freqs (
TileTensor[freq_dtype, Engine=freqs.Engine, address_space=freqs.address_space, linear_idx_type=freqs.linear_idx_type]): Interleaved RoPE coefficients for NeoX-paired inputs, either dtype; the rotation runs in the coefficients' own dtype, asrope_k_cachedoes. - k_cache (
PagedKVCache[scale_dtype_=k_cache.scale_dtype_, quantization_granularity_=k_cache.quantization_granularity_]): Paged key view for the selected layer. - v_cache (
PagedKVCache[k_cache.dtype_, k_cache.kv_params_, k_cache.page_size, k_cache.blocks_origin, k_cache.blocks_engine, k_cache.cache_lengths_origin, k_cache.cache_lengths_engine_, k_cache.lookup_table_origin, k_cache.lookup_table_engine, k_cache.scales_origin, k_cache.scales_engine, scale_dtype_=k_cache.scale_dtype_, quantization_granularity_=k_cache.quantization_granularity_]): Disjoint paged value view for the same layer. - q_output (
TileTensor[.bfloat16, Engine=q_output.Engine, address_space=q_output.address_space, linear_idx_type=q_output.linear_idx_type]): Rotated queries [tokens, query_heads, head_size]. - value_scale (
Float32): FP32 multiplier applied before V's BF16 rounding.