For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
k_rope_v_store_kernel
def k_rope_v_store_kernel[freq_dtype: DType](k: TileTensor[.bfloat16, Engine=k.Engine, address_space=k.address_space, linear_idx_type=k.linear_idx_type], v: TileTensor[.bfloat16, Engine=v.Engine, address_space=v.address_space, linear_idx_type=v.linear_idx_type], row_offsets: TileTensor[.uint32, Engine=row_offsets.Engine, address_space=row_offsets.address_space, linear_idx_type=row_offsets.linear_idx_type], freqs: TileTensor[freq_dtype, Engine=freqs.Engine, address_space=freqs.address_space, linear_idx_type=freqs.linear_idx_type], k_cache: PagedKVCache[scale_dtype_=k_cache.scale_dtype_, quantization_granularity_=k_cache.quantization_granularity_], v_cache: PagedKVCache[k_cache.dtype_, k_cache.kv_params_, k_cache.page_size, k_cache.blocks_origin, k_cache.blocks_engine, k_cache.cache_lengths_origin, k_cache.cache_lengths_engine_, k_cache.lookup_table_origin, k_cache.lookup_table_engine, k_cache.scales_origin, k_cache.scales_engine, scale_dtype_=k_cache.scale_dtype_, quantization_granularity_=k_cache.quantization_granularity_], value_scale: Float32)
Rotates K and writes scaled V, for a call that ropes no query.
Constraints:
As rope_qkv_store_kernel, minus the query: the cache must be
unquantized, non-MLA, BF16, and not split.
Parameters:
- freq_dtype (
DType): Element type of the RoPE coefficients (inferred).
Args:
- k (
TileTensor[.bfloat16, Engine=k.Engine, address_space=k.address_space, linear_idx_type=k.linear_idx_type]): BF16 unrotated keys [tokens, kv_heads, head_size]. - v (
TileTensor[.bfloat16, Engine=v.Engine, address_space=v.address_space, linear_idx_type=v.linear_idx_type]): BF16 values [tokens, kv_heads, head_size], unscaled. - row_offsets (
TileTensor[.uint32, Engine=row_offsets.Engine, address_space=row_offsets.address_space, linear_idx_type=row_offsets.linear_idx_type]): Ragged offsets [batch_size + 1]. - freqs (
TileTensor[freq_dtype, Engine=freqs.Engine, address_space=freqs.address_space, linear_idx_type=freqs.linear_idx_type]): Interleaved RoPE coefficients for NeoX-paired inputs, either dtype; the rotation runs in the coefficients' own dtype, asrope_k_cachedoes. - k_cache (
PagedKVCache[scale_dtype_=k_cache.scale_dtype_, quantization_granularity_=k_cache.quantization_granularity_]): Paged key view for the selected layer. - v_cache (
PagedKVCache[k_cache.dtype_, k_cache.kv_params_, k_cache.page_size, k_cache.blocks_origin, k_cache.blocks_engine, k_cache.cache_lengths_origin, k_cache.cache_lengths_engine_, k_cache.lookup_table_origin, k_cache.lookup_table_engine, k_cache.scales_origin, k_cache.scales_engine, scale_dtype_=k_cache.scale_dtype_, quantization_granularity_=k_cache.quantization_granularity_]): Disjoint paged value view for the same layer. - value_scale (
Float32): FP32 multiplier applied before V's BF16 rounding.