For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
rope_qkv_store_paged_ragged
def rope_qkv_store_paged_ragged[freq_dtype: DType](q: TileTensor[.bfloat16, Engine=q.Engine, address_space=q.address_space, linear_idx_type=q.linear_idx_type], k: TileTensor[.bfloat16, Engine=k.Engine, address_space=k.address_space, linear_idx_type=k.linear_idx_type], v: TileTensor[.bfloat16, Engine=v.Engine, address_space=v.address_space, linear_idx_type=v.linear_idx_type], row_offsets: TileTensor[.uint32, Engine=row_offsets.Engine, address_space=row_offsets.address_space, linear_idx_type=row_offsets.linear_idx_type], freqs: TileTensor[freq_dtype, Engine=freqs.Engine, address_space=freqs.address_space, linear_idx_type=freqs.linear_idx_type], collection: PagedKVCacheCollection[scale_dtype_=collection.scale_dtype_, quantization_granularity_=collection.quantization_granularity_], layer_idx: UInt32, q_output: TileTensor[.bfloat16, Engine=q_output.Engine, address_space=q_output.address_space, linear_idx_type=q_output.linear_idx_type], value_scale: Float32, ctx: DeviceContext)
Launches the head-major QKV rotation and store.
Query output rows at or past the final ragged offset are left unwritten.
Constraints:
The cache must be unquantized, non-MLA and BF16, with a row for the
key and a row for the value. Q, K and V must share a token extent and
be as wide as the cache head; row_offsets must be non-empty.
Parameters:
- freq_dtype (
DType): Element type of the RoPE coefficients (inferred).
Args:
- q (
TileTensor[.bfloat16, Engine=q.Engine, address_space=q.address_space, linear_idx_type=q.linear_idx_type]): BF16 queries [tokens, query_heads, head_size]. - k (
TileTensor[.bfloat16, Engine=k.Engine, address_space=k.address_space, linear_idx_type=k.linear_idx_type]): BF16 unrotated keys [tokens, kv_heads, head_size]. - v (
TileTensor[.bfloat16, Engine=v.Engine, address_space=v.address_space, linear_idx_type=v.linear_idx_type]): BF16 values [tokens, kv_heads, head_size], unscaled. - row_offsets (
TileTensor[.uint32, Engine=row_offsets.Engine, address_space=row_offsets.address_space, linear_idx_type=row_offsets.linear_idx_type]): Ragged offsets [batch_size + 1]. - freqs (
TileTensor[freq_dtype, Engine=freqs.Engine, address_space=freqs.address_space, linear_idx_type=freqs.linear_idx_type]): RoPE table [max_position, rotary_width], either dtype. - collection (
PagedKVCacheCollection[scale_dtype_=collection.scale_dtype_, quantization_granularity_=collection.quantization_granularity_]): Unquantized, non-MLA paged KV collection. - layer_idx (
UInt32): Layer to append to in both cache planes. - q_output (
TileTensor[.bfloat16, Engine=q_output.Engine, address_space=q_output.address_space, linear_idx_type=q_output.linear_idx_type]): Rotated queries [tokens, query_heads, head_size]. - value_scale (
Float32): FP32 multiplier; V times this scale is rounded to BF16 before storing. - ctx (
DeviceContext): GPU device context.
Raises:
Error: If the projections' extents differ, the cache layer is
invalid, or row_offsets is empty.