IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

rope_qkv_store_paged_ragged

def rope_qkv_store_paged_ragged[freq_dtype: DType](q: TileTensor[.bfloat16, Engine=q.Engine, address_space=q.address_space, linear_idx_type=q.linear_idx_type], k: TileTensor[.bfloat16, Engine=k.Engine, address_space=k.address_space, linear_idx_type=k.linear_idx_type], v: TileTensor[.bfloat16, Engine=v.Engine, address_space=v.address_space, linear_idx_type=v.linear_idx_type], row_offsets: TileTensor[.uint32, Engine=row_offsets.Engine, address_space=row_offsets.address_space, linear_idx_type=row_offsets.linear_idx_type], freqs: TileTensor[freq_dtype, Engine=freqs.Engine, address_space=freqs.address_space, linear_idx_type=freqs.linear_idx_type], collection: PagedKVCacheCollection[scale_dtype_=collection.scale_dtype_, quantization_granularity_=collection.quantization_granularity_], layer_idx: UInt32, q_output: TileTensor[.bfloat16, Engine=q_output.Engine, address_space=q_output.address_space, linear_idx_type=q_output.linear_idx_type], value_scale: Float32, ctx: DeviceContext)

Launches the head-major QKV rotation and store.

Query output rows at or past the final ragged offset are left unwritten.

Constraints:

The cache must be unquantized, non-MLA and BF16, with a row for the key and a row for the value. Q, K and V must share a token extent and be as wide as the cache head; row_offsets must be non-empty.

Parameters:

  • ​freq_dtype (DType): Element type of the RoPE coefficients (inferred).

Args:

Raises:

Error: If the projections' extents differ, the cache layer is invalid, or row_offsets is empty.

Was this page helpful?