IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

rope_qkv_store_kernel

def rope_qkv_store_kernel[freq_dtype: DType](q: TileTensor[.bfloat16, Engine=q.Engine, address_space=q.address_space, linear_idx_type=q.linear_idx_type], k: TileTensor[.bfloat16, Engine=k.Engine, address_space=k.address_space, linear_idx_type=k.linear_idx_type], v: TileTensor[.bfloat16, Engine=v.Engine, address_space=v.address_space, linear_idx_type=v.linear_idx_type], row_offsets: TileTensor[.uint32, Engine=row_offsets.Engine, address_space=row_offsets.address_space, linear_idx_type=row_offsets.linear_idx_type], freqs: TileTensor[freq_dtype, Engine=freqs.Engine, address_space=freqs.address_space, linear_idx_type=freqs.linear_idx_type], k_cache: PagedKVCache[scale_dtype_=k_cache.scale_dtype_, quantization_granularity_=k_cache.quantization_granularity_], v_cache: PagedKVCache[k_cache.dtype_, k_cache.kv_params_, k_cache.page_size, k_cache.blocks_origin, k_cache.blocks_engine, k_cache.cache_lengths_origin, k_cache.cache_lengths_engine_, k_cache.lookup_table_origin, k_cache.lookup_table_engine, k_cache.scales_origin, k_cache.scales_engine, scale_dtype_=k_cache.scale_dtype_, quantization_granularity_=k_cache.quantization_granularity_], q_output: TileTensor[.bfloat16, Engine=q_output.Engine, address_space=q_output.address_space, linear_idx_type=q_output.linear_idx_type], value_scale: Float32)

Rotates Q/K and writes scaled V from head-major projections.

Query output rows at or past the final ragged offset are left unwritten.

Constraints:

The cache must be unquantized, non-MLA, and BF16, with a row for the key and a row for the value: a cache whose rows pack a key and a narrower value is unsupported. Q, K and V must be BF16 rank 3, row offsets rank 1, coefficients rank 2 and the query output rank 3. Head counts, head width, rotary width and the cache extents must be static, and the token extents of Q, K and V must match. Head and rotary widths must be multiples of 8, head counts positive, and rotary width must be positive and fit the head.

Parameters:

  • ​freq_dtype (DType): Element type of the RoPE coefficients (inferred).

Args:

Was this page helpful?