For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
QKGammaRMSNormRopeSplitStore
struct QKGammaRMSNormRopeSplitStore
Registers the kv_cache.qk_gamma_rms_norm_rope_split_store.ragged.paged graph op.
Implemented traits
Methods
execute
static def execute[n_q_heads: Int](output: T, qkv: T, input_row_offsets: T, freqs_cis: T, q_gamma: T, k_gamma: T, kv_blocks: T, page_stride: T, cache_lengths: T, kv_lookup_table: T, max_prompt_length: T, max_cache_length: T, layer_idx: UInt32, epsilon: Float32, ctx: DeviceContext)
Writes the roped Q and stores the roped K and the V to the cache, with a learned per-channel weight on each of the Q and K norms.
Matches rms_norm(q, q_gamma), rms_norm(k, k_gamma) (float32
products, one rounding) and rope_split_store with a rotate-half
head layout, bit for bit up to the reduction order.
Parameters:
- n_q_heads (
Int): Query heads inqkv(this device's).
Args:
- output (
T): The roped Q[tokens, n_q_heads * head_dim], in theqkvdtype or a float8 type (rounded through theqkvdtype). - qkv (
T): The projection[tokens, q_dim + 2 * kv_dim + ...]. - input_row_offsets (
T): Ragged offsets[batch + 1]. - freqs_cis (
T): Interleaved (cos, sin) pairs[positions, head_dim]. - q_gamma (
T): The Q norm weight[head_dim]. - k_gamma (
T): The K norm weight[head_dim]. - kv_blocks (
T): The paged KV cache blocks. - page_stride (
T): The paged cache's page stride. - cache_lengths (
T): Each sequence's cached length. - kv_lookup_table (
T): The paged cache's page table. - max_prompt_length (
T): The batch's longest prompt. - max_cache_length (
T): The batch's longest cache. - layer_idx (
UInt32): The cache layer to store into. - epsilon (
Float32): The norms' epsilon. - ctx (
DeviceContext): Device context.
Raises:
Error: If the launch fails.