IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

QKGammaRMSNormRopeSplitStore

struct QKGammaRMSNormRopeSplitStore

Registers the kv_cache.qk_gamma_rms_norm_rope_split_store.ragged.paged graph op.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[n_q_heads: Int](output: T, qkv: T, input_row_offsets: T, freqs_cis: T, q_gamma: T, k_gamma: T, kv_blocks: T, page_stride: T, cache_lengths: T, kv_lookup_table: T, max_prompt_length: T, max_cache_length: T, layer_idx: UInt32, epsilon: Float32, ctx: DeviceContext)

Writes the roped Q and stores the roped K and the V to the cache, with a learned per-channel weight on each of the Q and K norms.

Matches rms_norm(q, q_gamma), rms_norm(k, k_gamma) (float32 products, one rounding) and rope_split_store with a rotate-half head layout, bit for bit up to the reduction order.

Parameters:

  • ​n_q_heads (Int): Query heads in qkv (this device's).

Args:

  • ​output (T): The roped Q [tokens, n_q_heads * head_dim], in the qkv dtype or a float8 type (rounded through the qkv dtype).
  • ​qkv (T): The projection [tokens, q_dim + 2 * kv_dim + ...].
  • ​input_row_offsets (T): Ragged offsets [batch + 1].
  • ​freqs_cis (T): Interleaved (cos, sin) pairs [positions, head_dim].
  • ​q_gamma (T): The Q norm weight [head_dim].
  • ​k_gamma (T): The K norm weight [head_dim].
  • ​kv_blocks (T): The paged KV cache blocks.
  • ​page_stride (T): The paged cache's page stride.
  • ​cache_lengths (T): Each sequence's cached length.
  • ​kv_lookup_table (T): The paged cache's page table.
  • ​max_prompt_length (T): The batch's longest prompt.
  • ​max_cache_length (T): The batch's longest cache.
  • ​layer_idx (UInt32): The cache layer to store into.
  • ​epsilon (Float32): The norms' epsilon.
  • ​ctx (DeviceContext): Device context.

Raises:

Error: If the launch fails.

Was this page helpful?