IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

QKRMSNormRopeSplitStore

struct QKRMSNormRopeSplitStore

Registers the kv_cache.qk_rms_norm_rope_split_store.ragged.paged graph op.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[n_q_heads: Int](output: T, qkv: T, input_row_offsets: T, freqs_cis: T, kv_blocks: T, page_stride: T, cache_lengths: T, kv_lookup_table: T, max_prompt_length: T, max_cache_length: T, layer_idx: UInt32, epsilon: Float32, q_scale: Float32, ctx: DeviceContext)

Writes the roped Q and stores the roped K and the V to the cache.

Parameters:

  • ​n_q_heads (Int): Query heads in qkv (this device's).

Args:

  • ​output (T): The roped Q [tokens, n_q_heads * head_dim].
  • ​qkv (T): The projection [tokens, q_dim + 2 * kv_dim + ...].
  • ​input_row_offsets (T): Ragged offsets [batch + 1].
  • ​freqs_cis (T): Interleaved (cos, sin) pairs [positions, head_dim], applied to rotate-half (safetensors) heads.
  • ​kv_blocks (T): The paged KV cache blocks.
  • ​page_stride (T): The paged cache's page stride.
  • ​cache_lengths (T): Each sequence's cached length.
  • ​kv_lookup_table (T): The paged cache's page table.
  • ​max_prompt_length (T): The batch's longest prompt.
  • ​max_cache_length (T): The batch's longest cache.
  • ​layer_idx (UInt32): The cache layer to store into.
  • ​epsilon (Float32): The norm's epsilon.
  • ​q_scale (Float32): The factor on the normed Q, rounded to the Q dtype first.
  • ​ctx (DeviceContext): Device context.

Raises:

Error: If the launch fails.

Was this page helpful?