For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
QKRMSNormRopeSplitStore
struct QKRMSNormRopeSplitStore
Registers the kv_cache.qk_rms_norm_rope_split_store.ragged.paged graph op.
Implemented traits
Methods
execute
static def execute[n_q_heads: Int](output: T, qkv: T, input_row_offsets: T, freqs_cis: T, kv_blocks: T, page_stride: T, cache_lengths: T, kv_lookup_table: T, max_prompt_length: T, max_cache_length: T, layer_idx: UInt32, epsilon: Float32, q_scale: Float32, ctx: DeviceContext)
Writes the roped Q and stores the roped K and the V to the cache.
Parameters:
- n_q_heads (
Int): Query heads inqkv(this device's).
Args:
- output (
T): The roped Q[tokens, n_q_heads * head_dim]. - qkv (
T): The projection[tokens, q_dim + 2 * kv_dim + ...]. - input_row_offsets (
T): Ragged offsets[batch + 1]. - freqs_cis (
T): Interleaved (cos, sin) pairs[positions, head_dim], applied to rotate-half (safetensors) heads. - kv_blocks (
T): The paged KV cache blocks. - page_stride (
T): The paged cache's page stride. - cache_lengths (
T): Each sequence's cached length. - kv_lookup_table (
T): The paged cache's page table. - max_prompt_length (
T): The batch's longest prompt. - max_cache_length (
T): The batch's longest cache. - layer_idx (
UInt32): The cache layer to store into. - epsilon (
Float32): The norm's epsilon. - q_scale (
Float32): The factor on the normed Q, rounded to the Q dtype first. - ctx (
DeviceContext): Device context.
Raises:
Error: If the launch fails.