IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

ApplyQKRMSNorm

struct ApplyQKRMSNorm

Fused per-element QK-RMSNorm apply for two operands Q and K.

Given the already cross-rank-reduced per-row statistics qk_var [M, 2] (col 0 = mean(q^2), col 1 = mean(k^2), float32) and per-column float32 scales gamma_q [Nq] / gamma_k [Nk], applies in a single launch:

q_out[m,c] = cast((cast(q[m,c], f32) * rsqrt(qk_var[m,0] + eps)) * gamma_q[c], q.dtype) and likewise for K with column 1. The grouping ((x * rs) * gamma) then cast matches the unfused graph this replaces for bit-accuracy. This fuses the QK-RMSNorm apply chain (~7 tiny elementwise/View kernels) into one launch, used for cross-head QK-RMSNorm under tensor parallelism.

Outputs (in order): q_out [M, Nq], k_out [M, Nk] (both q/k dtype). Inputs (in order): q [M, Nq], k [M, Nk] (activation dtype), qk_var [M, 2] (float32), gamma_q [Nq] (float32), gamma_k [Nk] (float32). Attribute: epsilon (float32 host scalar).

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute(q_out: T, k_out: T, q: T, k: T, qk_var: T, gamma_q: T, gamma_k: T, epsilon: Float32, ctx: DeviceContext)

Executes the norm.apply_qk_rms_norm graph op.

Args:

  • ​q_out (T): Output tensor receiving the normalized q.
  • ​k_out (T): Output tensor receiving the normalized k.
  • ​q (T): Query input tensor.
  • ​k (T): Key input tensor.
  • ​qk_var (T): Per-row variance tensor of shape [M, 2].
  • ​gamma_q (T): Per-column scale weights for q.
  • ​gamma_k (T): Per-column scale weights for k.
  • ​epsilon (Float32): Small constant for numerical stability.
  • ​ctx (DeviceContext): Device context used to enqueue the kernel.

Raises:

Error: If output shapes do not match the corresponding inputs.

Was this page helpful?