IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

head_rms_norm_rope

Muse Spark's per-head attention norms, one launch each.

Both kernels treat each 64-channel (token, head) row with 8 lanes of 8 channels (16 bytes) and no shared memory:

  • qk_norm_rope_store: of a [q | k | v | ...] projection, q = rope(rms_norm(q) * q_scale) to the output, k = rope(rms_norm(k)) and v to the paged cache. RoPE's rotate-half partner is 4 lanes over.
  • head_norm_gate: rms_norm(x) * sigmoid(gate) per head.

Every step rounds where the unfused ops round, and the norm adds in their order (head_norm), so the outputs are theirs bit for bit. The schedule and the arithmetic are the Moka component's (Kernels/test/gpu/Moka/kernels/norm/head_norm_rope_plan.mojo), whose ops call the lane functions below.

comptime values​

HEAD_DIM​

comptime HEAD_DIM = 64

In​

comptime In[T: DType] = Pointer[Scalar[T], ImmUntrackedOrigin]

Parameters​

LANES​

comptime LANES = 8

Lanes per head row.

Out​

comptime Out[T: DType] = Pointer[Scalar[T], MutUntrackedOrigin]

Parameters​

Piece​

comptime Piece = SIMD[.bfloat16, 8]

POISONS​

comptime POISONS = 5

Planted faults, poison 1 through 5, each of which a test must catch: 1 drops Q's scale, 2 stores K one position late, 3 ropes with the lane's own half for its partner, 4 drops the gate's sigmoid, 5 skips the last token.

THREADS​

comptime THREADS = 256

TILE_ROWS​

comptime TILE_ROWS = 32

Head rows per CTA.

VEC​

comptime VEC = 8

Structs​

  • ​GateArgs: head_norm_gate's operands, one kernel parameter.
  • ​QkArgs: qk_norm_rope_store's operands, one kernel parameter.

Functions​

  • ​gate_lane: Lane i's 8 channels of rms_norm(x) * sigmoid(gate).
  • ​head_norm: The lane's piece of rms_norm over its 8-lane head row.
  • ​head_norm_gate: Writes rms_norm(x) * sigmoid(gate) per head.
  • ​qk_lane: Lane i's 8 channels of its (token, head) row. Every lane of a token's warps must call it: the norm and RoPE shuffle across them.
  • ​qk_lanes: Lanes over tokens tokens: whole warps, since a token's rows are.
  • ​qk_norm_rope_store: Writes the roped Q and stores the roped K and the V.

Was this page helpful?