For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
head_rms_norm_rope
Muse Spark's per-head attention norms, one launch each.
Both kernels treat each 64-channel (token, head) row with 8 lanes of 8 channels (16 bytes) and no shared memory:
qk_norm_rope_store: of a[q | k | v | ...]projection,q = rope(rms_norm(q) * q_scale)to the output,k = rope(rms_norm(k))andvto the paged cache. RoPE's rotate-half partner is 4 lanes over.head_norm_gate:rms_norm(x) * sigmoid(gate)per head.
Every step rounds where the unfused ops round, and the norm adds in their
order (head_norm), so the outputs are theirs bit for bit. The schedule and
the arithmetic are the Moka component's
(Kernels/test/gpu/Moka/kernels/norm/head_norm_rope_plan.mojo), whose ops call
the lane functions below.
comptime values
HEAD_DIM
comptime HEAD_DIM = 64
In
LANES
comptime LANES = 8
Lanes per head row.
Out
Piece
comptime Piece = SIMD[.bfloat16, 8]
POISONS
comptime POISONS = 5
Planted faults, poison 1 through 5, each of which a test must catch: 1 drops Q's scale, 2 stores K one position late, 3 ropes with the lane's own half for its partner, 4 drops the gate's sigmoid, 5 skips the last token.
THREADS
comptime THREADS = 256
TILE_ROWS
comptime TILE_ROWS = 32
Head rows per CTA.
VEC
comptime VEC = 8
Structs
-
GateArgs:head_norm_gate's operands, one kernel parameter. -
QkArgs:qk_norm_rope_store's operands, one kernel parameter.
Functions
-
gate_lane: Lanei's 8 channels ofrms_norm(x) * sigmoid(gate). -
head_norm: The lane's piece ofrms_normover its 8-lane head row. -
head_norm_gate: Writesrms_norm(x) * sigmoid(gate)per head. -
qk_lane: Lanei's 8 channels of its (token, head) row. Every lane of a token's warps must call it: the norm and RoPE shuffle across them. -
qk_lanes: Lanes overtokenstokens: whole warps, since a token's rows are. -
qk_norm_rope_store: Writes the roped Q and stores the roped K and the V.