IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

qk_rms_norm_rope_store

Muse Spark's attention prologue and epilogue norms as graph ops.

kv_cache.qk_rms_norm_rope_split_store.ragged.paged: of the [q | k | v | ...] projection, a weightless per-head qk-norm, a Q pre-scale, RoPE and the KV store in one launch:

q = rope(rms_norm(q) * q_scale)     -> output
k = rope(rms_norm(k))               -> key cache
v                                   -> value cache

norm.head_rms_norm_sigmoid_gate: rms_norm(x) * sigmoid(gate) per head, with the gate the projection's last columns.

Both run nn.head_rms_norm_rope (the prologue below SM-count tokens runs a rowwise kernel) and match the unfused ops bit for bit.

Structs​

Was this page helpful?