For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
qk_rms_norm_rope_store
Muse Spark's attention prologue and epilogue norms as graph ops.
kv_cache.qk_rms_norm_rope_split_store.ragged.paged: of the [q | k | v | ...]
projection, a weightless per-head qk-norm, a Q pre-scale, RoPE and the KV
store in one launch:
q = rope(rms_norm(q) * q_scale) -> output
k = rope(rms_norm(k)) -> key cache
v -> value cachenorm.head_rms_norm_sigmoid_gate: rms_norm(x) * sigmoid(gate) per head,
with the gate the projection's last columns.
Both run nn.head_rms_norm_rope (the prologue below SM-count tokens runs a
rowwise kernel) and match the unfused ops bit for bit.
Structsβ
- β
HeadRMSNormSigmoidGate: Registers thenorm.head_rms_norm_sigmoid_gategraph op. - β
QKGammaRMSNormRopeSplitStore: Registers thekv_cache.qk_gamma_rms_norm_rope_split_store.ragged.pagedgraph op. - β
QKRMSNormRopeSplitStore: Registers thekv_cache.qk_rms_norm_rope_split_store.ragged.pagedgraph op.