For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
gated_residual_rms_norm
A residual boundary with a learned gate, in one persistent launch.
Muse Spark joins each branch to the residual stream as
x = branch * (1 + post_weight) # bf16, when scaled
n = cast(x * rsqrt(mean(x^2) + post_eps)) # weightless norm
stream = cast(alpha * residual + beta * n) # fp32 mix
normed = cast(stream * rsqrt(mean(stream^2) + norm_eps) * (1 + norm_weight))with beta = sigmoid(gate / T) and
alpha = sqrt(max(sigmoid(-gate / T) * (1 + beta), gate_eps)). Persistent
CTAs take rows cta, cta + G, ...: a load lane bulk-copies each row's branch
and residual into a two-deep ring, and eight work warps, a thread per four
16-byte pieces, compute their channels' gate coefficients once and keep them
and the weights in registers for the whole launch. Rows are summed in the
order of rms_norm's rowwise block tier, so the outputs match the unfused
ops bit for bit at any row count. The schedule and the
arithmetic are the Moka component's
(Kernels/test/gpu/Moka/kernels/norm/gated_residual_norm_plan.mojo), whose
ops call the helpers below.
Under SM-count rows gated_residual_rms_norm_split spreads each row over a
cluster of SPLIT CTAs instead, a thread per 16-byte piece; it is the port
of Kernels/test/gpu/Moka/kernels/norm/gated_split_plan.mojo.
comptime values
COEF_BYTES
comptime COEF_BYTES = 98304
Per channel: alpha, beta (fp32), 1 + post_weight, 1 + norm_weight.
HIDDEN
comptime HIDDEN = 8192
Muse Spark's hidden size, the only width the kernel takes.
In
Out
Piece
comptime Piece = SIMD[.bfloat16, 8]
PIECES
comptime PIECES = 4
16-byte pieces of a row per work thread: rms_norm's 256-thread block tier's tiles, so each work thread is one of its threads.
POISONS
comptime POISONS = 5
Planted faults, poison 1 through 5, each of which a test must catch: 1 swaps alpha and beta, 2 skips the branch scale, 3 drops the norm weight's offset, 4 normalizes the stream by the branch's sum, 5 skips the last row.
ROW_BYTES
comptime ROW_BYTES = 16384
SLOTS
comptime SLOTS = 2
SMEM
comptime SMEM = (Int((add (mul (Int(256) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size), 64), 163840)) + Int(32))
The ring slots (branch then residual rows), the channel coefficients, the warps' two sums double buffered by row parity, then the full and empty barriers. Each thread writes and reads only its own channels' coefficients: in registers they spilled every row.
SPLIT
comptime SPLIT = 8
CTAs of a cluster that share one row under SM-count rows.
SPLIT_THREADS
comptime SPLIT_THREADS = 128
Threads of a split CTA: one 16-byte piece each, so CTA c's warp w is the 1024-thread block tier's warp c * 4 + w.
VEC
comptime VEC = 8
The bf16 channels per 16-byte access.
Wide
comptime Wide = SIMD[.float32, 8]
WIDE_WARPS
comptime WIDE_WARPS = (Int(4) * (Int(256) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size))
Warps of the 1024-thread block tier rms_norm takes under SM-count rows; piece p of work warp w is its warp p * WORK_WARPS + w.
WORK_WARPS
comptime WORK_WARPS = (Int(256) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)
WORKERS
comptime WORKERS = ((Int(256) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size) * DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)
Structs
-
GatedResidualArgs: The boundary's operands, one kernel parameter. Rows are dense.
Functions
-
block_sum: The row's total from lanelholding warpl's published sum (0 pastred_count), summed asrms_norm's block reduce sums its warps. -
chan: The first channel of work threadrt's piecep: a warp's pieces are contiguous, so its 16-byte accesses coalesce. -
gate_coefficients: Returns(alpha, beta)of the gate's logits, computed as the unfused graph computes them (which foldsgate / Tinto a reciprocal multiply). -
gated_residual_rms_norm: Writes the gated residual stream and its next pre-norm. -
gated_residual_rms_norm_split: Writes the gated residual stream and its next pre-norm for fewer rows than SMs: a cluster ofSPLITCTAs per row, each CTA's warp sums gathered into rank 0 and its total sent back, summed inrms_norm's 1024-thread block tier's order so the bits match the unfused ops. -
inv_rms: -
mix: The new residual stream: the branch's norm mixed into the residual. -
next_norm: The next pre-norm;gammais1 + norm_weight, rounded to bf16. -
red_count: The published warp sums: the block tier's warp count. -
red_index: Where work warpwidpublishes its sumpofwarp_sums. -
scaled_branch: The branch before its norm;scale_weightis1 + post_weight, rounded to bf16 as the graph does. -
to_bf16: Rounds to bf16 two channels percvt.rn.bf16x2.f32: a plain cast takes one XU conversion per channel, the kernel's busiest pipe. -
warp_sums: The row's per-warp sums of squares asrms_norm's block tier forms them: under SM-count rows (wide, 1024 threads) one per piece, else one over the thread's pieces in element 0. Every lane holds them.