IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

gated_residual_rms_norm

A residual boundary with a learned gate, in one persistent launch.

Muse Spark joins each branch to the residual stream as

x = branch * (1 + post_weight)                    # bf16, when scaled
n = cast(x * rsqrt(mean(x^2) + post_eps))         # weightless norm
stream = cast(alpha * residual + beta * n)        # fp32 mix
normed = cast(stream * rsqrt(mean(stream^2) + norm_eps) * (1 + norm_weight))

with beta = sigmoid(gate / T) and alpha = sqrt(max(sigmoid(-gate / T) * (1 + beta), gate_eps)). Persistent CTAs take rows cta, cta + G, ...: a load lane bulk-copies each row's branch and residual into a two-deep ring, and eight work warps, a thread per four 16-byte pieces, compute their channels' gate coefficients once and keep them and the weights in registers for the whole launch. Rows are summed in the order of rms_norm's rowwise block tier, so the outputs match the unfused ops bit for bit at any row count. The schedule and the arithmetic are the Moka component's (Kernels/test/gpu/Moka/kernels/norm/gated_residual_norm_plan.mojo), whose ops call the helpers below.

Under SM-count rows gated_residual_rms_norm_split spreads each row over a cluster of SPLIT CTAs instead, a thread per 16-byte piece; it is the port of Kernels/test/gpu/Moka/kernels/norm/gated_split_plan.mojo.

comptime values​

COEF_BYTES​

comptime COEF_BYTES = 98304

Per channel: alpha, beta (fp32), 1 + post_weight, 1 + norm_weight.

HIDDEN​

comptime HIDDEN = 8192

Muse Spark's hidden size, the only width the kernel takes.

In​

comptime In[T: DType] = Pointer[Scalar[T], ImmUntrackedOrigin]

Parameters​

Out​

comptime Out[T: DType] = Pointer[Scalar[T], MutUntrackedOrigin]

Parameters​

Piece​

comptime Piece = SIMD[.bfloat16, 8]

PIECES​

comptime PIECES = 4

16-byte pieces of a row per work thread: rms_norm's 256-thread block tier's tiles, so each work thread is one of its threads.

POISONS​

comptime POISONS = 5

Planted faults, poison 1 through 5, each of which a test must catch: 1 swaps alpha and beta, 2 skips the branch scale, 3 drops the norm weight's offset, 4 normalizes the stream by the branch's sum, 5 skips the last row.

ROW_BYTES​

comptime ROW_BYTES = 16384

SLOTS​

comptime SLOTS = 2

SMEM​

comptime SMEM = (Int((add (mul (Int(256) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size), 64), 163840)) + Int(32))

The ring slots (branch then residual rows), the channel coefficients, the warps' two sums double buffered by row parity, then the full and empty barriers. Each thread writes and reads only its own channels' coefficients: in registers they spilled every row.

SPLIT​

comptime SPLIT = 8

CTAs of a cluster that share one row under SM-count rows.

SPLIT_THREADS​

comptime SPLIT_THREADS = 128

Threads of a split CTA: one 16-byte piece each, so CTA c's warp w is the 1024-thread block tier's warp c * 4 + w.

VEC​

comptime VEC = 8

The bf16 channels per 16-byte access.

Wide​

comptime Wide = SIMD[.float32, 8]

WIDE_WARPS​

comptime WIDE_WARPS = (Int(4) * (Int(256) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size))

Warps of the 1024-thread block tier rms_norm takes under SM-count rows; piece p of work warp w is its warp p * WORK_WARPS + w.

WORK_WARPS​

comptime WORK_WARPS = (Int(256) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)

WORKERS​

comptime WORKERS = ((Int(256) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size) * DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)

Structs​

Functions​

  • ​block_sum: The row's total from lane l holding warp l's published sum (0 past red_count), summed as rms_norm's block reduce sums its warps.
  • ​chan: The first channel of work thread rt's piece p: a warp's pieces are contiguous, so its 16-byte accesses coalesce.
  • ​gate_coefficients: Returns (alpha, beta) of the gate's logits, computed as the unfused graph computes them (which folds gate / T into a reciprocal multiply).
  • ​gated_residual_rms_norm: Writes the gated residual stream and its next pre-norm.
  • ​gated_residual_rms_norm_split: Writes the gated residual stream and its next pre-norm for fewer rows than SMs: a cluster of SPLIT CTAs per row, each CTA's warp sums gathered into rank 0 and its total sent back, summed in rms_norm's 1024-thread block tier's order so the bits match the unfused ops.
  • ​inv_rms:
  • ​mix: The new residual stream: the branch's norm mixed into the residual.
  • ​next_norm: The next pre-norm; gamma is 1 + norm_weight, rounded to bf16.
  • ​red_count: The published warp sums: the block tier's warp count.
  • ​red_index: Where work warp wid publishes its sum p of warp_sums.
  • ​scaled_branch: The branch before its norm; scale_weight is 1 + post_weight, rounded to bf16 as the graph does.
  • ​to_bf16: Rounds to bf16 two channels per cvt.rn.bf16x2.f32: a plain cast takes one XU conversion per channel, the kernel's busiest pipe.
  • ​warp_sums: The row's per-warp sums of squares as rms_norm's block tier forms them: under SM-count rows (wide, 1024 threads) one per piece, else one over the thread's pieces in element 0. Every lane holds them.

Was this page helpful?