IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

allgather_gated_residual_rmsnorm

def allgather_gated_residual_rmsnorm[dtype: DType, gate_dtype: DType, ngpus: Int, in_layout: TensorLayout, in_origin: Origin[mut=in_origin.mut], //, scale_branch: Bool](input_buffers: Array[TileTensor[dtype, in_layout, in_origin], ngpus], residual: TileTensor[dtype, Engine=residual.Engine, address_space=residual.address_space, linear_idx_type=residual.linear_idx_type], post_weight: TileTensor[dtype, Engine=post_weight.Engine, address_space=post_weight.address_space, linear_idx_type=post_weight.linear_idx_type], gate: TileTensor[gate_dtype, Engine=gate.Engine, address_space=gate.address_space, linear_idx_type=gate.linear_idx_type], norm_weight: TileTensor[dtype, Engine=norm_weight.Engine, address_space=norm_weight.address_space, linear_idx_type=norm_weight.linear_idx_type], stream_out: TileTensor[dtype, Engine=stream_out.Engine, address_space=stream_out.address_space, linear_idx_type=stream_out.linear_idx_type], normed_out: TileTensor[dtype, Engine=normed_out.Engine, address_space=normed_out.address_space, linear_idx_type=normed_out.linear_idx_type], post_eps: Float32, norm_eps: Float32, temperature: Float32, gate_eps: Float32, rank_sigs: Array[Pointer[Signal, MutAnyOrigin], ngpus], ctx: DeviceContext, my_rank: Int)

All-gathers a branch's column shards and joins it to the residual stream.

Per gathered row b, as Muse Spark's unfused residual boundary computes it:

n = rms_norm(b * (1 + post_weight))          # weightless norm
stream = alpha * residual + beta * n         # fp32, rounded once
normed = rms_norm(stream) * (1 + norm_weight)

with beta = sigmoid(gate / T) and alpha = sqrt(max(sigmoid(-gate / T) * (1 + beta), gate_eps)). Each step rounds to dtype where the graph does; the row sums reduce in a different order than rms_norm's, so the outputs agree to its rounding.

Parameters:

  • ​dtype (DType): Element type of the activations and norm weights.
  • ​gate_dtype (DType): Element type of the gate's logits.
  • ​ngpus (Int): Number of devices, all in one group.
  • ​in_layout (TensorLayout): Layout of the branch shards.
  • ​in_origin (Origin[mut=mut]): Origin of the branch shards.
  • ​scale_branch (Bool): Whether the branch is scaled by 1 + post_weight before its norm; otherwise post_weight is not read.

Args:

Was this page helpful?