For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
allgather_gated_residual_rmsnorm
def allgather_gated_residual_rmsnorm[dtype: DType, gate_dtype: DType, ngpus: Int, in_layout: TensorLayout, in_origin: Origin[mut=in_origin.mut], //, scale_branch: Bool](input_buffers: Array[TileTensor[dtype, in_layout, in_origin], ngpus], residual: TileTensor[dtype, Engine=residual.Engine, address_space=residual.address_space, linear_idx_type=residual.linear_idx_type], post_weight: TileTensor[dtype, Engine=post_weight.Engine, address_space=post_weight.address_space, linear_idx_type=post_weight.linear_idx_type], gate: TileTensor[gate_dtype, Engine=gate.Engine, address_space=gate.address_space, linear_idx_type=gate.linear_idx_type], norm_weight: TileTensor[dtype, Engine=norm_weight.Engine, address_space=norm_weight.address_space, linear_idx_type=norm_weight.linear_idx_type], stream_out: TileTensor[dtype, Engine=stream_out.Engine, address_space=stream_out.address_space, linear_idx_type=stream_out.linear_idx_type], normed_out: TileTensor[dtype, Engine=normed_out.Engine, address_space=normed_out.address_space, linear_idx_type=normed_out.linear_idx_type], post_eps: Float32, norm_eps: Float32, temperature: Float32, gate_eps: Float32, rank_sigs: Array[Pointer[Signal, MutAnyOrigin], ngpus], ctx: DeviceContext, my_rank: Int)
All-gathers a branch's column shards and joins it to the residual stream.
Per gathered row b, as Muse Spark's unfused residual boundary computes
it:
n = rms_norm(b * (1 + post_weight)) # weightless norm
stream = alpha * residual + beta * n # fp32, rounded once
normed = rms_norm(stream) * (1 + norm_weight)with beta = sigmoid(gate / T) and
alpha = sqrt(max(sigmoid(-gate / T) * (1 + beta), gate_eps)). Each step
rounds to dtype where the graph does; the row sums reduce in a different
order than rms_norm's, so the outputs agree to its rounding.
Parameters:
- dtype (
DType): Element type of the activations and norm weights. - gate_dtype (
DType): Element type of the gate's logits. - ngpus (
Int): Number of devices, all in one group. - in_layout (
TensorLayout): Layout of the branch shards. - in_origin (
Origin[mut=mut]): Origin of the branch shards. - scale_branch (
Bool): Whether the branch is scaled by1 + post_weightbefore its norm; otherwisepost_weightis not read.
Args:
- input_buffers (
Array[TileTensor[dtype, in_layout, in_origin], ngpus]): Every device's[rows, cols / ngpus]branch shard, holding a contiguous block of columns in rank order. - residual (
TileTensor[dtype, Engine=residual.Engine, address_space=residual.address_space, linear_idx_type=residual.linear_idx_type]): This device's residual stream[rows, cols]. - post_weight (
TileTensor[dtype, Engine=post_weight.Engine, address_space=post_weight.address_space, linear_idx_type=post_weight.linear_idx_type]): The branch's input scale weight[cols]. - gate (
TileTensor[gate_dtype, Engine=gate.Engine, address_space=gate.address_space, linear_idx_type=gate.linear_idx_type]): The residual gate's logits[cols]. - norm_weight (
TileTensor[dtype, Engine=norm_weight.Engine, address_space=norm_weight.address_space, linear_idx_type=norm_weight.linear_idx_type]): The next pre-norm's weight[cols]. - stream_out (
TileTensor[dtype, Engine=stream_out.Engine, address_space=stream_out.address_space, linear_idx_type=stream_out.linear_idx_type]): The new residual stream[rows, cols]. - normed_out (
TileTensor[dtype, Engine=normed_out.Engine, address_space=normed_out.address_space, linear_idx_type=normed_out.linear_idx_type]): Its next pre-norm[rows, cols]. - post_eps (
Float32): The branch norm's epsilon. - norm_eps (
Float32): The next pre-norm's epsilon. - temperature (
Float32): The gate's temperature. - gate_eps (
Float32): The floor under the residual coefficient's root. - rank_sigs (
Array[Pointer[Signal, MutAnyOrigin], ngpus]): Every device's Signal pointer, by rank. - ctx (
DeviceContext): Device context for this GPU. - my_rank (
Int): This device's rank.