IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

DistributedReduceScatterRMSNorm

struct DistributedReduceScatterRMSNorm

Registers the mo.composite.distributed.reduce_scatter_rms_norm graph op with the graph compiler.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[Normed: TypeList[Normed.values], Sums: TypeList[Sums.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], Gammas: TypeList[Gammas.values], Residuals: TypeList[Residuals.values], //, _trace_name: StaticString, epsilon: Float32, weight_offset: Float32, group_size: Int = Int(0), has_residual: Bool = False, quant_bits: Int = Int(0)](outputs_normed: TensorTuple[*Normed.values], outputs_sum: TensorTuple[*Sums.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], gammas: TensorTuple[*Gammas.values], residuals: TensorTuple[*Residuals.values], dev_ctxs_input: DeviceContextArray)

Fused reduce-scatter sum + RMSNorm + residual add (bf16 out).

Reduce-scatters inputs (one [rows, cols] tensor per device) along rows, adds residuals, and RMSNorm-normalizes each owned shard in the same launch, writing the normed shard to outputs_normed and the summed shard (the residual stream) to outputs_sum.

Under has_residual, residuals carries the TP-replicated residual stream. Each device adds only its own row shard of it, which is why such callers must NOT pre-add it on the group leader: the reduce-scatter sums across ranks, so a leader-side add lands once for the whole group and this per-rank add reproduces it exactly -- but only because the residual is bit-identical on every rank of the group. Folding it here deletes a full-width elementwise add that ran on the group leader alone.

Without has_residual the operands are still passed (the op's variadic groups must match in size) but never read, and both arms are exactly the reduce-scatter + norm this op was before the fold existed.

Limitations: - Maximum of 8 GPUs supported (matches MAX_GPUS in comm/sync.mojo). - Requires P2P, and identical tensor shapes within each group.

Parameters:

  • ​Normed (TypeList[values]): The element types of outputs_normed (inferred).
  • ​Sums (TypeList[values]): The element types of outputs_sum (inferred).
  • ​Ins (TypeList[values]): The element types of inputs (inferred).
  • ​Sigs (TypeList[values]): The element types of signal_buffers (inferred).
  • ​Gammas (TypeList[values]): The element types of gammas (inferred).
  • ​Residuals (TypeList[values]): The element types of residuals (inferred).
  • ​_trace_name (StaticString): Trace name for profiling.
  • ​epsilon (Float32): RMSNorm epsilon, shared by every device.
  • ​weight_offset (Float32): Gamma offset, shared by every device.
  • ​group_size (Int): Number of contiguous devices per independent reduce-scatter group; must be at least 2 and must evenly divide the total number of devices. Equal to num_devices for a full-world collective. The builder always sets it; the 0 attribute default is not a usable value.
  • ​has_residual (Bool): Fold residuals into the reduce-scatter sum. Off leaves both arms byte-for-byte the plain reduce-scatter + norm.
  • ​quant_bits (Int): 4 or 8 to send the two-launch arm's peer traffic quantized (INT4 or MXFP8, lossy) when the whole world reduces together, the columns are a multiple of 32 and the signal buffers hold the scratch; 0 keeps it exact.

Args:

Was this page helpful?