For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
DistributedReduceScatterRMSNorm
struct DistributedReduceScatterRMSNorm
Registers the mo.composite.distributed.reduce_scatter_rms_norm graph op with the graph compiler.
Implemented traits
Methods
execute
static def execute[Normed: TypeList[Normed.values], Sums: TypeList[Sums.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], Gammas: TypeList[Gammas.values], Residuals: TypeList[Residuals.values], //, _trace_name: StaticString, epsilon: Float32, weight_offset: Float32, group_size: Int = Int(0), has_residual: Bool = False, quant_bits: Int = Int(0)](outputs_normed: TensorTuple[*Normed.values], outputs_sum: TensorTuple[*Sums.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], gammas: TensorTuple[*Gammas.values], residuals: TensorTuple[*Residuals.values], dev_ctxs_input: DeviceContextArray)
Fused reduce-scatter sum + RMSNorm + residual add (bf16 out).
Reduce-scatters inputs (one [rows, cols] tensor per device) along
rows, adds residuals, and RMSNorm-normalizes each owned shard in the
same launch, writing the normed shard to outputs_normed and the summed
shard (the residual stream) to outputs_sum.
Under has_residual, residuals carries the TP-replicated residual
stream. Each device adds only its own row shard of it, which is why
such callers must NOT pre-add it on the group leader: the
reduce-scatter sums across ranks, so a leader-side add lands once for
the whole group and this per-rank add reproduces it exactly -- but only
because the residual is bit-identical on every rank of the group.
Folding it here deletes a full-width elementwise add that ran on the
group leader alone.
Without has_residual the operands are still passed (the op's variadic
groups must match in size) but never read, and both arms are exactly
the reduce-scatter + norm this op was before the fold existed.
Limitations: - Maximum of 8 GPUs supported (matches MAX_GPUS in comm/sync.mojo). - Requires P2P, and identical tensor shapes within each group.
Parameters:
- Normed (
TypeList[values]): The element types ofoutputs_normed(inferred). - Sums (
TypeList[values]): The element types ofoutputs_sum(inferred). - Ins (
TypeList[values]): The element types ofinputs(inferred). - Sigs (
TypeList[values]): The element types ofsignal_buffers(inferred). - Gammas (
TypeList[values]): The element types ofgammas(inferred). - Residuals (
TypeList[values]): The element types ofresiduals(inferred). - _trace_name (
StaticString): Trace name for profiling. - epsilon (
Float32): RMSNorm epsilon, shared by every device. - weight_offset (
Float32): Gamma offset, shared by every device. - group_size (
Int): Number of contiguous devices per independent reduce-scatter group; must be at least 2 and must evenly divide the total number of devices. Equal tonum_devicesfor a full-world collective. The builder always sets it; the0attribute default is not a usable value. - has_residual (
Bool): Foldresidualsinto the reduce-scatter sum. Off leaves both arms byte-for-byte the plain reduce-scatter + norm. - quant_bits (
Int): 4 or 8 to send the two-launch arm's peer traffic quantized (INT4 or MXFP8, lossy) when the whole world reduces together, the columns are a multiple of 32 and the signal buffers hold the scratch; 0 keeps it exact.
Args:
- outputs_normed (
TensorTuple[*Normed.values]): Per-device normed output shards. - outputs_sum (
TensorTuple[*Sums.values]): Per-device reduce-scatter sum shards (residual stream). - inputs (
TensorTuple[*Ins.values]): Per-device input tensors to reduce and scatter. - signal_buffers (
TensorTuple[*Sigs.values]): Per-device synchronization buffers. - gammas (
TensorTuple[*Gammas.values]): Per-device RMSNorm gamma weights (in_dtype, length cols). - residuals (
TensorTuple[*Residuals.values]): Per-device residual stream, same shape asinputsand bit-identical across each group. Read only underhas_residual. - dev_ctxs_input (
DeviceContextArray): Device contexts for participating GPUs.