IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

DistributedAllReduceRMSNorm

struct DistributedAllReduceRMSNorm

Registers the mo.composite.distributed.allreduce_rms_norm graph op with the graph compiler.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[Normed: TypeList[Normed.values], Summed: TypeList[Summed.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], Gammas: TypeList[Gammas.values], //, _trace_name: StaticString, epsilon: Float32, weight_offset: Float32, multiply_before_cast: Bool = False](outputs_normed: TensorTuple[*Normed.values], outputs_summed: TensorTuple[*Summed.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], gammas: TensorTuple[*Gammas.values], dev_ctxs_input: DeviceContextArray)

Fused allreduce sum + RMSNorm (bf16 or float32, no quantization).

Sum-reduces inputs (one [rows, cols] tensor per device) across devices and RMSNorm-normalizes the exact f32 peer sum in the same launch, writing the normed rows to outputs_normed. Above the fuse threshold, on non-AMD targets and without P2P access, the dispatch falls back to a standalone allreduce into outputs_summed plus a separate norm; only that arm writes outputs_summed.

Limitations: - The fused arm requires P2P access between the participating GPUs; without it the op runs the standalone allreduce + norm, whose allreduce falls back to staged peer copies. - The kernel stages the 2-stage path's scratch behind each Signal header; a short buffer routes to the 1-stage kernel.

Parameters:

  • ​Normed (TypeList[values]): The element types of outputs_normed (inferred).
  • ​Summed (TypeList[values]): The element types of outputs_summed (inferred).
  • ​Ins (TypeList[values]): The element types of inputs (inferred).
  • ​Sigs (TypeList[values]): The element types of signal_buffers (inferred).
  • ​Gammas (TypeList[values]): The element types of gammas (inferred).
  • ​_trace_name (StaticString): Trace name for profiling.
  • ​epsilon (Float32): RMSNorm epsilon, shared by every device.
  • ​weight_offset (Float32): Gamma offset, shared by every device.
  • ​multiply_before_cast (Bool): The norm's gamma cast order, matching ops.rms_norm.

Args:

Was this page helpful?