For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
DistributedAllReduceRMSNorm
struct DistributedAllReduceRMSNorm
Registers the mo.composite.distributed.allreduce_rms_norm graph op with the graph compiler.
Implemented traits
Methods
execute
static def execute[Normed: TypeList[Normed.values], Summed: TypeList[Summed.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], Gammas: TypeList[Gammas.values], //, _trace_name: StaticString, epsilon: Float32, weight_offset: Float32, multiply_before_cast: Bool = False](outputs_normed: TensorTuple[*Normed.values], outputs_summed: TensorTuple[*Summed.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], gammas: TensorTuple[*Gammas.values], dev_ctxs_input: DeviceContextArray)
Fused allreduce sum + RMSNorm (bf16 or float32, no quantization).
Sum-reduces inputs (one [rows, cols] tensor per device) across
devices and RMSNorm-normalizes the exact f32 peer sum in the same
launch, writing the normed rows to outputs_normed. Above the fuse
threshold, on non-AMD targets and without P2P access, the dispatch
falls back to a standalone allreduce into outputs_summed plus a
separate norm; only that arm writes outputs_summed.
Limitations:
- The fused arm requires P2P access between the participating
GPUs; without it the op runs the standalone allreduce + norm,
whose allreduce falls back to staged peer copies.
- The kernel stages the 2-stage path's scratch behind each
Signal header; a short buffer routes to the 1-stage kernel.
Parameters:
- Normed (
TypeList[values]): The element types ofoutputs_normed(inferred). - Summed (
TypeList[values]): The element types ofoutputs_summed(inferred). - Ins (
TypeList[values]): The element types ofinputs(inferred). - Sigs (
TypeList[values]): The element types ofsignal_buffers(inferred). - Gammas (
TypeList[values]): The element types ofgammas(inferred). - _trace_name (
StaticString): Trace name for profiling. - epsilon (
Float32): RMSNorm epsilon, shared by every device. - weight_offset (
Float32): Gamma offset, shared by every device. - multiply_before_cast (
Bool): The norm's gamma cast order, matchingops.rms_norm.
Args:
- outputs_normed (
TensorTuple[*Normed.values]): Per-device normed output (same shape asinputs). - outputs_summed (
TensorTuple[*Summed.values]): Per-device raw reduced sum, written by the two-launch arm only. - inputs (
TensorTuple[*Ins.values]): Per-device inputs to reduce. - signal_buffers (
TensorTuple[*Sigs.values]): Per-device synchronization buffers. - gammas (
TensorTuple[*Gammas.values]): Per-device RMSNorm gamma weights (length cols), in the input dtype or, on a float32 input, bfloat16/float16: the kernels widen it exactly in registers. - dev_ctxs_input (
DeviceContextArray): Device contexts for participating GPUs.