IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

DistributedAllGatherRMSNorm

struct DistributedAllGatherRMSNorm

Registers the mo.composite.distributed.allgather_rms_norm graph op with the graph compiler.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[Normed: TypeList[Normed.values], Residuals: TypeList[Residuals.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], Gammas: TypeList[Gammas.values], //, _trace_name: StaticString, epsilon: Float32, weight_offset: Float32, group_size: Int = Int(0), gather_axis: Int = Int(0), round_offset_gamma: Bool = False](outputs_normed: TensorTuple[*Normed.values], outputs_residual: TensorTuple[*Residuals.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], gammas: TensorTuple[*Gammas.values], dev_ctxs_input: DeviceContextArray)

Fused all-gather + RMSNorm (bf16 in/out, no quantization).

All-gathers inputs (per-device [shard_i, cols] row-shards) into the full replicated [rows, cols] stream and RMSNorms every gathered row in one launch: normed to outputs_normed, gathered residual to outputs_residual (both full and replicated).

Limitations: - Maximum of 8 GPUs supported (matches MAX_GPUS in comm/sync.mojo). - Requires P2P; within a group, shard shapes may differ only in the gathered axis.

Parameters:

  • ​Normed (TypeList[values]): The element types of outputs_normed (inferred).
  • ​Residuals (TypeList[values]): The element types of outputs_residual (inferred).
  • ​Ins (TypeList[values]): The element types of inputs (inferred).
  • ​Sigs (TypeList[values]): The element types of signal_buffers (inferred).
  • ​Gammas (TypeList[values]): The element types of gammas (inferred).
  • ​_trace_name (StaticString): Trace name for profiling.
  • ​epsilon (Float32): RMSNorm epsilon, shared by every device.
  • ​weight_offset (Float32): Gamma offset, shared by every device.
  • ​group_size (Int): Number of contiguous devices per independent all-gather group; must be at least 2 and must evenly divide the total number of devices. Equal to num_devices for a full-world collective. The builder always sets it; the 0 attribute default is not a usable value.
  • ​gather_axis (Int): 0 to gather row shards, 1 column shards.
  • ​round_offset_gamma (Bool): Whether gamma + weight_offset is rounded to dtype before the f32 multiply.

Args:

Was this page helpful?