For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).
Mojo struct
DistributedReduceScatterRMSNorm
struct DistributedReduceScatterRMSNorm
Registers the mo.composite.distributed.reduce_scatter_rms_norm graph op with the graph compiler.
Implemented traits
Methods
execute
static def execute[dtype: DType, rank: Int, target: StringSlice[ImmStaticOrigin], _trace_name: StringSlice[ImmStaticOrigin]](outputs_normed: VariadicTensors[IOSpec[_, _].Output, static_specs=outputs_normed.static_specs], outputs_sum: VariadicTensors[IOSpec[_, _].Output, static_specs=outputs_sum.static_specs], inputs: VariadicTensors[IOSpec[_, _].Input, static_specs=inputs.static_specs], signal_buffers: VariadicTensors[IOSpec[_, _].MutableInput, static_specs=signal_buffers.static_specs], gammas: VariadicTensors[IOSpec[_, _].Input, static_specs=gammas.static_specs], epsilons: VariadicTensors[IOSpec[_, _].Input, static_specs=epsilons.static_specs], weight_offsets: VariadicTensors[IOSpec[_, _].Input, static_specs=weight_offsets.static_specs], dev_ctxs_input: DeviceContextArray)
Fused reduce-scatter sum + RMSNorm (bf16 in/out, no quantization).
Reduce-scatters inputs (one [rows, cols] tensor per device) along
rows and RMSNorm-normalizes each owned shard in the same launch, writing
the normed shard to outputs_normed and the reduce-scatter sum shard
(the residual stream) to outputs_sum.
Limitations: - Maximum of 8 GPUs supported (matches MAX_GPUS in comm/sync.mojo). - Full-world reduce-scatter only (no device grouping); requires P2P.
Parameters:
- dtype (
DType): Element type of the input/output tensors. - rank (
Int): Tensor rank of the inputs and outputs. - target (
StringSlice[ImmStaticOrigin]): Target device string for tracing. - _trace_name (
StringSlice[ImmStaticOrigin]): Trace name for profiling.
Args:
- outputs_normed (
VariadicTensors[IOSpec[_, _].Output, static_specs=outputs_normed.static_specs]): Per-device normed output shards. - outputs_sum (
VariadicTensors[IOSpec[_, _].Output, static_specs=outputs_sum.static_specs]): Per-device reduce-scatter sum shards (residual stream). - inputs (
VariadicTensors[IOSpec[_, _].Input, static_specs=inputs.static_specs]): Per-device input tensors to reduce and scatter. - signal_buffers (
VariadicTensors[IOSpec[_, _].MutableInput, static_specs=signal_buffers.static_specs]): Per-device synchronization buffers. - gammas (
VariadicTensors[IOSpec[_, _].Input, static_specs=gammas.static_specs]): Per-device RMSNorm gamma weights (in_dtype, length cols). - epsilons (
VariadicTensors[IOSpec[_, _].Input, static_specs=epsilons.static_specs]): Per-device RMSNorm epsilon scalars (float32). - weight_offsets (
VariadicTensors[IOSpec[_, _].Input, static_specs=weight_offsets.static_specs]): Per-device gamma offset scalars (in_dtype). - dev_ctxs_input (
DeviceContextArray): Device contexts for participating GPUs.
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!