IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

DistributedReduceScatterSum

struct DistributedReduceScatterSum

Registers the mo.distributed.reducescatter.sum graph op with the graph compiler.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[Outs: TypeList[Outs.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], //, _trace_name: StaticString, axis: Int = Int(0), group_size: Int = Int(0)](outputs: TensorTuple[*Outs.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], dev_ctxs_input: DeviceContextArray)

Distributed reduce-scatter operation implementation for sum reduction.

Limitations: - Maximum of 8 GPUs supported (matches MAX_GPUS in comm/sync.mojo) - Tensor element count must be multiple of SIMD width - Requires identical tensor shapes within each reduce-scatter group

Parameters:

  • ​Outs (TypeList[values]): The element types of outputs (inferred).
  • ​Ins (TypeList[values]): The element types of inputs (inferred).
  • ​Sigs (TypeList[values]): The element types of signal_buffers (inferred).
  • ​_trace_name (StaticString): Trace name for profiling.
  • ​axis (Int): Axis along which to scatter the reduced result (defaults to 0).
  • ​group_size (Int): Number of devices per reduce-scatter group; must be at least 1 and must evenly divide the total number of devices (defaults to 0).

Args:

  • ​outputs (TensorTuple[*Outs.values]): Output tensors (one per GPU) to store scattered reduced results.
  • ​inputs (TensorTuple[*Ins.values]): Input tensors (one per GPU) containing values to reduce.
  • ​signal_buffers (TensorTuple[*Sigs.values]): Preallocated synchronization buffers for cross-GPU coordination.
  • ​dev_ctxs_input (DeviceContextArray): Device contexts for participating GPUs.

Was this page helpful?