IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

BundledAllReduceSum

struct BundledAllReduceSum

Registers the mo.bundled.allreduce.sum graph op with the graph compiler.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], //, _trace_name: StaticString, group_size: Int = Int(0), peer_rank: Optional[Int] = None](output: T, inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], ctx: DeviceContext)

Per-device allreduce sum, for use with mo.parallel dispatch.

This path does not take the kernel-load gate: mo.parallel emits N independent async stubs, with no single point to claim or release it, so a first-time kernel load can still interleave with these launches.

Parameters:

  • ​Ins (TypeList[values]): The element types of inputs (inferred).
  • ​Sigs (TypeList[values]): The element types of signal_buffers (inferred).
  • ​_trace_name (StaticString): Trace name used for profiling and debugging.
  • ​group_size (Int): Number of contiguous devices per independent allreduce group; 0 means one full-world group.
  • ​peer_rank (Optional[Int]): THIS GPU's index in the world, so in inputs and signal_buffers; None takes it from the device id.

Args:

  • ​output (T): Output tensor for THIS GPU.
  • ​inputs (TensorTuple[*Ins.values]): Input tensors from ALL GPUs in the world.
  • ​signal_buffers (TensorTuple[*Sigs.values]): Signal buffers for ALL GPUs in the world.
  • ​ctx (DeviceContext): Device context for THIS GPU.

build_graph​

static def build_graph[Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], //, _trace_name: StaticString, group_size: Int = Int(0), peer_rank: Optional[Int] = None](output: T, inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], builder: LegacyDeviceGraphBuilder)

Records the per-device allreduce sum into a device graph.

The peers arrive as stable address slots: the recorded kernels read each peer's data pointer out of its slot at replay time, so the graph does not bake in the peer buffers themselves.

Parameters:

  • ​Ins (TypeList[values]): The element types of inputs (inferred).
  • ​Sigs (TypeList[values]): The element types of signal_buffers (inferred).
  • ​_trace_name (StaticString): Trace name used for profiling and debugging.
  • ​group_size (Int): Number of contiguous devices per independent allreduce group; 0 means one full-world group.
  • ​peer_rank (Optional[Int]): THIS GPU's index in the world, so in inputs and signal_buffers; None takes it from the device id.

Args:

  • ​output (T): Output tensor for THIS GPU.
  • ​inputs (TensorTuple[*Ins.values]): Slot-backed input tensors from ALL GPUs in the world.
  • ​signal_buffers (TensorTuple[*Sigs.values]): Signal buffers for ALL GPUs in the world.
  • ​builder (LegacyDeviceGraphBuilder): Device graph builder for THIS GPU.

Was this page helpful?