IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

BundledAllReduceSum

struct BundledAllReduceSum

Registers the mo.bundled.allreduce.sum graph op with the graph compiler.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[dtype: DType, rank: Int, target: StringSpan[ImmStaticOrigin], _trace_name: StringSpan[ImmStaticOrigin]](output: ManagedTensorSlice[IOSpec[_, _].FusedOutput, static_spec=output.static_spec], inputs: VariadicTensors[IOSpec[_, _].Input, static_specs=inputs.static_specs], signal_buffers: VariadicTensors[IOSpec[_, _].MutableInput, static_specs=signal_buffers.static_specs], ctx: DeviceContextBase)

Per-device allreduce sum, for use with mo.parallel dispatch.

This path does not take the kernel-load gate: mo.parallel emits N independent async stubs, with no single point to claim or release it, so a first-time kernel load can still interleave with these launches.

Parameters:

  • ​dtype (DType): Element type of the input and output tensors.
  • ​rank (Int): Number of dimensions in the input and output tensors.
  • ​target (StringSpan[ImmStaticOrigin]): Target device identifier for code generation.
  • ​_trace_name (StringSpan[ImmStaticOrigin]): Trace name used for profiling and debugging.

Args:

  • ​output (ManagedTensorSlice[IOSpec[_, _].FusedOutput, static_spec=output.static_spec]): Output tensor for THIS GPU.
  • ​inputs (VariadicTensors[IOSpec[_, _].Input, static_specs=inputs.static_specs]): Input tensors from ALL participating GPUs.
  • ​signal_buffers (VariadicTensors[IOSpec[_, _].MutableInput, static_specs=signal_buffers.static_specs]): Signal buffers for ALL participating GPUs.
  • ​ctx (DeviceContextBase): Device context for THIS GPU.

build_graph​

static def build_graph[dtype: DType, rank: Int, target: StringSpan[ImmStaticOrigin], _trace_name: StringSpan[ImmStaticOrigin]](output: ManagedTensorSlice[IOSpec[_, _].FusedOutput, static_spec=output.static_spec], inputs: StableVariadicTensors[IOSpec[_, _].Input, static_specs=inputs.static_specs], signal_buffers: VariadicTensors[IOSpec[_, _].MutableInput, static_specs=signal_buffers.static_specs], builder: DeviceGraphBuilder)

Records the per-device allreduce sum into a device graph.

The peers arrive as stable address slots: the recorded kernels read each peer's data pointer out of its slot at replay time, so the graph does not bake in the peer buffers themselves.

Parameters:

  • ​dtype (DType): Element type of the input and output tensors.
  • ​rank (Int): Number of dimensions in the input and output tensors.
  • ​target (StringSpan[ImmStaticOrigin]): Target device identifier for code generation.
  • ​_trace_name (StringSpan[ImmStaticOrigin]): Trace name used for profiling and debugging.

Args:

  • ​output (ManagedTensorSlice[IOSpec[_, _].FusedOutput, static_spec=output.static_spec]): Output tensor for THIS GPU.
  • ​inputs (StableVariadicTensors[IOSpec[_, _].Input, static_specs=inputs.static_specs]): Slot-backed input tensors from ALL participating GPUs.
  • ​signal_buffers (VariadicTensors[IOSpec[_, _].MutableInput, static_specs=signal_buffers.static_specs]): Signal buffers for ALL participating GPUs.
  • ​builder (DeviceGraphBuilder): Device graph builder for THIS GPU.

Was this page helpful?