IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

BundledAllGather

struct BundledAllGather

Registers the mo.bundled.allgather graph op with the graph compiler.

The per-device form of mo.distributed.allgather, full world only: one instance per GPU gathers every peer's tensor into its own outputs, one per peer in peer order. execute reads the peers through live data pointers; build_graph reads them through stable address slots (Stable inputs).

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[Outs: TypeList[Outs.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], //, _trace_name: StaticString, peer_rank: Optional[Int] = None](outputs: TensorTuple[*Outs.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], ctx: DeviceContext)

Per-device allgather through live addresses.

Parameters:

  • ​Outs (TypeList[values]): The element types of outputs (inferred).
  • ​Ins (TypeList[values]): The element types of inputs (inferred).
  • ​Sigs (TypeList[values]): The element types of signal_buffers (inferred).
  • ​_trace_name (StaticString): Trace name used for profiling and debugging.
  • ​peer_rank (Optional[Int]): THIS GPU's index in inputs and signal_buffers; None takes it from the device id.

Args:

build_graph​

static def build_graph[Outs: TypeList[Outs.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], //, _trace_name: StaticString, peer_rank: Optional[Int] = None](outputs: TensorTuple[*Outs.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], builder: LegacyDeviceGraphBuilder)

Records the per-device allgather into a device graph.

Parameters:

  • ​Outs (TypeList[values]): The element types of outputs (inferred).
  • ​Ins (TypeList[values]): The element types of inputs (inferred).
  • ​Sigs (TypeList[values]): The element types of signal_buffers (inferred).
  • ​_trace_name (StaticString): Trace name used for profiling and debugging.
  • ​peer_rank (Optional[Int]): THIS GPU's index in inputs and signal_buffers; None takes it from the device id.

Args:

Was this page helpful?