IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

DistributedAllToAll

struct DistributedAllToAll

Registers the mo.distributed.all_to_all graph op with the graph compiler.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[Outs: TypeList[Outs.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], //, _trace_name: StaticString, split_axis: Int, concat_axis: Int, group_size: Int](outputs: TensorTuple[*Outs.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], dev_ctxs_input: DeviceContextArray)

Distributed all-to-all implementation.

Each device splits its input into group_size chunks along split_axis, sends chunk j to its group's rank j, and concatenates the chunks it receives in source rank order along concat_axis.

Parameters:

  • ​Outs (TypeList[values]): The element types of outputs (inferred).
  • ​Ins (TypeList[values]): The element types of inputs (inferred).
  • ​Sigs (TypeList[values]): The element types of signal_buffers (inferred).
  • ​_trace_name (StaticString): Trace name for profiling.
  • ​split_axis (Int): The input axis split into one chunk per peer.
  • ​concat_axis (Int): The output axis the received chunks concatenate on.
  • ​group_size (Int): Number of contiguous devices per all-to-all group; must be at least 2 and evenly divide the number of devices.

Args:

Was this page helpful?