For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
DistributedAllToAll
struct DistributedAllToAll
Registers the mo.distributed.all_to_all graph op with the graph compiler.
Implemented traits
Methods
execute
static def execute[Outs: TypeList[Outs.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], //, _trace_name: StaticString, split_axis: Int, concat_axis: Int, group_size: Int](outputs: TensorTuple[*Outs.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], dev_ctxs_input: DeviceContextArray)
Distributed all-to-all implementation.
Each device splits its input into group_size chunks along
split_axis, sends chunk j to its group's rank j, and
concatenates the chunks it receives in source rank order along
concat_axis.
Parameters:
- Outs (
TypeList[values]): The element types ofoutputs(inferred). - Ins (
TypeList[values]): The element types ofinputs(inferred). - Sigs (
TypeList[values]): The element types ofsignal_buffers(inferred). - _trace_name (
StaticString): Trace name for profiling. - split_axis (
Int): The input axis split into one chunk per peer. - concat_axis (
Int): The output axis the received chunks concatenate on. - group_size (
Int): Number of contiguous devices per all-to-all group; must be at least 2 and evenly divide the number of devices.
Args:
- outputs (
TensorTuple[*Outs.values]): Output tensors, one per GPU. - inputs (
TensorTuple[*Ins.values]): Input tensors, one per GPU; same shape within a group. - signal_buffers (
TensorTuple[*Sigs.values]): Preallocated synchronization buffers. - dev_ctxs_input (
DeviceContextArray): Device contexts for participating GPUs.