For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
alltoall
Multi-GPU all-to-all over peer-to-peer reads.
Every device of a group splits its input into group_size equal chunks along
a split axis, sends chunk j to the group's rank j, and concatenates the
chunks it receives, in source rank order, along a concat axis. It is
torch.distributed.all_to_all_single generalized to any pair of axes, and
the transpose of a sharding from one tensor axis to another.
Each device pulls its chunks straight out of its peers' inputs, the way the
P2P allgather does, between a start and an end barrier on the shared
Signal buffers. A chunk is never contiguous in general: viewed against the
input it is outer_s runs of in_inner elements, one every
group_size * in_inner, and viewed against the output it is outer_c runs
of out_inner elements, again one every group_size * out_inner. Both views
number the chunk's elements in the same row-major order, so one flat chunk
index addresses both sides of every copy.
Functions
-
all_to_all: Per-device all-to-all: one instance per GPU fills its own output.