IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

alltoall

Multi-GPU all-to-all over peer-to-peer reads.

Every device of a group splits its input into group_size equal chunks along a split axis, sends chunk j to the group's rank j, and concatenates the chunks it receives, in source rank order, along a concat axis. It is torch.distributed.all_to_all_single generalized to any pair of axes, and the transpose of a sharding from one tensor axis to another.

Each device pulls its chunks straight out of its peers' inputs, the way the P2P allgather does, between a start and an end barrier on the shared Signal buffers. A chunk is never contiguous in general: viewed against the input it is outer_s runs of in_inner elements, one every group_size * in_inner, and viewed against the output it is outer_c runs of out_inner elements, again one every group_size * out_inner. Both views number the chunk's elements in the same row-major order, so one flat chunk index addresses both sides of every copy.

Functions​

  • ​all_to_all: Per-device all-to-all: one instance per GPU fills its own output.

Was this page helpful?