IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

all_to_all

def all_to_all[dtype: DType, ngpus: Int, *, group_size: Int = ngpus](input_ptrs: Array[Pointer[Scalar[dtype], ImmutAnyOrigin], ngpus], output: MutPointer[Scalar[dtype], MutAnyOrigin], num_elements: Int, in_inner: Int, out_inner: Int, rank_sigs: Array[Pointer[Signal, MutAnyOrigin], ngpus], ctx: DeviceContext, my_rank: Int, _max_num_blocks: Optional[Int] = None)

Per-device all-to-all: one instance per GPU fills its own output.

Rank r of a group splits its input into group_size chunks and sends chunk j to the group's rank j; each rank's output is the chunks it receives, from source rank 0 to group_size - 1. In tensor terms, with the input split along axis a and the output concatenated along axis b:

  • in_inner is the product of the input's dims from a on, divided by group_size: one chunk's contiguous run in the input.
  • out_inner is the product of the output's dims from b on, divided by group_size: one chunk's contiguous run in the output.

in_inner == out_inner == num_elements / group_size is the plain all_to_all_single over a leading axis. The caller launches one instance per device in parallel (e.g. via _launch_device_collective).

World view vs. group: ngpus is the total number of devices; input_ptrs and rank_sigs carry every device's, indexed by global rank, and my_rank is this device's global rank. Devices [g * group_size, (g + 1) * group_size) form group g. Every input of a group holds num_elements elements.

Parameters:

  • ​dtype (DType): Data type of the tensor elements.
  • ​ngpus (Int): Total number of devices in the world.
  • ​group_size (Int): Number of devices per independent all-to-all group. Must evenly divide ngpus. Defaults to ngpus.

Args:

Raises:

Error: P2P access is not available, or the chunk geometry does not divide num_elements.

Was this page helpful?