For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
all_to_all
def all_to_all[dtype: DType, ngpus: Int, *, group_size: Int = ngpus](input_ptrs: Array[Pointer[Scalar[dtype], ImmutAnyOrigin], ngpus], output: MutPointer[Scalar[dtype], MutAnyOrigin], num_elements: Int, in_inner: Int, out_inner: Int, rank_sigs: Array[Pointer[Signal, MutAnyOrigin], ngpus], ctx: DeviceContext, my_rank: Int, _max_num_blocks: Optional[Int] = None)
Per-device all-to-all: one instance per GPU fills its own output.
Rank r of a group splits its input into group_size chunks and sends
chunk j to the group's rank j; each rank's output is the chunks it
receives, from source rank 0 to group_size - 1. In tensor terms,
with the input split along axis a and the output concatenated along
axis b:
in_inneris the product of the input's dims fromaon, divided bygroup_size: one chunk's contiguous run in the input.out_inneris the product of the output's dims frombon, divided bygroup_size: one chunk's contiguous run in the output.
in_inner == out_inner == num_elements / group_size is the plain
all_to_all_single over a leading axis. The caller launches one
instance per device in parallel (e.g. via _launch_device_collective).
World view vs. group: ngpus is the total number of devices;
input_ptrs and rank_sigs carry every device's, indexed by global
rank, and my_rank is this device's global rank. Devices
[g * group_size, (g + 1) * group_size) form group g. Every input of
a group holds num_elements elements.
Parameters:
- dtype (
DType): Data type of the tensor elements. - ngpus (
Int): Total number of devices in the world. - group_size (
Int): Number of devices per independent all-to-all group. Must evenly dividengpus. Defaults tongpus.
Args:
- input_ptrs (
Array[Pointer[Scalar[dtype], ImmutAnyOrigin], ngpus]): Every device's input, indexed by global rank. - output (
MutPointer[Scalar[dtype], MutAnyOrigin]): This device's output,num_elementselements. - num_elements (
Int): Elements in every input and output of this group. - in_inner (
Int): One chunk's contiguous run in the input, in elements. - out_inner (
Int): One chunk's contiguous run in the output, in elements. - rank_sigs (
Array[Pointer[Signal, MutAnyOrigin], ngpus]): Every device's Signal pointer, indexed by global rank. - ctx (
DeviceContext): Device context for this GPU. - my_rank (
Int): Global rank of this GPU in[0, ngpus). - _max_num_blocks (
Optional[Int]): Maximum number of blocks for the kernel launch.
Raises:
Error: P2P access is not available, or the chunk geometry does not
divide num_elements.