For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
allreduce
Multi-GPU allreduce implementation for efficient tensor reduction across GPUs.
This module provides an optimized implementation of allreduce operations across multiple GPUs, supporting both peer-to-peer (P2P) and non-P2P communication patterns. The implementation automatically selects between two approaches based on hardware capabilities:
-
P2P-based implementation (when P2P access is available):
- Uses direct GPU-to-GPU memory access for better performance
- Implements both single-stage and two-stage algorithms:
- Single-stage for latency-bound transfers (small tensors)
- Two-stage (reduce-scatter + all-gather) for bandwidth-bound transfers (large tensors)
- Optimized for NVLink bandwidth utilization
- Uses vectorized memory access and higher precision accumulation
-
Non-P2P fallback implementation:
- Copies data through host memory when direct GPU access isn't possible
- Simple but functional approach for systems without P2P support
The implementation is tuned for common GPU architectures (A100, H100) and includes parameters that can be adjusted for different hardware configurations.
Per-Device Architecture
The allreduce operation follows a per-device execution model:
-
Single-Device Instances: Each GPU runs its own instance of the allreduce operation.
-
Parallel Execution: The Python/Graph API layer is responsible for:
- Creating one allreduce op instance per participating GPU.
- Ensuring all instances execute in parallel.
- Ensuring correctness by staging mo.fence.
-
Device Affinity: Each allreduce instance:
- Executes on its assigned GPU (specified via device context).
- Reads from all GPUs' input buffers (requires P2P access).
- Writes only to its own output buffer.
- Uses the same synchronization signals as other instances.
-
Requirements:
- Peer-to-peer access must be enabled between all participating GPUs.
- All instances must launch before any can complete (for synchronization).
- The device context determines which GPU executes each instance.
Limitations:
- Maximum of 8 GPUs supported.
- Multimem mode still requires the element count to be a multiple of SIMD width.
- All input/output buffers must have identical shapes.
Non-multimem 1-stage P2P and naive epilogue accept arbitrary N: the 1-stage
kernel runs a vectorized grid loop on the SIMD-aligned prefix and finishes the
last < simd_width elements with a grid-strided scalar reduce-store. Above
two ranks on NVIDIA, its vector loop loads the peers round-robin with
_load_peers_rotated and sums them in global rank order; otherwise,
including on AMD, it uses _load_reduce. The naive epilogue kernel uses the
same SIMD-prefix + scalar-tail pattern for accum → out.
Every non-multimem algorithm here gives every rank the same bits: the sum of an
element depends on its ngpus input values alone, so a new variant must add
the peers in global rank order, whatever order it issues their loads in.
Visual Overview
-
1-Stage P2P (latency-bound)
Each GPU r reads its portion from every peer buffer directly (via P2P), accumulates, then writes to its result using the epilogue:
GPU r (result_r) src_tensors[0] ─┐ src_tensors[1] ─┼──► Σ (high-precision accum) ──► output_lambda ──► result_r ... ─┘Notes:
- Non-multimem: SIMD-vector reduce on the aligned prefix; optional
scalar tail when
Nis not a multiple of SIMD width. Multimem: unchanged full-vector loads (Nmust be SIMD-aligned). - Good for small/latency-bound tensors.
- Non-multimem: SIMD-vector reduce on the aligned prefix; optional
scalar tail when
-
2-Stage P2P (bandwidth-bound)
Stage 1 (reduce-scatter): Each GPU r reduces its assigned partition and writes into its own signal payload (the bytes after the Signal header).
src_tensors[*] ──► reduce(partition r) ──► rank_sigs[r].payload (per-GPU)Stage 2 (all-gather): Each GPU r gathers all partitions from peers' payloads and writes them to its result using the epilogue.
[payload_0], [payload_1], ..., [payload_{ngpus-1}] ──► result_r (via output_lambda)
For the naive allreduce (no P2P) per-device flow and staging details, see the
_allreduce_naive_single docstring in this file.
comptime values
allreduce_tuning_table
comptime allreduce_tuning_table = Table(List(AllReduceTuningConfig(ngpus=Int(-1), num_bytes=Int(-1), sm_version=StringSpan("sm_90a"), num_blocks=Int(216), algorithm=AllReduceAlgorithm.ONE_STAGE, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(2), num_bytes=Int(2147483648), sm_version=StringSpan("sm_90a"), num_blocks=Int(216), algorithm=AllReduceAlgorithm.ONE_STAGE, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(4), num_bytes=Int(524288), sm_version=StringSpan("sm_90a"), num_blocks=Int(216), algorithm=AllReduceAlgorithm.ONE_STAGE, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(4), num_bytes=Int(134217728), sm_version=StringSpan("sm_90a"), num_blocks=Int(232), algorithm=AllReduceAlgorithm.TWO_STAGE, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(4), num_bytes=Int(2147483648), sm_version=StringSpan("sm_90a"), num_blocks=Int(216), algorithm=AllReduceAlgorithm.TWO_STAGE, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(8), num_bytes=Int(524288), sm_version=StringSpan("sm_90a"), num_blocks=Int(216), algorithm=AllReduceAlgorithm.ONE_STAGE, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(8), num_bytes=Int(2147483648), sm_version=StringSpan("sm_90a"), num_blocks=Int(216), algorithm=AllReduceAlgorithm.TWO_STAGE, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(-1), num_bytes=Int(-1), sm_version=StringSpan("sm_100a"), num_blocks=Int(512), algorithm=AllReduceAlgorithm.ONE_STAGE, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(2), num_bytes=Int(1048576), sm_version=StringSpan("sm_100a"), num_blocks=Int(512), algorithm=AllReduceAlgorithm.LAMPORT, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(2), num_bytes=Int(2147483648), sm_version=StringSpan("sm_100a"), num_blocks=Int(512), algorithm=AllReduceAlgorithm.ONE_STAGE, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(4), num_bytes=Int(1048576), sm_version=StringSpan("sm_100a"), num_blocks=Int(512), algorithm=AllReduceAlgorithm.LAMPORT, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(4), num_bytes=Int(4194304), sm_version=StringSpan("sm_100a"), num_blocks=Int(512), algorithm=AllReduceAlgorithm.ONE_STAGE, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(4), num_bytes=Int(2147483648), sm_version=StringSpan("sm_100a"), num_blocks=Int(512), algorithm=AllReduceAlgorithm.TWO_STAGE, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(8), num_bytes=Int(1048576), sm_version=StringSpan("sm_100a"), num_blocks=Int(512), algorithm=AllReduceAlgorithm.LAMPORT, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(8), num_bytes=Int(2097152), sm_version=StringSpan("sm_100a"), num_blocks=Int(512), algorithm=AllReduceAlgorithm.ONE_STAGE, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(8), num_bytes=Int(2147483648), sm_version=StringSpan("sm_100a"), num_blocks=Int(512), algorithm=AllReduceAlgorithm.TWO_STAGE, allreduce_num_blocks=OptionalReg(Int(384))), AllReduceTuningConfig(ngpus=Int(-1), num_bytes=Int(-1), sm_version=StringSpan("sm_103a"), num_blocks=Int(512), algorithm=AllReduceAlgorithm.ONE_STAGE, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(-1), num_bytes=Int(-1), sm_version=StringSpan("CDNA3"), num_blocks=Int(32), algorithm=AllReduceAlgorithm.ONE_STAGE, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(-1), num_bytes=Int(-1), sm_version=StringSpan("CDNA4"), num_blocks=Int(64), algorithm=AllReduceAlgorithm.ONE_STAGE, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(2), num_bytes=Int(2147483648), sm_version=StringSpan("CDNA4"), num_blocks=Int(64), algorithm=AllReduceAlgorithm.ONE_STAGE, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(4), num_bytes=Int(262144), sm_version=StringSpan("CDNA4"), num_blocks=Int(64), algorithm=AllReduceAlgorithm.ONE_STAGE, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(4), num_bytes=Int(2147483648), sm_version=StringSpan("CDNA4"), num_blocks=Int(64), algorithm=AllReduceAlgorithm.TWO_STAGE, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(8), num_bytes=Int(262144), sm_version=StringSpan("CDNA4"), num_blocks=Int(64), algorithm=AllReduceAlgorithm.ONE_STAGE, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(8), num_bytes=Int(2147483648), sm_version=StringSpan("CDNA4"), num_blocks=Int(44), algorithm=AllReduceAlgorithm.TWO_STAGE, allreduce_num_blocks=OptionalReg(None)), AllReduceTuningConfig(ngpus=Int(-1), num_bytes=Int(-1), sm_version=StringSpan("default"), num_blocks=Int(512), algorithm=AllReduceAlgorithm.ONE_STAGE, allreduce_num_blocks=OptionalReg(None)), __list_literal__=NoneType(None)), String("allreduce_table"))
LAMPORT_NARROW_BAND_MAX_BYTES
comptime LAMPORT_NARROW_BAND_MAX_BYTES = 131072
Per-rank message size (exclusive) below which the narrow launch wins.
LAMPORT_NARROW_BAND_MIN_BYTES
comptime LAMPORT_NARROW_BAND_MIN_BYTES = 16384
Per-rank message size (exclusive) above which the narrow launch wins.
LAMPORT_NARROW_BLOCK_SIZE
comptime LAMPORT_NARROW_BLOCK_SIZE = 128
Block size for the Lamport path's middle message band on 8x sm_100a.
LAMPORT_WIDE_BLOCK_SIZE
comptime LAMPORT_WIDE_BLOCK_SIZE = 256
Block size for the Lamport path's small and large bands on 8x sm_100a.
Structs
-
AllReduceAlgorithm: Selects which P2P allreduce kernel_allreduce_p2plaunches. -
AllReduceTuningConfig: Parameters: ngpus: Number of GPUs for running allreduce. num_bytes: Total number of input bytes supported by the config. sm_version: SM version (as string). num_blocks: Number of thread blocks for running allreduce. allreduce_num_blocks: Thread blocks for the plain allreduce only, or None to usenum_blocks. The fused collectives that read this table keepnum_blocks.
Functions
-
allreduce: Per-device allreduce: one instance per GPU builds its own output.