IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

allreduce

def allreduce[dtype: DType, in_layout: TensorLayout, in_origin: ImmOrigin, rank_sigs_origin: MutOrigin, out_layout: TensorLayout, out_origin: MutOrigin, //, ngpus: Int, pdl_level: PDLLevel = PDLLevel(), *, use_multimem: Bool = False](input_tensors: Array[TileTensor[dtype, in_layout, in_origin], Int(1) if use_multimem else ngpus], output_tensor: TileTensor[dtype, out_layout, out_origin], rank_sigs: Array[Pointer[Signal, rank_sigs_origin], ngpus], ctx: DeviceContextBase, _max_num_blocks: Optional[Int] = None)

Per-GPU allreduce for use in multi-threaded contexts.

Currently requires prior single-threaded call to init_comms, as thread-safe version not yet implemented.

def allreduce[dtype: DType, in_layout: TensorLayout, in_origin: ImmOrigin, rank_sigs_origin: MutOrigin, out_layout: TensorLayout, //, ngpus: Int, output_lambda: def[dtype: DType, width: SIMDLength, *, alignment: Int](Coord[*?], SIMD[dtype, width]) capturing thin -> None, pdl_level: PDLLevel = PDLLevel(), *, use_multimem: Bool = False](input_tensors: Array[TileTensor[dtype, in_layout, in_origin], Int(1) if use_multimem else ngpus], output_layout: out_layout, rank_sigs: Array[Pointer[Signal, rank_sigs_origin], ngpus], ctx: DeviceContextBase, _max_num_blocks: Optional[Int] = None)

Per-GPU allreduce that hands every reduced value to output_lambda.

The vendor collective can only write its result to memory, so it reduces into a scratch buffer of the input dtype, and an elementwise pass then reads each value back and calls output_lambda with its output coordinates. The output itself is given as a layout: the epilogue owns every write, and its destination need not share the input dtype or a contiguous layout. Requires a prior single-threaded call to init_comms.

Was this page helpful?