For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
allreduce
def allreduce[dtype: DType, in_layout: TensorLayout, in_origin: ImmOrigin, rank_sigs_origin: MutOrigin, out_layout: TensorLayout, out_origin: MutOrigin, //, ngpus: Int, pdl_level: PDLLevel = PDLLevel(), *, use_multimem: Bool = False](input_tensors: Array[TileTensor[dtype, in_layout, in_origin], Int(1) if use_multimem else ngpus], output_tensor: TileTensor[dtype, out_layout, out_origin], rank_sigs: Array[Pointer[Signal, rank_sigs_origin], ngpus], ctx: DeviceContext, _max_num_blocks: Optional[Int] = None)
Per-GPU allreduce for use in multi-threaded contexts.
Currently requires prior single-threaded call to init_comms, as thread-safe version not yet implemented.
def allreduce[dtype: DType, in_layout: TensorLayout, in_origin: ImmOrigin, rank_sigs_origin: MutOrigin, out_layout: TensorLayout, OutputLambdaType: CollectiveEpilogueFn, //, ngpus: Int, pdl_level: PDLLevel = PDLLevel(), *, use_multimem: Bool = False](input_tensors: Array[TileTensor[dtype, in_layout, in_origin], Int(1) if use_multimem else ngpus], output_layout: out_layout, output_lambda: OutputLambdaType, rank_sigs: Array[Pointer[Signal, rank_sigs_origin], ngpus], ctx: DeviceContext, _max_num_blocks: Optional[Int] = None)
Per-GPU allreduce that hands every reduced value to output_lambda.
The vendor collective can only write its result to memory, so it reduces
into a scratch buffer of the input dtype, and an elementwise pass then
reads each value back and calls output_lambda with its output
coordinates. The output itself is given as a layout: the epilogue owns
every write, and its destination need not share the input dtype or a
contiguous layout. Requires a prior single-threaded call to init_comms.