For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
DistributedAllGather
struct DistributedAllGather
Registers the mo.distributed.allgather graph op with the graph compiler.
Implemented traits
Methods
execute
static def execute[Outs: TypeList[Outs.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], //, _trace_name: StaticString, group_size: Int = Int(0), slice_rows: Bool = False](outputs: TensorTuple[*Outs.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], dev_ctxs_input: DeviceContextArray)
Distributed allgather operation implementation.
Parameters:
- Outs (
TypeList[values]): The element types ofoutputs(inferred). - Ins (
TypeList[values]): The element types ofinputs(inferred). - Sigs (
TypeList[values]): The element types ofsignal_buffers(inferred). - _trace_name (
StaticString): Trace name for profiling. - group_size (
Int): Number of devices per allgather group; must be at least 1 and must evenly divide the total number of devices (defaults to 0). - slice_rows (
Bool): Whether each device receives only its own contiguous block ofdim0 / num_devicesrows of every gathered piece rather than the whole piece (seeallgather_rows_slice). Requires one full-world group.
Args:
- outputs (
TensorTuple[*Outs.values]): Output tensors (one per GPU) to store gathered results. - inputs (
TensorTuple[*Ins.values]): Input tensors (one per GPU) containing values to gather. - signal_buffers (
TensorTuple[*Sigs.values]): Device buffer values used for synchronization. - dev_ctxs_input (
DeviceContextArray): Device contexts for participating GPUs.