IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

allgather_rows_slice

def allgather_rows_slice[dtype: DType, ngpus: Int, in_layout: TensorLayout, in_origin: Origin[mut=in_origin.mut], out_layout: TensorLayout, out_origin: MutOrigin, in_engine: TensorEngine, out_engine: TensorEngine](input_buffers: Array[TileTensor[dtype, in_layout, in_origin, Engine=in_engine], ngpus], output_buffers: Array[TileTensor[dtype, out_layout, out_origin, Engine=out_engine], (ngpus * ngpus)], rank_sigs: Array[Pointer[Signal, MutAnyOrigin], ngpus], row_elems: Int, ctx: DeviceContext, my_rank: Int, _max_num_blocks: Optional[Int] = None)

Per-device all-gather in which each device keeps only its own row block.

Every device holds a row-major [rows, row_elems] shard of one tensor (typically a column-sharded GEMM output). Device d receives rows [d * rows / ngpus, (d + 1) * rows / ngpus) of EVERY peer's shard, so output_buffers[d * ngpus + j] holds that row block of shard j. Concatenating those blocks along the column axis gives the same bytes as allgather + concat + a row slice, with ngpus times fewer bytes moved.

A row block is contiguous in a row-major shard, so this is allgather over per-reader windows: the pointer each reader pulls from peer j is offset by my_rank row blocks. Barriers, P2P/TMA paths and the signal protocol are allgather's own.

Data movement only; no arithmetic touches the elements, so the result is bit-identical to the reference composition for every dtype.

Parameters:

  • ​dtype (DType): Data type of the tensor elements.
  • ​ngpus (Int): Number of devices; one full-world group.
  • ​in_layout (TensorLayout): Layout of the input TileTensors.
  • ​in_origin (Origin[mut=mut]): Origin of the input TileTensors.
  • ​out_layout (TensorLayout): Layout of the output TileTensors.
  • ​out_origin (MutOrigin): Origin of the output TileTensors.
  • ​in_engine (TensorEngine): Engine of the input TileTensors.
  • ​out_engine (TensorEngine): Engine of the output TileTensors.

Args:

Raises:

Error: The shards differ in size, rows is not a multiple of ngpus, a row block is not a multiple of 16 bytes (the window offset must stay vector-aligned), or an output buffer is not one row block.

Was this page helpful?