For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
allgather_rows_slice
def allgather_rows_slice[dtype: DType, ngpus: Int, in_layout: TensorLayout, in_origin: Origin[mut=in_origin.mut], out_layout: TensorLayout, out_origin: MutOrigin, in_engine: TensorEngine, out_engine: TensorEngine](input_buffers: Array[TileTensor[dtype, in_layout, in_origin, Engine=in_engine], ngpus], output_buffers: Array[TileTensor[dtype, out_layout, out_origin, Engine=out_engine], (ngpus * ngpus)], rank_sigs: Array[Pointer[Signal, MutAnyOrigin], ngpus], row_elems: Int, ctx: DeviceContext, my_rank: Int, _max_num_blocks: Optional[Int] = None)
Per-device all-gather in which each device keeps only its own row block.
Every device holds a row-major [rows, row_elems] shard of one tensor
(typically a column-sharded GEMM output). Device d receives rows
[d * rows / ngpus, (d + 1) * rows / ngpus) of EVERY peer's shard, so
output_buffers[d * ngpus + j] holds that row block of shard j.
Concatenating those blocks along the column axis gives the same bytes as
allgather + concat + a row slice, with ngpus times fewer bytes moved.
A row block is contiguous in a row-major shard, so this is allgather
over per-reader windows: the pointer each reader pulls from peer j is
offset by my_rank row blocks. Barriers, P2P/TMA paths and the signal
protocol are allgather's own.
Data movement only; no arithmetic touches the elements, so the result is bit-identical to the reference composition for every dtype.
Parameters:
- dtype (
DType): Data type of the tensor elements. - ngpus (
Int): Number of devices; one full-world group. - in_layout (
TensorLayout): Layout of the input TileTensors. - in_origin (
Origin[mut=mut]): Origin of the input TileTensors. - out_layout (
TensorLayout): Layout of the output TileTensors. - out_origin (
MutOrigin): Origin of the output TileTensors. - in_engine (
TensorEngine): Engine of the input TileTensors. - out_engine (
TensorEngine): Engine of the output TileTensors.
Args:
- input_buffers (
Array[TileTensor[dtype, in_layout, in_origin, Engine=in_engine], ngpus]): Every device's flat[rows * row_elems]shard, indexed by GLOBAL device rank. All shards must be the same size. - output_buffers (
Array[TileTensor[dtype, out_layout, out_origin, Engine=out_engine], (ngpus * ngpus)]): Every device'sngpusoutputs, laid out[device * ngpus + source], eachrows / ngpus * row_elemselements. - rank_sigs (
Array[Pointer[Signal, MutAnyOrigin], ngpus]): Allngpusdevices' Signal pointers. - row_elems (
Int): Elements per row (the product of all dimensions after the first). - ctx (
DeviceContext): Device context for THIS GPU. - my_rank (
Int): GLOBAL rank of this GPU in[0, ngpus). - _max_num_blocks (
Optional[Int]): Maximum number of blocks for kernel launch (optional).
Raises:
Error: The shards differ in size, rows is not a multiple of ngpus,
a row block is not a multiple of 16 bytes (the window offset must
stay vector-aligned), or an output buffer is not one row block.