For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Distributed tensors
A distributed tensor is logically an N-dimensional array whose data can be laid
out on one or more devices. Use one to split data and computation across devices
while your code continues to refer to one global array. MAX uses the same
Tensor abstraction for
distributed and non-distributed tensors, so ops and eager layers accept both.
A shard is the local tensor for one device in a distributed tensor. Each device
holds its shard's data. The placement describes how that shard relates to the
global tensor, such as a slice, a full copy, or one term of a sum. When you call
an op, MAX records it in a computation graph. The executor can run a supported
graph through the interpreter or compile the graph and run the resulting model.
The runtime launches each kernel on one device with the local buffers for that
device's shard. A collective such as
allreduce_sum()
communicates across devices to combine their values.
Global shape
The global shape describes the dimensions of the full array, regardless of how
its values are distributed across devices. A
DeviceMesh describes
the devices and their axes. A
DeviceMapping
combines a mesh with one placement per mesh axis. It serves as the multi-device
counterpart of a Device in APIs
that create or move tensors.
This example splits 8 rows across 2 CPU devices on a mesh axis named tp
(tensor parallelism):
from max.driver import CPU
from max.functional import full
from max.tensor.sharding import DeviceMapping, DeviceMesh, Sharded
mesh = DeviceMesh((CPU(), CPU()), (2,), ("tp",))
x = full([8, 4], 1.0, device=DeviceMapping(mesh, (Sharded(0),)))
print("distributed:", x.is_distributed)
print("global shape:", tuple(int(d) for d in x.shape))distributed: True
global shape: (8, 4)x.shape reports the global shape, so x.shape[0] equals 8. This axis is a
PerShardDim
that records the global size and each device's size. It prints as 8 and
supports comparisons with integers and other
Dim values.
Local shards
The local shape describes the dimensions of one device's shard.
is_distributed reports whether the tensor's mesh
holds more than one device, regardless of placement.
local_shards returns one unsharded tensor per device,
and each tensor reports its local shape:
print("shards:", x.num_shards)
for shard in x.local_shards:
print(tuple(int(d) for d in shard.shape))shards: 2
(4, 4)
(4, 4)In this example, each device holds 4 rows, while x.shape[0] reports the global
total of 8.
Sharded(0) places rows 0 through 3 in x.local_shards[0] and rows 4 through
7 in x.local_shards[1]. A shard is the whole tensor only when every mesh axis
is replicated or the mesh has one device.
Colors show which local shard holds each row. Select a cell for its coordinate mapping.
Use the arrow keys to move between cells. Each cell includes its global coordinate and corresponding local shard.x(8, 4)x.local_shards[0](4, 4)x.local_shards[1](4, 4)x[5, 2] → x.local_shards[1][1, 2]What a placement means
A placement describes the relation between the per-device values on one mesh axis and the tensor:
Replicated: Every device on the axis stores a full copy.Sharded: The tensor axis splits across the devices on the mesh axis.Partial: Each device stores one term of a sum over the mesh axis.Unknown: Per-device values with no established relation between them; an op without a sharding rule returns this placement, andrebind_mapping()or a collective establishes the relation.
A mapping can give each mesh axis a different placement. On a mesh with dp
and tp axes, an activation can split its batch along dp and stay whole
along tp.
Use visualize() to
inspect how a placement distributes a tensor across the mesh and check that the
result matches your intent. Pass a dictionary to compare placements side by
side. This example uses
TensorLayout
values, so you can inspect the mappings without allocating tensor data:
from max.driver import CPU
from max.dtype import DType
from max.tensor.sharding import (
DeviceMapping,
DeviceMesh,
Partial,
Placement,
Sharded,
TensorLayout,
Unknown,
visualize,
)
mesh = DeviceMesh((CPU(), CPU()), (2,), ("tp",))
def layout(placement: Placement) -> TensorLayout:
return TensorLayout(
DType.float32, [8, 4], DeviceMapping(mesh, (placement,))
)
visualize(
{
"sharded": layout(Sharded(0)),
"partial": layout(Partial()),
"unknown": layout(Unknown()),
},
tree=False,
)A Partial cell represents terms of a sum across devices; an Unknown cell
represents values with no established relation across devices. A terminal
labels these cells partial and unknown:
sharded: [8, 4] Sharded(axis=0) partial: [8, 4] Partial() unknown: [8, 4] Unknown()
┌──────────┐ ┌────────────────┐ ┌────────────────┐
│ device 0 │ │ devices 0,1 │ │ devices 0,1 │
│ [:4, :] │ │ [:, :] partial │ │ [:, :] unknown │
├──────────┤ └────────────────┘ └────────────────┘
│ device 1 │
│ [4:, :] │
└──────────┘To compare a sharded result with an unsharded run, gather it first:
materialize()
allreduces Partial axes
and allgathers Sharded
ones, and
to_numpy()
does the same.
Uneven shards
The shards on a mesh axis share one shape only when the number of devices
splitting a tensor axis divides that axis's size. For a static dimension, MAX
gives every device the axis size divided by the device count, rounded down, and
the first devices take one more each until the remainder runs out. A tensor axis
that several mesh axes split is one split over every device they span, in
row-major device order. Sharded(0) on both axes of a 2x2 mesh splits 10 rows
four ways:
from max.driver import CPU
from max.functional import full
from max.tensor.sharding import DeviceMapping, DeviceMesh, Sharded
mesh = DeviceMesh((CPU(),) * 4, (2, 2), ("dp", "tp"))
x = full([10, 4], 1.0, device=DeviceMapping(mesh, (Sharded(0), Sharded(0))))
print("rows:", x.shape[0])
for i, shard in enumerate(x.local_shards):
print(f"device {i}:", tuple(int(d) for d in shard.shape))rows: 10
device 0: (3, 4)
device 1: (3, 4)
device 2: (2, 4)
device 3: (2, 4)x.shape[0] reports the global 10 rows, and each entry of local_shards
reports the rows its device holds.
Colors show which local shard holds each row. Select a cell for its coordinate mapping.
Use the arrow keys to move between cells. Each cell includes its global coordinate and corresponding local shard.x(10, 4)x.local_shards[0](3, 4)x.local_shards[1](3, 4)x.local_shards[2](2, 4)x.local_shards[3](2, 4)x[8, 2] → x.local_shards[3][0, 2]Resharding
A reshard changes the placement of a tensor's values on its mesh. For example,
changing from Sharded to Replicated gathers the shards onto every device,
and changing Partial to Replicated sums their terms. These transitions can
require communication between devices. Use
transfer_to() to
request a target DeviceMapping explicitly.
This example gathers the shards of x so every device holds the full
(8, 4) tensor:
from max.driver import CPU
from max.functional import full, transfer_to
from max.tensor.sharding import DeviceMapping, DeviceMesh, Replicated, Sharded
mesh = DeviceMesh((CPU(), CPU()), (2,), ("tp",))
x = full([8, 4], 1.0, device=DeviceMapping(mesh, (Sharded(0),)))
y = transfer_to(x, DeviceMapping(mesh, (Replicated(),)))
print("x placements:", x.placements)
print("y placements:", y.placements)
for shard in y.local_shards:
print(tuple(int(d) for d in shard.shape))x placements: (Sharded(axis=0),)
y placements: (Replicated(),)
(8, 4)
(8, 4)transfer_to() returns y with the requested placement on the same mesh, and
x keeps its placement. Every entry of y.local_shards holds the whole
tensor, and each device runs the allgather() that fills it.
x.local_shardsSharded(0)[0](4, 4)[1](4, 4)x: Sharded(0)An op's sharding rule describes the input placements it accepts and the output
placements it produces. MAX chooses a placement plan from that rule. If the
plan requires an input placement different from the current placement, MAX must
reshard that input before the op can run.
auto_reshard()
controls how MAX handles a reshard selected for an op inside its block:
"silent": Performs the reshard without a message."warn": Performs the reshard and emits aUserWarningthat describes the move."raise": Rejects the reshard and raisesShardingErrorbefore the op runs.
Outside an auto_reshard() block, MAX allows the transitions in
DEFAULT_TRANSITIONS
and uses the "raise" mode. An op raises
ShardingError in
every mode when no placement plan uses the allowed transitions.
Next steps
To use placements in a multi-GPU architecture, continue to
Run on multiple GPUs.
For details about local shards and host-side accessors, see the
Tensor API reference.