IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Distributed tensors

A distributed tensor is logically an N-dimensional array whose data can be laid out on one or more devices. Use one to split data and computation across devices while your code continues to refer to one global array. MAX uses the same Tensor abstraction for distributed and non-distributed tensors, so ops and eager layers accept both.

A shard is the local tensor for one device in a distributed tensor. Each device holds its shard's data. The placement describes how that shard relates to the global tensor, such as a slice, a full copy, or one term of a sum. When you call an op, MAX records it in a computation graph. The executor can run a supported graph through the interpreter or compile the graph and run the resulting model. The runtime launches each kernel on one device with the local buffers for that device's shard. A collective such as allreduce_sum() communicates across devices to combine their values.

Global shape​

The global shape describes the dimensions of the full array, regardless of how its values are distributed across devices. A DeviceMesh describes the devices and their axes. A DeviceMapping combines a mesh with one placement per mesh axis. It serves as the multi-device counterpart of a Device in APIs that create or move tensors.

This example splits 8 rows across 2 CPU devices on a mesh axis named tp (tensor parallelism):

from max.driver import CPU
from max.functional import full
from max.tensor.sharding import DeviceMapping, DeviceMesh, Sharded

mesh = DeviceMesh((CPU(), CPU()), (2,), ("tp",))
x = full([8, 4], 1.0, device=DeviceMapping(mesh, (Sharded(0),)))

print("distributed:", x.is_distributed)
print("global shape:", tuple(int(d) for d in x.shape))
distributed: True
global shape: (8, 4)

x.shape reports the global shape, so x.shape[0] equals 8. This axis is a PerShardDim that records the global size and each device's size. It prints as 8 and supports comparisons with integers and other Dim values.

Local shards​

The local shape describes the dimensions of one device's shard. is_distributed reports whether the tensor's mesh holds more than one device, regardless of placement. local_shards returns one unsharded tensor per device, and each tensor reports its local shape:

print("shards:", x.num_shards)
for shard in x.local_shards:
    print(tuple(int(d) for d in shard.shape))
shards: 2
(4, 4)
(4, 4)

In this example, each device holds 4 rows, while x.shape[0] reports the global total of 8. Sharded(0) places rows 0 through 3 in x.local_shards[0] and rows 4 through 7 in x.local_shards[1]. A shard is the whole tensor only when every mesh axis is replicated or the mesh has one device.

Colors show which local shard holds each row. Select a cell for its coordinate mapping.

Use the arrow keys to move between cells. Each cell includes its global coordinate and corresponding local shard.
x(8, 4)
x.local_shards[0](4, 4)
x.local_shards[1](4, 4)
The selected global cell maps to local row 1.
x[5, 2] → x.local_shards[1][1, 2]

What a placement means​

A placement describes the relation between the per-device values on one mesh axis and the tensor:

  • Replicated: Every device on the axis stores a full copy.
  • Sharded: The tensor axis splits across the devices on the mesh axis.
  • Partial: Each device stores one term of a sum over the mesh axis.
  • Unknown: Per-device values with no established relation between them; an op without a sharding rule returns this placement, and rebind_mapping() or a collective establishes the relation.

A mapping can give each mesh axis a different placement. On a mesh with dp and tp axes, an activation can split its batch along dp and stay whole along tp.

Use visualize() to inspect how a placement distributes a tensor across the mesh and check that the result matches your intent. Pass a dictionary to compare placements side by side. This example uses TensorLayout values, so you can inspect the mappings without allocating tensor data:

from max.driver import CPU
from max.dtype import DType
from max.tensor.sharding import (
    DeviceMapping,
    DeviceMesh,
    Partial,
    Placement,
    Sharded,
    TensorLayout,
    Unknown,
    visualize,
)

mesh = DeviceMesh((CPU(), CPU()), (2,), ("tp",))


def layout(placement: Placement) -> TensorLayout:
    return TensorLayout(
        DType.float32, [8, 4], DeviceMapping(mesh, (placement,))
    )


visualize(
    {
        "sharded": layout(Sharded(0)),
        "partial": layout(Partial()),
        "unknown": layout(Unknown()),
    },
    tree=False,
)

A Partial cell represents terms of a sum across devices; an Unknown cell represents values with no established relation across devices. A terminal labels these cells partial and unknown:

sharded: [8, 4] Sharded(axis=0)   partial: [8, 4] Partial()   unknown: [8, 4] Unknown()
┌──────────┐                      ┌────────────────┐          ┌────────────────┐
│ device 0 │                      │  devices 0,1   │          │  devices 0,1   │
│ [:4, :]  │                      │ [:, :] partial │          │ [:, :] unknown │
├──────────┤                      └────────────────┘          └────────────────┘
│ device 1 │
│ [4:, :]  │
└──────────┘

To compare a sharded result with an unsharded run, gather it first: materialize() allreduces Partial axes and allgathers Sharded ones, and to_numpy() does the same.

Uneven shards​

The shards on a mesh axis share one shape only when the number of devices splitting a tensor axis divides that axis's size. For a static dimension, MAX gives every device the axis size divided by the device count, rounded down, and the first devices take one more each until the remainder runs out. A tensor axis that several mesh axes split is one split over every device they span, in row-major device order. Sharded(0) on both axes of a 2x2 mesh splits 10 rows four ways:

from max.driver import CPU
from max.functional import full
from max.tensor.sharding import DeviceMapping, DeviceMesh, Sharded

mesh = DeviceMesh((CPU(),) * 4, (2, 2), ("dp", "tp"))
x = full([10, 4], 1.0, device=DeviceMapping(mesh, (Sharded(0), Sharded(0))))

print("rows:", x.shape[0])
for i, shard in enumerate(x.local_shards):
    print(f"device {i}:", tuple(int(d) for d in shard.shape))
rows: 10
device 0: (3, 4)
device 1: (3, 4)
device 2: (2, 4)
device 3: (2, 4)

x.shape[0] reports the global 10 rows, and each entry of local_shards reports the rows its device holds.

Colors show which local shard holds each row. Select a cell for its coordinate mapping.

Use the arrow keys to move between cells. Each cell includes its global coordinate and corresponding local shard.
x(10, 4)
x.local_shards[0](3, 4)
x.local_shards[1](3, 4)
x.local_shards[2](2, 4)
x.local_shards[3](2, 4)
The selected global cell maps to local row 0.
x[8, 2] → x.local_shards[3][0, 2]

Resharding​

A reshard changes the placement of a tensor's values on its mesh. For example, changing from Sharded to Replicated gathers the shards onto every device, and changing Partial to Replicated sums their terms. These transitions can require communication between devices. Use transfer_to() to request a target DeviceMapping explicitly.

This example gathers the shards of x so every device holds the full (8, 4) tensor:

from max.driver import CPU
from max.functional import full, transfer_to
from max.tensor.sharding import DeviceMapping, DeviceMesh, Replicated, Sharded

mesh = DeviceMesh((CPU(), CPU()), (2,), ("tp",))
x = full([8, 4], 1.0, device=DeviceMapping(mesh, (Sharded(0),)))
y = transfer_to(x, DeviceMapping(mesh, (Replicated(),)))

print("x placements:", x.placements)
print("y placements:", y.placements)
for shard in y.local_shards:
    print(tuple(int(d) for d in shard.shape))
x placements: (Sharded(axis=0),)
y placements: (Replicated(),)
(8, 4)
(8, 4)

transfer_to() returns y with the requested placement on the same mesh, and x keeps its placement. Every entry of y.local_shards holds the whole tensor, and each device runs the allgather() that fills it.

Placement
x.local_shardsSharded(0)
[0](4, 4)
[1](4, 4)
Each device holds one 4-row shard of x.x: Sharded(0)

An op's sharding rule describes the input placements it accepts and the output placements it produces. MAX chooses a placement plan from that rule. If the plan requires an input placement different from the current placement, MAX must reshard that input before the op can run. auto_reshard() controls how MAX handles a reshard selected for an op inside its block:

  • "silent": Performs the reshard without a message.
  • "warn": Performs the reshard and emits a UserWarning that describes the move.
  • "raise": Rejects the reshard and raises ShardingError before the op runs.

Outside an auto_reshard() block, MAX allows the transitions in DEFAULT_TRANSITIONS and uses the "raise" mode. An op raises ShardingError in every mode when no placement plan uses the allowed transitions.

Next steps​

To use placements in a multi-GPU architecture, continue to Run on multiple GPUs. For details about local shards and host-side accessors, see the Tensor API reference.

Was this page helpful?