For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Build a model graph with Module
A Module holds your
model's parameters and defines the computation that runs on them. The same
forward() method runs eagerly on real tensors, traces to a
Graph, or compiles into a callable
the runtime executes, depending on the context you call it in.
You use modules to build a model definition, and to extend a MAX-supported model with custom layers. For a component the built-in layers don't provide, see Write a custom layer.
How a module runs
A module declares the parameters it owns, then defines forward() over
Tensor values.
Operations come from
max.functional, which you
import as F.
A module runs in one of these ways:
- Eager: calling the module executes
forward()immediately on real tensors. This is how you check shapes and values on a small input. - Traced:
trace()records the computation as a graph without compiling it. Each parameter becomes an external constant the graph refers to by name. - Compiled:
compile()traces the module and compiles the graph into a callable, binding parameter values from a state dict.
The pipeline model compiles your module once at load time and calls the
compiled result for every batch. It sets the default dtype while your
_instantiate_module() builds the module and places it on a device, so layer
constructors take shapes and options only.
Define and compose modules
Compose a module from built-in layers when they cover what you need:
Linear,
Embedding,
RMSNorm,
LayerNorm, and
others. Subclass Module directly and assign each child layer to an attribute.
Here two Linear projections form a feed-forward block, and a
ModuleList stacks two
blocks:
from max.driver import CPU
from max import functional as F
from max.nn import Linear, Module, ModuleList
from max.tensor import Tensor, default_device
class FeedForward(Module[[Tensor], Tensor]):
"""Two linear projections with a SiLU activation between them."""
def __init__(self, dim: int, hidden_dim: int) -> None:
super().__init__()
self.fc1 = Linear(dim, hidden_dim, bias=False)
self.fc2 = Linear(hidden_dim, dim, bias=False)
def forward(self, x: Tensor) -> Tensor:
return self.fc2(F.silu(self.fc1(x)))
class TwoLayerStack(Module[[Tensor], Tensor]):
"""Two feed-forward blocks in a ModuleList."""
def __init__(self, dim: int) -> None:
super().__init__()
self.layers = ModuleList([FeedForward(dim, dim) for _ in range(2)])
def forward(self, x: Tensor) -> Tensor:
for layer in self.layers:
x = layer(x)
return x
with default_device(CPU()):
model = TwoLayerStack(dim=8)
for name, _ in model.parameters:
print(name)Each parameter's name is its attribute path through the module tree, with
ModuleList entries named by index:
layers.0.fc1.weight
layers.0.fc2.weight
layers.1.fc1.weight
layers.1.fc2.weightLinear includes a bias by default. The example passes bias=False, so each
projection has only a weight.
The @module_dataclass decorator generates a module's constructor from its
field annotations. Write a custom layer shows that form.
Load weights into the module
parameters yields each parameter under its dotted attribute path, such as
layers.0.self_attn.q_proj.weight, and load_state_dict() reads a state dict
keyed by those paths:
from max.driver import CPU
from max.nn import Linear
from max.tensor import Tensor, default_device
with default_device(CPU()):
linear = Linear(4, 8, bias=False)
# The module names its parameters for you.
print([name for name, _ in linear.parameters])
# Load a checkpoint's values by parameter name.
linear.load_state_dict({"weight": Tensor.ones([8, 4])})
out = linear(Tensor.ones([2, 4]))
print(float(out.to_numpy()[0, 0]))With every weight set to one, each output element sums the input's four ones:
['weight']
4.0load_state_dict() raises KeyError when a parameter has no entry in the
state dict. With its default strict=True, it also raises ValueError for an
entry that matches no parameter, so a mismatched architecture fails before the
model runs. Here the checkpoint has a bias that the layer doesn't:
from max.driver import CPU
from max.nn import Linear
from max.tensor import Tensor, default_device
with default_device(CPU()):
linear = Linear(4, 8, bias=False)
try:
linear.load_state_dict(
{"weight": Tensor.ones([8, 4]), "bias": Tensor.ones([8])}
)
except ValueError as error:
print(error)The error names the entries that matched no parameter:
load_state_dict did not use some weights: {'bias'}A shape mismatch raises ValueError too, and so does a dtype mismatch. Pass
auto_cast=True to cast between float32 and bfloat16 on load.
A checkpoint's names often differ from a module's parameter names. A weight adapter renames checkpoint keys to the module's parameter names.
Walk a module's parameters
A module's parameter names are paths through its attributes.
module.parameters
yields every parameter in the module and its submodules as (name, tensor)
pairs, where the name is the tensor's dotted path through the module, such as
encoder.layer1.weight.
module.local_parameters
yields the tensors held directly on that module, keyed by attribute name.
module.children
yields the direct submodules, and
module.descendants
yields every module below it keyed by path.
A module is also a tree: each attribute is a child keyed by attribute name,
modules, lists, tuples, dicts, and other containers are interior nodes, and
every other value is a leaf, such as None or an integer.
module.parameters yields the items of tree.paths(module, leaf=Tensor). The
max.tree API provides methods for traversing and
modifying that tree, such as:
tree.paths()returns a dict that maps each leaf's dotted path to the leaf.tree.update()writes path-keyed values into the tree in place.tree.map()builds a new tree with each leaf replaced by what a function returns.
Every walk takes leaf, a type, a tuple of types, or a predicate that says
where the walk stops. The following example reads a module's paths with and
without leaf=Tensor, then doubles every weight with tree.map():
from max import tree
from max.driver import CPU
from max.nn import Linear, Module
from max.tensor import Tensor, default_device
class Projection(Module[[Tensor], Tensor]):
"""One linear projection."""
def __init__(self, dim: int) -> None:
super().__init__()
self.linear = Linear(dim, dim, bias=False)
def forward(self, x: Tensor) -> Tensor:
return self.linear(x)
with default_device(CPU()):
model = Projection(4)
model.load_state_dict({"linear.weight": Tensor.ones([4, 4])})
# Without leaf, every attribute value is a leaf.
print(list(tree.paths(model)))
# With leaf=Tensor, the walk stops at tensors only.
print(list(tree.paths(model, leaf=Tensor)))
# The result keeps the structure: a Projection whose weights are doubled.
doubled = tree.map(lambda t: t * 2.0, model, leaf=Tensor)
weight = tree.paths(doubled, leaf=Tensor)["linear.weight"]
print(type(doubled).__name__, float(weight.to_numpy()[0, 0]))The walk without leaf also reaches the layer's non-tensor attributes,
quant_config and bias, and every weight in the mapped copy is 2.0:
['linear.quant_config', 'linear.weight', 'linear.bias']
['linear.weight']
Projection 2.0load_state_dict() reads the parameters with
tree.paths(module, leaf=Tensor) and writes the loaded weights back with
tree.update().
tree.map() returns a module of the same type with every tensor replaced, so
one call builds a copy of a module for each device of a distributed layer.
local_shards returns one tensor per device, and device picks one by index:
tree.map(lambda t: t.local_shards[device], module, leaf=Tensor, shared=True)That call returns a copy whose tensors are each one device's shard. With
shared=True, a tensor that two attributes reference maps once, so the
copy keeps them as one object.
Trace a module to a graph
Trace a module to inspect the MLIR that compile() compiles. The trace
declares each parameter as an external constant that refers to the weight by
name:
from max.driver import CPU
from max.dtype import DType
from max.nn import Linear
from max.tensor.tensor import Tensor, TensorType, default_device
with default_device(CPU()):
linear = Linear(4, 8, bias=False)
x_type = TensorType(DType.float32, [2, 4], device=linear.device)
# Tracing builds the graph without compiling it.
staged = linear.trace(x_type)
for line in str(staged).splitlines():
if "mo.constant.external" in line:
print(line.strip())
# Compiling produces a callable bound to the module's current weights.
compiled = linear.compile(x_type)
print(compiled(Tensor.ones([2, 4])).shape)The trace declares the weight parameter as an external constant, and the
compiled callable returns a tensor of the projected shape:
%1 = mo.constant.external [name = "weight", align = 1, device = #M.device_ref<"cpu", 0>] : !mo.tensor<[8, 4], f32>
[Dim(2), Dim(8)]compile() takes the input types and, optionally, the weights. Without
weights, it binds the module's current parameter values. A pipeline model
passes the input types its batch_processor builds and the adapted checkpoint
weights, and invokes the returned callable once per batch.
Next steps
From here, go deeper on the pieces a module touches:
- Write a custom layer: subclass a built-in layer, define one from scratch, or wrap a custom op.
- Eager execution: run
TensorandFoperations directly to check a forward pass. - Weight adapters: rename checkpoint keys to the
module's parameter names before
load_state_dict(). - Model bring-up workflow: see how modules become a served architecture, from config mapping to the KV cache.