For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Model bring-up workflow
Model bring-up is the process of adapting a pretrained model from Hugging Face or PyTorch to run in MAX.
To bring a model to MAX, you must implement a model architecture. A model architecture is a Python package that defines the model's modules, maps its configuration, and loads its weights. Once implemented, you can serve the model with hardware-optimized kernels, production batching, KV cache management, and multi-GPU distribution.
Writing a custom architecture means choosing a reference implementation, copying the parts your model needs into a new package, and adapting its configuration and weights.
Keep a model directory self-contained: it imports its own files and the shared
MAX libraries (max.nn, max.graph, max.pipelines.lib,
max.pipelines.kv_cache, max.serve), and no other model. A model that
imports from another model breaks when that model changes; a self-contained one
is deleted by removing its directory and its registry entry.
A symbol several models need belongs in the shared libraries: move it there and let every model import it from the library. Code that only this model needs stays in the package.
Architecture directory structure
Most text-generation model architectures follow this standard directory layout, which the OLMo 3 reference architecture uses:
{your_model}/
├── layers/
├── __init__.py
├── arch.py
├── batch_processor.py
├── model_config.py
├── model.py
├── {model_name}.py
└── weight_adapters.pyEach file serves a distinct role:
layers/: holds custom layer implementations for components thatmax.nndoesn't provide.__init__.py: exposes theARCHITECTURESlist that MAX reads to register each entry.arch.py: registers the architecture with MAX. It declares the architecture's name, supported encodings, and the classes that implement the config, model, and weight adapter. The name must match thearchitectures[0]value in the model's Hugging Faceconfig.json. You don't declare memory estimates: at startup the model worker measures peak memory during warmups and sizes the KV cache from it.batch_processor.py: optional batch processor that prepares model inputs for each scheduled batch and declares the input types your module compiles against. Create one only when the model has unique batching logic. If batching matches an existing family, register that family's processor through thebatchingfield inarch.py(for example,batching=Llama3ModuleV3BatchProcessor). See input preparation for a batch processor.model_config.py: translates Hugging Faceconfig.jsonfields into a typed configuration object that your modules read when MAX builds them.model.py: the pipeline model class registered inarch.py. Start from the ModuleV3 pipeline model base or a copy of the reference architecture's pipeline model, and override the parts that differ.{model_name}.py: defines the root module and the transformer stack. Most models with unique sub-layers, an encoder, or a structure that differs from the generic transformer require this file.weight_adapters.py: translates Hugging Face checkpoint weight names to the parameter names of your MAX module.
The following sections describe how each component works and whether it needs customization.
Write modules with max.nn
Write each model component as a subclass of
Module from
max.nn. A module declares its
argument and return types in its signature, such as
Module[[Tensor, PagedCacheValues, Tensor, Tensor], tuple[Tensor, ...]], and
implements
forward()
over Tensor values.
Operations come from
max.functional, which you
import as F.
Layer constructors take shapes and options only, such as
Linear(in_dim, out_dim, bias=False) or
Embedding(vocab_size, dim=hidden_size).
The pipeline sets the dtype for the whole build, and your pipeline model places
the module on a device with .to(device).
The pipeline model compiles your module for serving. Its base class,
ModuleV3PipelineModelWithKVCache from
max.pipelines.lib, runs these steps while it
builds the decoder:
- Loads the checkpoint and applies your weight adapter.
- Calls your
_create_model_config()to build the config object. - Calls your
_prepare_state_dict()to cast or rewrite weights that depend on the config. - Calls your
_init_distributed_runtime()to set up a distributed runtime. - Builds your module inside
F.lazy()with the config's dtype as the default, by calling your_instantiate_module().F.lazy()defers tensor realization while_instantiate_module()builds the module. - Traces and compiles the module with
Module.stage(), whichcompile()wraps. The batch processor supplies the input types, and each parameter takes its value from the state dict by name.
The following model.py from the olmo3 reference architecture implements
both hooks. It also sets model_config_cls, the config class the pipeline
model uses to size the KV cache before the decoder is built. A
ModuleV3PipelineModelWithKVCache subclass that leaves it unset raises an
error at construction:
from typing import Any, ClassVar
from max.pipelines.lib import ModuleV3PipelineModelWithKVCache
class Olmo3Model(ModuleV3PipelineModelWithKVCache[TextContext]):
model_config_cls: ClassVar[type[Any]] = Olmo3Config
def _create_model_config(self, state_dict):
model_config = Olmo3Config.initialize(
self.pipeline_config, max_seq_len=self.max_seq_len
)
model_config.finalize(
huggingface_config=self.huggingface_config,
state_dict=state_dict,
return_logits=self.return_logits,
)
return model_config
def _instantiate_module(self, model_config):
nn_model = Olmo3(model_config, self.kv_params)
nn_model.to(self.devices[0])
return nn_modelDuring development, the same forward() also runs eagerly on real tensors.
For a short example of a module and its parameters, see
Build a model graph with Module. To learn
the Tensor and F APIs on a small forward pass, see
Eager execution.
Choose a reference architecture
The reference architectures in MAX's codebase are complete implementations of
this package structure. Each one is a working implementation that max serve
can load. Browse the
reference architectures
alongside this guide to see how the files fit together in a finished
implementation.
Most large language models are variations on a small set of transformer blueprints. The attention mechanism, normalization placement, and residual structure in your model usually match an existing reference. Copying the closest one brings its modules, KV cache wiring, and kernel choices into your package, and the copy keeps no import edge back to the model it came from.
Your implementation files define only what differs. model_config.py bridges
the Hugging Face config fields to MAX's representation. weight_adapters.py
remaps checkpoint keys to match MAX's naming. model.py holds a copy of the
reference pipeline model adapted to your model, and arch.py registers the
quantization encodings your checkpoint supports. If your model's structure is
identical to a reference and only those surface details differ, those files are
all you need.
Choose a reference by its structural features:
- Llama 3: Use for a standard dense transformer with GQA, RoPE, pre-RMSNorm, and SwiGLU. Its config also reads MuP scalars. It's the starting point for most models.
- OLMo 3: Use when your model applies QK-norm across the full projection width and mixes sliding-window and full attention layers, with YaRN RoPE scaling.
- Gemma 3: Use when your model applies QK-norm per head, places RMSNorm both before and after each sublayer, and alternates local and global attention.
- Phi-3:
Use when your checkpoint stores fused
qkv_projandgate_up_projweights. It subclasses the Llama 3 pipeline model and adds only a weight adapter. - GPT-OSS: Use when your model routes tokens to sparse Mixture-of-Experts (MoE) layers.
- DeepSeek-V2: Use when your model uses Multi-Head Latent Attention (MLA), which compresses the KV cache using low-rank projections, on a single GPU.
- DeepSeek-V3: Use when your model combines MLA with MoE routing and runs across GPUs with tensor, data, or expert parallelism.
- Nemotron-H: Use when your model interleaves Mamba state-space layers with attention layers.
For vision-language models, start from gemma3multimodal_modulev3,
idefics3_modulev3, or kimik2_5_modulev3. Each reference organizes its
files differently. For example, the llama3_modulev3 module file is
llama3.py.
Every LLM implementation requires config field mapping, weight naming, and KV cache integration. If your model's structure departs from the reference in a few attributes or methods, override the model components that differ. If the differences are structural, extend the module tree to implement those components from scratch. Examples include a different attention mechanism, custom routing, or a positional encoding variant that an attribute override can't express.
Map config fields
MAX reads a model's Hugging Face config.json through a typed configuration
object, and your modules read their sizes and options from that object. The
general pattern is to subclass the reference config class and set the fields
where the Hugging Face convention differs from MAX's representation. The
Files and versions tab on a Hugging Face model page includes the
config.json your config class needs to map. For a field-by-field guide to
reading a config and sorting its fields into the work each implies, see
Read a model config. Use the config to
identify which fields your subclass must set and which MAX field names they
correspond to:
{
"architectures": ["YourModelForCausalLM"],
"model_type": "your_model",
"hidden_size": 4096,
"num_attention_heads": 32,
"num_key_value_heads": 8,
"num_hidden_layers": 32,
"vocab_size": 256000,
"logit_scale": 0.0625
}A config class reads raw Hugging Face fields in its initialize() class
method, which receives the pipeline config and a required keyword-only
max_seq_len. Later, finalize() sets the fields that depend on the
checkpoint, such as whether the embeddings are tied. Copy the closest
reference config into your package and subclass the copy. Call
super().initialize() so the parent maps every field it knows, then set the
fields that differ.
The following example subclasses a copy of Llama3Config from
llama3_modulev3, saved as llama3_config.py in your package. That config
already has a logits_scaling field, which the Llama 3 module divides the
logits by. This checkpoint stores the reciprocal as logit_scale:
from dataclasses import dataclass
from .llama3_config import Llama3Config
@dataclass(kw_only=True)
class YourConfig(Llama3Config):
@classmethod
def initialize(cls, pipeline_config, model_config=None, *, max_seq_len):
config = super().initialize(
pipeline_config, model_config, max_seq_len=max_seq_len
)
hf_config = (model_config or pipeline_config.model).huggingface_config
logit_scale = getattr(hf_config, "logit_scale", 1.0)
config.logits_scaling = 1.0 / logit_scale # 0.0625 → 16.0
return configYour pipeline model names YourConfig in model_config_cls and builds it in
_create_model_config(), the way the model.py example earlier on this page
does.
Map checkpoint weights
Every parameter of your MAX module has a name, a shape, and a dtype. The
name is the parameter's attribute path from the root module. Because the
olmo3 root module holds its transformer as self.language_model, its names
look like language_model.layers.0.self_attn.q_proj.weight. Checkpoint
weights rarely match those names, so you write a weight adapter function. The
function converts the checkpoint into the weights your module expects.
To list the names your module expects, build it under F.lazy() and read its
parameters:
from max import functional as F
with F.lazy():
model = YourRootModule(config, kv_params)
expected = {name for name, _ in model.parameters}compile() checks the adapted weights against these parameters:
- A parameter with no matching entry raises a
KeyError. - An entry whose shape or dtype differs from its parameter raises a
ValueError. Passingauto_cast=Truetocompile()permits casts betweenfloat32andbfloat16. - A checkpoint entry that matches no parameter raises no error, and
compile()leaves it out. Compare the adapter's output keys withexpectedto catch weights your module never loads.
For models with tied embeddings, the module creates no lm_head parameter,
and forward() computes the logits as h @ self.embed_tokens.weight.T. If the
checkpoint still stores an lm_head.weight entry, the entry matches no
parameter, and compile() leaves it out.
Renaming is the most common transformation needed in a weight adapter. You might also need to cast dtypes, drop weights the module doesn't use, and transform tensor layouts. To learn how to implement these transformations, see Write a weight adapter.
Connect to the KV cache
MAX pre-allocates the KV cache at startup from values your config provides,
before the first request arrives. The allocation depends on the number of KV
heads, the head dimension, the number of layers, and the maximum sequence
length. Your config class supplies these values by subclassing
ArchConfigWithStoredKVParams
and
ArchConfigWithKVCache.
ArchConfigWithStoredKVParams returns your config's kv_params field from
get_kv_params(),
which MAX calls to size the cache. It returns the max_seq_len field from
get_max_seq_len(), so set that field from the max_seq_len argument that
initialize() receives. Your initialize() fills kv_params by calling the
construct_kv_params()
class method. The default implementation reads these Hugging Face fields:
num_key_value_heads: total KV heads across all devices. For grouped-query attention models, this field differs fromnum_attention_heads. MAX sizes the cache fromnum_key_value_heads.head_dim, throughget_head_dim(): dimensionality of each attention head. When the config has nohead_dim, it'shidden_size // num_attention_heads.num_hidden_layers, throughget_num_layers(): number of hidden layers.
Override get_head_dim(), get_num_layers(), or construct_kv_params() when
your Hugging Face config names these values differently. MAX sizes the cache
with the overrides on your pipeline model's model_config_cls, and the
registry uses the config class in arch.py, so point both at your config
class. The kv_params that initialize() stores must come from the same
overrides. Some reference configs name the class explicitly instead of using
cls: Llama3Config.initialize() calls Llama3Config.construct_kv_params().
If you
subclass one of those, recompute kv_params in your initialize() after the
super() call. The following example shows the shape of the default
implementation:
@classmethod
def construct_kv_params(
cls,
huggingface_config: AutoConfig,
pipeline_config: PipelineConfig,
devices: list[DeviceRef],
kv_cache_config: KVCacheConfig,
cache_dtype: DType,
*,
allow_kv_head_replication: bool = False,
) -> KVCacheParamInterface:
return kv_cache_config.to_params(
dtype=cache_dtype,
allow_kv_head_replication=allow_kv_head_replication,
n_kv_heads=huggingface_config.num_key_value_heads,
head_dim=cls.get_head_dim(huggingface_config),
num_layers=cls.get_num_layers(huggingface_config),
devices=devices,
data_parallel_degree=pipeline_config.model.data_parallel_degree,
)Most architectures return a flat
KVCacheParams.
Architectures with multiple or hybrid KV caches (for example, separate
sliding-window and full-attention caches) return a
MultiKVCacheParams
tree, which the cache manager consumes through the shared
KVCacheParamInterface.
Pass the cache into the module
The compiled module receives the KV cache as flat inputs after the token
inputs. The root module's forward() collects them in *variadic_args and
unflattens them with its kv_params. For a single cache, call
self.kv_params.unflatten_kv_inputs(). For a sliding-window and global
cache pair, call unflatten_basic_kv_tree(). The olmo3 root module wraps
each cache in PagedCacheValues from
max.nn.common_layers.kv_cache:
class Olmo3(Module[[Tensor, Tensor, Tensor], ModuleV3Outputs]):
def __init__(self, config: Olmo3Config, kv_params: KVCacheParamInterface):
super().__init__()
# ...
self.language_model = Olmo3TextModel(config)
self.kv_params = kv_params
def forward(
self,
tokens: Tensor,
return_n_logits: Tensor,
input_row_offsets: Tensor,
*variadic_args: Tensor,
) -> ModuleV3Outputs:
kv_inputs = iter(x._graph_value for x in variadic_args)
sliding_inputs, global_inputs = self.kv_params.unflatten_basic_kv_tree(
kv_inputs
)
sliding_kv = PagedCacheValues.from_upstream(
sliding_inputs, tokens.mapping
)
global_kv = PagedCacheValues.from_upstream(
global_inputs, tokens.mapping
)
return self.language_model(
tokens, sliding_kv, global_kv, return_n_logits, input_row_offsets
)Each attention layer stores its layer index and receives a PagedCacheValues.
It calls fused kernels from
max.nn.kernels:
rope_split_store_ragged() applies RoPE and writes keys and values to the
cache, and flash_attention_ragged() reads them back. MAX manages the paged
memory behind the cache, using the settings in
KVCacheConfig.
Reuse model components
Copy the closest model's classes into your package and adapt the copy. Record the model and commit each copied file came from in a comment at the top of the file.
Hugging Face follows the same pattern: many model classes in transformers
subclass a family base and override only what changed. Adapt the copy the same
way.
Adapt the copied pipeline model
Your model.py holds a copy of the reference pipeline model. Override only the
class attributes or methods that differ. The Llama 3 pipeline model exposes
config_class, norm_method, and attention_bias as class attributes. For
example, a model that matches Llama 3 except for bias on the attention
projections needs a single class attribute:
from max.pipelines.lib import ModuleV3PipelineModelWithKVCache
class YourModel(ModuleV3PipelineModelWithKVCache):
attention_bias: bool = TrueThe copied file defines _create_model_config(), _instantiate_module(), and
the compile step from the reference architecture. Each new parameter needs a
checkpoint entry, or compile() raises a KeyError. compile() leaves out a
checkpoint key with no parameter and raises no error, so compare your adapter's
keys with the module's parameter names.
Extend the compute graph
Subclassing overrides attributes or methods within an existing module tree.
Extending it changes the structure itself: how many normalization layers a block
applies, whether attention and MLP share their input, which parameters are
learnable. What those changes look like depends on your model's architecture.
max.nn provides the building
blocks, such as
RMSNorm,
LayerNorm,
Linear, and Embedding. Its common_layers package adds attention, MLP,
and rotary embedding layers. You can express most structural differences by
composing these existing layers. The examples show cases where a model's
structure differs from the reference and requires module-level changes.
Post-norm decoder block: The Llama 3 block normalizes each sublayer's
input. The olmo3 block normalizes each sublayer's output before the
residual add:
# Llama 3: pre-norm
h = x + self.self_attn(self.input_layernorm(x), kv_collection, ...)
out = h + self.mlp(self.post_attention_layernorm(h))
# OLMo 3: post-norm
h = x + self.post_attention_layernorm(self.self_attn(x, kv_collection, ...))
out = h + self.post_feedforward_layernorm(self.mlp(h))Parallel decoder block: Some architectures apply attention and MLP to the same normed input and add both to the residual stream in one step:
normed = self.input_layernorm(x)
x = x + self.self_attn(normed, ...) + self.mlp(normed)The parallel form requires one input_layernorm per block and a
single residual add.
When building any module that holds a list of layers, store them in
ModuleList.
ModuleList(layers) takes one iterable and names each layer by its numeric
index (0, 1, …), so parameter names resolve to layers.0.*, layers.1.*,
and so on.
Learnable LayerNorm: The llama3_modulev3 model builds
LayerNorm when
norm_method="layer_norm". By default, LayerNorm creates a learnable weight
and bias. Match its parameters to your checkpoint:
- If the checkpoint has a norm weight but no bias, pass
use_bias=False. Otherwise,compile()raises aKeyErrorfor the missing bias. - If the checkpoint has no norm parameters, pass
elementwise_affine=False. Theolmo_modulev3config sets this through itsnorm_elementwise_affinefield.
from max.nn.norm import LayerNorm
self.input_layernorm = LayerNorm(config.hidden_size, use_bias=False)With elementwise_affine=False, a norm weight in the checkpoint matches no
parameter, so MAX leaves it out without an error. The norm then runs with a
constant scale of 1.
When you reuse subcomponents from a reference architecture (attention layers,
normalization, MLP), check which parameters they define. Build the
subcomponent under F.lazy() and compare its parameters with your
checkpoint's keys before reusing it.
Implement custom layers
layers/ holds the individual building blocks that {model_name}.py composes
into the full architecture. Each file in layers/ implements one component as
a Module subclass. {model_name}.py describes how they connect, what the
residual stream looks like, and how the model loops over layers.
When your architecture requires multiple custom components (custom attention,
MoE routing, a non-standard transformer block), implement each in a separate
file in layers/. Keep {model_name}.py for the root module and the layer
loop.
A layer can call any operation in max.functional. For a graph op
or kernel that takes
TensorValue arguments,
F.functional(fn)
wraps the function into one that takes and returns Tensor values. The
functional_kernels module wraps its kernels this way.
Run on multiple GPUs
A multi-GPU architecture builds its module on a
DeviceMesh in
_instantiate_module(). Constructing the module inside
default_device(mesh)
creates each weight on the mesh. Module.to() moves a module to one device
only, and raises for a mesh with more than one device. The gemma3_modulev3
reference architecture builds a one-axis tensor-parallel mesh:
from max.tensor.sharding import DeviceMesh
from max.tensor import default_device
def _instantiate_module(self, model_config):
mesh = DeviceMesh(tuple(self.devices), (len(self.devices),), ("tp",))
with default_device(mesh):
return Gemma3(model_config, self.kv_params)Layers declare how their weights split across the mesh axes.
max.nn.common_layers.linear provides ColumnParallelLinear and
RowParallelLinear, plus col_parallel() and row_parallel() to shard an
existing Linear. VocabParallelEmbedding in max.nn.common_layers.embedding
splits the vocabulary. The sharding picker inserts the collectives that these
placements require. For an explicit collective, call
F.allreduce_sum(),
F.allgather(),
F.reduce_scatter(),
or
F.transfer_to().
The parallelism strategies cover how
MAX splits a served model across several GPUs. Inside an auto_reshard block
you can make each inserted collective visible, or refuse it, while you bring up
a model.
Set multi_gpu_supported=True in arch.py. For expert parallelism, the
deepseekV3_modulev3 pipeline model overrides _init_distributed_runtime()
to set up an EPBatchManager. Its mesh has one axis, named "dp" when the
data-parallel degree is above 1 and "tp" otherwise.
Expose the architecture
The arch.py file ties the rest of the package together. It includes one or
more
SupportedArchitecture
instances that point MAX at your config class, model class, weight adapters, and
more. Set name to the architectures[0] value from your model's
config.json:
from max.graph.weights import WeightsFormat
from max.pipelines.architectures.llama3_modulev3.batch_processor import (
Llama3ModuleV3BatchProcessor,
)
from max.pipelines.context import TextContext
from max.pipelines.lib import SupportedArchitecture, TextTokenizer
from max.pipelines.modeling.types import PipelineTask
from . import weight_adapters
from .model import YourModel
from .model_config import YourConfig
your_model_arch = SupportedArchitecture(
name="YourModelForCausalLM",
example_repo_ids=["your-org/your-model"],
task=PipelineTask.TEXT_GENERATION,
pipeline_model=YourModel,
config=YourConfig,
default_encoding=YourConfig.DEFAULT_ENCODING,
supported_encodings=YourConfig.SUPPORTED_ENCODINGS,
tokenizer=TextTokenizer,
context_type=TextContext,
default_weights_format=WeightsFormat.safetensors,
weight_adapters={
WeightsFormat.safetensors: weight_adapters.convert_safetensor_state_dict,
},
# Reuse the Llama 3 family's batch processor.
batching=Llama3ModuleV3BatchProcessor,
multi_gpu_supported=False,
supports_overlap_scheduler=False,
supports_device_graph_capture=False,
)After you create a SupportedArchitecture instance, expose it through your
package's __init__.py so MAX can register it when serving:
from .arch import your_model_arch
ARCHITECTURES = [your_model_arch]The ARCHITECTURES list is what MAX reads from a package you load with
--custom-architectures: that flag imports the module and registers every entry
it finds. A package that ships inside the repository also needs a _LazyArch
entry in
max/python/max/pipelines/architectures/__init__.py and an entry in
max/python/max/pipelines/architectures/all_arches.bzl. register_all_models()
registers the lazy_architectures entries lazily, and ALL_ARCHITECTURES fills
the deps of the entrypoints and test targets.
Most packages expose a single architecture, but you can list more than one when:
-
There are multiple tasks for the same model. The registry allows entries that share a name as long as their task differs (for example, a text generation architecture and an embedding architecture for the same model).
-
There are multiple Hugging Face class names. A single package with related variants whose
config.jsoncovers a model family might register entries for severalarchitectures[0]strings.
Next steps
With your architecture implemented, see Serve custom model architectures to run a generation, package your implementation, and load it at serve time.
If your model requires a kernel that MAX doesn't provide, see Custom ops for the interface, registration, and worked examples.
Once your architecture loads and generates text, verify its forward pass
numerically against a PyTorch reference. See
Logit comparison for quick verification and
Layer comparison to localize the first layer
that diverges when the logits don't match. To run that per-layer comparison with
an AI coding agent, use the
debug-model
skill.
Once the forward pass matches, confirm the model's task accuracy holds across a
full benchmark with the
eval-model
skill. It runs standard datasets against the served model and compares the
scores against the original model.