For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Model bring-up workflow
Model bring-up is the process of adapting a pretrained model from Hugging Face or PyTorch to run in MAX.
To bring a model to MAX, you must implement a model architecture. A model architecture is a Python package that defines the model's modules, maps its configuration, and loads its weights. Once implemented, you can serve the model with hardware-optimized kernels, production batching, KV cache management, and multi-GPU distribution.
This guide shows how to write a custom architecture by choosing a reference implementation, adapting its configuration and weights, and extending its modules where your model differs.
Architecture directory structure
Most text-generation model architectures follow this standard
directory layout, which the olmo3/ reference architecture
uses:
{your_model}/
├── layers/
├── __init__.py
├── arch.py
├── batch_processor.py
├── model_config.py
├── model.py
├── {model_name}.py
└── weight_adapters.pyEach file serves a distinct role:
layers/: holds custom layer implementations for components thatmax.experimental.nndoesn't provide.__init__.py: exposes theARCHITECTURESlist that MAX reads to register each entry.arch.py: registers the architecture with MAX. It declares the architecture's name, supported encodings, and the classes that implement the config, model, and weight adapter. The name must match thearchitectures[0]value in the model's Hugging Faceconfig.json. Models that use a KV cache should also set thememory_plannerfield (usuallyPagedMemoryPlanner) so MAX can size the KV cache and batch limits.batch_processor.py: optional batch processor that prepares model inputs for each scheduled batch and declares the input types your module compiles against. Create one only when the model has unique batching logic. If batching matches an existing family, register that family's processor through thebatchingfield inarch.py(for example,batching=Llama3ModuleV3BatchProcessor). See Delegating input preparation to a batch processor.model_config.py: translates Hugging Faceconfig.jsonfields into a typed configuration object that your modules read when MAX builds them.model.py: the pipeline model class registered inarch.py. It subclassesModuleV3PipelineModelWithKVCache, or the reference architecture's pipeline model when you build on one, and overrides the parts that differ.{model_name}.py: defines the root module and the transformer stack. Most models with unique sub-layers, an encoder, or a structure that differs from the generic transformer require this file.weight_adapters.py: translates Hugging Face checkpoint weight names to the parameter names of your MAX module.
The following sections describe how each component works and whether it needs customization.
Write modules with max.experimental.nn
Write each model component as a subclass of
Module from
max.experimental.nn. A module declares its
argument and return types in its signature, such as
Module[[Tensor, PagedCacheValues, Tensor, Tensor], tuple[Tensor, ...]], and
implements
forward()
over Tensor values.
Operations come from
max.experimental.functional, which you
import as F.
Layer constructors take shapes and options only, such as
Linear(in_dim, out_dim, bias=False)
or Embedding(vocab_size, dim=hidden_size). The pipeline sets the
dtype for the whole build, and your pipeline model places the module on a device
with .to(device).
The pipeline model compiles your module for serving. Its base class,
ModuleV3PipelineModelWithKVCache from
max.pipelines.lib, runs these steps in load_model():
- Loads the checkpoint and applies your weight adapter.
- Calls your
_create_model_config()to build the config object. - Calls your
_prepare_state_dict()to cast or rewrite weights that depend on the config. - Calls your
_init_distributed_runtime()to set up a distributed runtime. - Builds your module inside
F.lazy()with the config's dtype as the default, by calling your_instantiate_module().F.lazy()defers tensor realization while_instantiate_module()builds the module. - Compiles the module with
compile(*input_types, weights=state_dict). The batch processor supplies the input types, andcompile()supplies each parameter's value from the state dict by name.
The following model.py from the olmo3 reference architecture implements
both hooks. It also sets model_config_cls, the config class the pipeline
model uses to size the KV cache before load_model() runs. A
ModuleV3PipelineModelWithKVCache subclass that leaves it unset raises an
error at construction:
from typing import Any, ClassVar
from max.pipelines.lib import ModuleV3PipelineModelWithKVCache
class Olmo3Model(ModuleV3PipelineModelWithKVCache[TextContext]):
model_config_cls: ClassVar[type[Any]] = Olmo3Config
def _create_model_config(self, state_dict):
model_config = Olmo3Config.initialize(
self.pipeline_config, max_seq_len=self.max_seq_len
)
model_config.finalize(
huggingface_config=self.huggingface_config,
state_dict=state_dict,
return_logits=self.return_logits,
)
return model_config
def _instantiate_module(self, model_config):
nn_model = Olmo3(model_config, self.kv_params)
nn_model.to(self.devices[0])
return nn_modelDuring development, the same forward() also runs eagerly on real tensors.
For a short example of a module and its parameters, see
Neural network modules. To learn
the Tensor and F APIs on a small forward pass, see
Eager execution.
Choose a reference architecture
The reference architectures in MAX's codebase are complete implementations of
this package structure. Each one is a working implementation that max serve
can load. Browse the
architectures/ directory
alongside this guide to see how the files fit together in a finished
implementation.
Most large language models are variations on a small set of transformer blueprints. The attention mechanism, normalization placement, and residual structure in your model usually match an existing reference. Starting from the closest one means you inherit its modules, KV cache wiring, and kernel choices for everything you share.
Your implementation files define only what differs. model_config.py bridges
the Hugging Face config fields to MAX's representation. weight_adapters.py
remaps checkpoint keys to match MAX's naming. model.py subclasses the
reference pipeline model, and arch.py registers the quantization encodings
your checkpoint supports. If your model's structure is identical to a reference
and only those surface details differ, those files are all you need.
Choose a reference by its structural features:
llama3_modulev3/: Use for a standard dense transformer with GQA, RoPE, pre-RMSNorm, and SwiGLU. Its config also reads MuP scalars. It's the starting point for most models.olmo3/: Use when your model applies QK-norm across the full projection width and mixes sliding-window and full attention layers, with YaRN RoPE scaling.gemma3_modulev3/: Use when your model applies QK-norm per head, places RMSNorm both before and after each sublayer, and alternates local and global attention.phi3_modulev3/: Use when your checkpoint stores fusedqkv_projandgate_up_projweights. It subclasses the Llama 3 pipeline model and adds only a weight adapter.gpt_oss_modulev3/: Use when your model routes tokens to sparse Mixture-of-Experts (MoE) layers.deepseekV2_modulev3/: Use when your model uses Multi-Head Latent Attention (MLA), which compresses the KV cache using low-rank projections, on a single GPU.deepseekV3_modulev3/: Use when your model combines MLA with MoE routing and runs across GPUs with tensor, data, or expert parallelism.nemotron_h_modulev3/: Use when your model interleaves Mamba state-space layers with attention layers.
For vision-language models, start from gemma3multimodal_modulev3,
idefics3_modulev3, or kimik2_5_modulev3. Each reference organizes its
files differently. For example, the llama3_modulev3 module file is
llama3.py.
The sections below build on each other. Every LLM implementation requires config field mapping, weight naming, and KV cache integration. If your model's structure departs from the reference in a few attributes or methods, override the model components that differ. If the differences are structural, extend the module tree to implement those components from scratch. Examples include a different attention mechanism, custom routing, or a positional encoding variant that an attribute override can't express.
Map config fields
MAX reads a model's Hugging Face config.json through a typed configuration
object, and your modules read their sizes and options from that object. The
general pattern is to subclass the reference config class and set the fields
where the Hugging Face convention differs from MAX's representation. The
Files and versions tab on a Hugging Face model page includes the
config.json your config class needs to map. For a field-by-field guide to
reading a config and sorting its fields into the work each implies, see
Read a model config. Use the config to
identify which fields your subclass must set and which MAX field names they
correspond to:
{
"architectures": ["YourModelForCausalLM"],
"model_type": "your_model",
"hidden_size": 4096,
"num_attention_heads": 32,
"num_key_value_heads": 8,
"num_hidden_layers": 32,
"vocab_size": 256000,
"logit_scale": 0.0625
}A config class reads raw Hugging Face fields in its initialize() class
method, which receives the pipeline config and a required keyword-only
max_seq_len. Later, finalize() sets the fields that depend on the
checkpoint, such as whether the embeddings are tied. To extend a reference
config, call super().initialize() so the parent maps every field it knows,
then set the fields that differ.
The following example extends Llama3Config from llama3_modulev3. That
config already has a logits_scaling field, which the Llama 3 module divides
the logits by. This checkpoint stores the reciprocal as logit_scale:
from dataclasses import dataclass
from max.pipelines.architectures.llama3_modulev3.model_config import (
Llama3Config,
)
@dataclass(kw_only=True)
class YourConfig(Llama3Config):
@classmethod
def initialize(cls, pipeline_config, model_config=None, *, max_seq_len):
config = super().initialize(
pipeline_config, model_config, max_seq_len=max_seq_len
)
hf_config = (model_config or pipeline_config.model).huggingface_config
logit_scale = getattr(hf_config, "logit_scale", 1.0)
config.logits_scaling = 1.0 / logit_scale # 0.0625 → 16.0
return configThe pipeline model decides which config class to build. The llama3_modulev3
pipeline model reads it from a config_class attribute, and sizes the KV cache
with the class in model_config_cls, so your model.py points both
attributes at the new class:
from typing import Any, ClassVar
from max.pipelines.architectures.llama3_modulev3.model import Llama3Model
from .model_config import YourConfig
class YourModel(Llama3Model):
model_config_cls: ClassVar[type[Any]] = YourConfig
config_class = YourConfigOther reference pipeline models, such as olmo3, name their config class
directly inside _create_model_config(). To extend one of those, override
_create_model_config() in your subclass and call YourConfig.initialize()
there.
Map checkpoint weights
Every parameter of your MAX module has a name, a shape, and a dtype. The
name is the parameter's attribute path from the root module. Because the
olmo3 root module holds its transformer as self.language_model, its names
look like language_model.layers.0.self_attn.q_proj.weight. Checkpoint
weights rarely match those names, so you write a weight adapter function. The
function converts the checkpoint into the weights your module expects.
To list the names your module expects, build it under F.lazy() and read its
parameters:
from max.experimental import functional as F
with F.lazy():
model = YourRootModule(config, kv_params)
expected = {name for name, _ in model.parameters}compile() checks the adapted weights against these parameters:
- A parameter with no matching entry raises a
KeyError. - An entry whose shape or dtype differs from its parameter raises a
ValueError. Passingauto_cast=Truetocompile()permits casts betweenfloat32andbfloat16. - A checkpoint entry that matches no parameter raises no error, and
compile()leaves it out. Compare the adapter's output keys withexpectedto catch weights your module never loads.
For models with tied embeddings, the module creates no lm_head parameter,
and forward() computes the logits as h @ self.embed_tokens.weight.T. If the
checkpoint still stores an lm_head.weight entry, the entry matches no
parameter, and compile() leaves it out.
Renaming is the most common transformation needed in a weight adapter. You might also need to cast dtypes, drop weights the module doesn't use, and transform tensor layouts. To learn how to implement these transformations, see Write a weight adapter.
Connect to the KV cache
MAX pre-allocates the KV cache at startup from values your config provides,
before the first request arrives. The allocation depends on the number of KV
heads, the head dimension, the number of layers, and the maximum sequence
length. Your config class supplies these values by subclassing
ArchConfigWithStoredKVParams and
ArchConfigWithKVCache.
ArchConfigWithStoredKVParams returns your config's kv_params field from
get_kv_params(), which MAX calls to size the cache. It
returns the max_seq_len field from get_max_seq_len(), so set that field
from the max_seq_len argument that initialize() receives. Your
initialize() fills kv_params by calling the
construct_kv_params() class method. The default
implementation reads these Hugging Face fields:
num_key_value_heads: total KV heads across all devices. For grouped-query attention models, this field differs fromnum_attention_heads. MAX sizes the cache fromnum_key_value_heads.head_dim, throughget_head_dim(): dimensionality of each attention head. When the config has nohead_dim, it'shidden_size // num_attention_heads.num_hidden_layers, throughget_num_layers(): number of hidden layers.
Override get_head_dim(), get_num_layers(), or construct_kv_params() when
your Hugging Face config names these values differently. MAX sizes the cache
with the overrides on your pipeline model's model_config_cls, and memory
planning uses the config class in arch.py, so point both at your config
class. The kv_params that initialize() stores must come from the same
overrides. Some reference configs call their own class by name:
Llama3Config.initialize() calls Llama3Config.construct_kv_params(). If you
subclass one of those, recompute kv_params in your initialize() after the
super() call. The following example shows the shape of the default
implementation:
@classmethod
def construct_kv_params(
cls,
huggingface_config: AutoConfig,
pipeline_config: PipelineConfig,
devices: list[DeviceRef],
kv_cache_config: KVCacheConfig,
cache_dtype: DType,
*,
allow_kv_head_replication: bool = False,
) -> KVCacheParamInterface:
return kv_cache_config.to_params(
dtype=cache_dtype,
allow_kv_head_replication=allow_kv_head_replication,
n_kv_heads=huggingface_config.num_key_value_heads,
head_dim=cls.get_head_dim(huggingface_config),
num_layers=cls.get_num_layers(huggingface_config),
devices=devices,
data_parallel_degree=pipeline_config.model.data_parallel_degree,
)Most architectures return a flat
KVCacheParams.
Architectures with multiple or hybrid KV caches (for example, separate
sliding-window and full-attention caches) return a
MultiKVCacheParams
tree, which the cache manager consumes through the shared
KVCacheParamInterface.
Pass the cache into the module
The compiled module receives the KV cache as flat inputs after the token
inputs. The root module's forward() collects them in *variadic_args and
unflattens them with its kv_params. For a single cache, call
self.kv_params.unflatten_kv_inputs(). For a sliding-window and global
cache pair, call unflatten_basic_kv_tree(). The olmo3 root module wraps
each cache in PagedCacheValues from
max.experimental.nn.common_layers.kv_cache:
class Olmo3(Module[[Tensor, Tensor, Tensor], tuple[Tensor]]):
def __init__(self, config: Olmo3Config, kv_params: KVCacheParamInterface):
super().__init__()
# ...
self.language_model = Olmo3TextModel(config)
self.kv_params = kv_params
def forward(
self,
tokens: Tensor,
return_n_logits: Tensor,
input_row_offsets: Tensor,
*variadic_args: Tensor,
) -> tuple[Tensor]:
kv_inputs = iter(x._graph_value for x in variadic_args)
sliding_inputs, global_inputs = self.kv_params.unflatten_basic_kv_tree(
kv_inputs
)
sliding_kv = PagedCacheValues.from_upstream(
sliding_inputs, tokens.mapping
)
global_kv = PagedCacheValues.from_upstream(
global_inputs, tokens.mapping
)
return self.language_model(
tokens, sliding_kv, global_kv, return_n_logits, input_row_offsets
)Each attention layer stores its layer index and receives a PagedCacheValues.
It calls fused kernels from
max.experimental.nn.common_layers.functional_kernels:
rope_split_store_ragged() applies RoPE and writes keys
and values to the cache, and flash_attention_ragged() reads
them back. MAX manages the paged memory behind the cache, using the settings in
KVCacheConfig.
Override model components
Overriding model components builds on config mapping and weight translation from
the previous sections. The addition is a model.py that subclasses the
reference pipeline model and overrides only the components that differ. Hugging
Face model implementations already follow this pattern: in the transformers
library, many model classes subclass a family base and override only what
changed. The same approach applies to MAX's reference architectures.
Subclass the reference model
model.py imports and subclasses the reference pipeline model. Override only
the class attributes or methods that differ from the reference. The
llama3_modulev3 pipeline model exposes config_class, norm_method, and
attention_bias as class attributes. For example, a model that matches Llama
3 except for bias on the attention projections needs a single class
attribute:
from max.pipelines.architectures.llama3_modulev3.model import Llama3Model
class YourModel(Llama3Model):
attention_bias: bool = TrueThe subclass inherits _create_model_config(), _instantiate_module(), and
the compile step from the reference architecture. Each new parameter needs a
checkpoint entry, or compile() raises a KeyError. compile() leaves out a
checkpoint key with no parameter and raises no error, so compare your adapter's
keys with the module's parameter names.
Extend the compute graph
Subclassing overrides attributes or methods within an existing module tree.
Extending it changes the structure itself: how many normalization layers a
block applies, whether attention and MLP share their input, which parameters are
learnable. What those changes look like depends on your model's
architecture. max.experimental.nn provides
the building blocks, such as RMSNorm, LayerNorm,
Linear, and Embedding. Its common_layers package adds attention, MLP,
and rotary embedding layers. You can express most structural differences by
composing these existing layers. The examples below show cases where a
model's structure differs from the reference and requires module-level
changes.
Post-norm decoder block: The Llama 3 block normalizes each sublayer's
input. The olmo3 block normalizes each sublayer's output before the
residual add:
# Llama 3: pre-norm
h = x + self.self_attn(self.input_layernorm(x), kv_collection, ...)
out = h + self.mlp(self.post_attention_layernorm(h))
# OLMo 3: post-norm
h = x + self.post_attention_layernorm(self.self_attn(x, kv_collection, ...))
out = h + self.post_feedforward_layernorm(self.mlp(h))Parallel decoder block: Some architectures apply attention and MLP to the same normed input and add both to the residual stream in one step:
normed = self.input_layernorm(x)
x = x + self.self_attn(normed, ...) + self.mlp(normed)The parallel form requires one input_layernorm per block and a
single residual add.
When building any module that holds a list of layers, store them in
ModuleList.
ModuleList(layers) takes one iterable and names each layer by its numeric
index (0, 1, …), so parameter names resolve to layers.0.*, layers.1.*,
and so on.
Learnable LayerNorm: The llama3_modulev3 model builds
LayerNorm when norm_method="layer_norm". By default,
LayerNorm creates a learnable weight and bias. Match its parameters to
your checkpoint:
- If the checkpoint has a norm weight but no bias, pass
use_bias=False. Otherwise,compile()raises aKeyErrorfor the missing bias. - If the checkpoint has no norm parameters, pass
elementwise_affine=False. Theolmo_modulev3config sets this through itsnorm_elementwise_affinefield.
from max.experimental.nn.norm import LayerNorm
self.input_layernorm = LayerNorm(config.hidden_size, use_bias=False)With elementwise_affine=False, a norm weight in the checkpoint matches no
parameter, so MAX leaves it out without an error. The norm then runs with a
constant scale of 1.
When you reuse subcomponents from a reference architecture (attention layers,
normalization, MLP), check which parameters they define. Build the
subcomponent under F.lazy() and compare its parameters with your
checkpoint's keys before reusing it.
Implement custom layers
layers/ holds the individual building blocks that {model_name}.py composes
into the full architecture. Each file in layers/ implements one component as
a Module subclass. {model_name}.py describes how they connect, what the
residual stream looks like, and how the model loops over layers.
When your architecture requires multiple custom components (custom attention,
MoE routing, a non-standard transformer block), implement each in a separate
file in layers/. Keep {model_name}.py for the root module and the layer
loop.
A layer can call any operation in max.experimental.functional. For a graph
op or kernel that takes TensorValue arguments,
F.functional(fn) wraps the function into one that takes and
returns Tensor values. The functional_kernels module wraps its kernels
this way.
Run on multiple GPUs
A multi-GPU architecture builds its module on a
DeviceMesh
in _instantiate_module(). Constructing the module inside
default_device(mesh) creates each weight on the mesh.
Module.to() moves a module to one device only, and raises for a mesh with
more than one device. The gemma3_modulev3 reference architecture builds a
one-axis tensor-parallel mesh:
from max.experimental.sharding import DeviceMesh
from max.experimental.tensor import default_device
def _instantiate_module(self, model_config):
mesh = DeviceMesh(tuple(self.devices), (len(self.devices),), ("tp",))
with default_device(mesh):
return Gemma3(model_config, self.kv_params)Layers declare how their weights split across the mesh axes.
max.experimental.nn.common_layers.linear provides ColumnParallelLinear
and RowParallelLinear, plus col_parallel() and row_parallel() to shard
an existing Linear. VocabParallelEmbedding in
max.experimental.nn.common_layers.embedding splits the vocabulary. The
sharding solver inserts the collectives that these placements require. For
an explicit collective, call F.allreduce_sum(),
F.allgather(), F.reduce_scatter(), or
F.transfer_to().
Set multi_gpu_supported=True in arch.py. For expert parallelism, the
deepseekV3_modulev3 pipeline model overrides _init_distributed_runtime()
to set up an EPBatchManager. Its mesh has one axis, named "dp" when the
data-parallel degree is above 1 and "tp" otherwise.
Expose the architecture
The arch.py file ties the rest of the package together. It should include one
or more
SupportedArchitecture
instances that point MAX at your config class, model class, weight adapters, and
more. Set name to the architectures[0] value from your model's
config.json:
from max.graph.weights import WeightsFormat
from max.pipelines.architectures.llama3_modulev3.batch_processor import (
Llama3ModuleV3BatchProcessor,
)
from max.pipelines.context import TextContext
from max.pipelines.kv_cache.memory_planner import PagedMemoryPlanner
from max.pipelines.lib import SupportedArchitecture, TextTokenizer
from max.pipelines.modeling.types import PipelineTask
from . import weight_adapters
from .model import YourModel
from .model_config import YourConfig
your_model_arch = SupportedArchitecture(
name="YourModelForCausalLM",
example_repo_ids=["your-org/your-model"],
task=PipelineTask.TEXT_GENERATION,
pipeline_model=YourModel,
config=YourConfig,
default_encoding=YourConfig.DEFAULT_ENCODING,
supported_encodings=YourConfig.SUPPORTED_ENCODINGS,
tokenizer=TextTokenizer,
context_type=TextContext,
default_weights_format=WeightsFormat.safetensors,
weight_adapters={
WeightsFormat.safetensors: weight_adapters.convert_safetensor_state_dict,
},
# Reuse the Llama 3 family's batch processor and the paged memory planner.
batching=Llama3ModuleV3BatchProcessor,
memory_planner=PagedMemoryPlanner,
multi_gpu_supported=False,
supports_overlap_scheduler=False,
supports_device_graph_capture=False,
)After you create a SupportedArchitecture instance, expose it through your
package's __init__.py so MAX can register it when serving:
from .arch import your_model_arch
ARCHITECTURES = [your_model_arch]Most packages expose a single architecture, but you can list more than one when:
-
There are multiple tasks for the same model. The registry allows entries that share a name as long as their task differs (for example, a text generation architecture and an embedding architecture for the same model).
-
There are multiple Hugging Face class names. A single package with related variants whose
config.jsoncovers a model family might register entries for severalarchitectures[0]strings.
Next steps
With your architecture implemented, see Serve custom model architectures to run a generation, package your implementation, and load it at serve time.
If your model requires a kernel that MAX doesn't provide, see Custom ops for the interface, registration, and worked examples.
Once your architecture loads and generates text, verify its forward pass
numerically against a PyTorch reference. See
Logit comparison for quick verification and
Layer comparison to localize the first layer
that diverges when the logits don't match. To run that per-layer comparison with
an AI coding agent, use the
debug-model
skill.
Once the forward pass matches, confirm the model's task accuracy holds across a
full benchmark with the
eval-model
skill. It runs standard datasets against the served model and compares the
scores against the original model.