For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Weight adapters
Most model architectures need a weight adapter. Write a weight adapter to
convert checkpoint weights (most often published on Hugging Face) into the
format expected by your MAX
Module. Then, integrate
the weight adapter into the model pipeline.
What a weight adapter does
A weight adapter is a function you define in weight_adapters.py that converts
checkpoint weights into a state dictionary that MAX compiles into your
Module.
Depending on how the checkpoint differs from your implementation, a weight adapter can perform one or more of the following transformations:
- Rename keys to match the parameter names expected by the
Module. In many architectures, renaming keys is the only transformation required. - Cast dtypes to the types expected by the model.
- Remove weights that exist only in the checkpoint and not in the
Module. - Transform tensor layouts by reshaping, reordering, or repacking tensors into the format expected by the model.
The following sections show how to implement each transformation.
When the weight adapter runs
The following steps show when the weight adapter runs within the entire weight loading flow:
- The pipeline calls
load_weights()to load the checkpoint into aWeightsobject. - The pipeline calls
weights_format()to detect the checkpoint format from the weight file extensions. - The pipeline retrieves the registered weight adapter for that format.
- The pipeline constructs your
pipeline model with
weightsandadapteras arguments. Your pipeline model subclassesModuleV3PipelineModelWithKVCache, whose constructor builds the decoder throughload_graphs(). load_graphs()loads the checkpoint, and passes its entries to the weight adapter along with thehuggingface_configandpipeline_configkeyword arguments.- The weight adapter renames and converts the tensors and returns a
dict[str, WeightData]. load_graphs()builds the model config with_create_model_config().graph_specs()calls_prepare_state_dict(), a hook where your pipeline model can cast or rewrite weights that depend on the config, and_init_distributed_runtime().graph_specs()constructs yourModulewith_instantiate_module()inside anF.lazy()block, which defers tensor realization.compile_and_init_graphs()traces and compiles each graph throughModule.stage(), whichcompile()wraps. Each parameter of theModuletakes the state dictionary entry with the same name.
Checkpoint and module weight formats
Before you write a weight adapter, inspect both the checkpoint and your Module
to understand what transformations you need to implement to make the checkpoint
compatible.
Inspect the checkpoint
To inspect the checkpoint, pass its path to
load_weights()
and iterate over the entries to print each tensor's name, shape, and dtype:
from pathlib import Path
from max.graph.weights import load_weights
weights = load_weights([Path("/path/to/model.safetensors")]) # or .gguf
for name, weight in weights.items():
data = weight.data()
shape = list(int(d) for d in data.shape)
print(f"{name}: {shape} {data.dtype}")The output may look similar to the following:
model.embed_tokens.weight: [vocab_size, hidden_size] DType.bfloat16
model.layers.0.input_layernorm.weight: [hidden_size] DType.bfloat16
model.layers.0.self_attn.q_proj.weight: [hidden_size, hidden_size] DType.bfloat16
...Some checkpoints store their weights in multiple shard files. Inspect the
shards by passing multiple paths to load_weights():
sharded_paths = [
Path("model-00001-of-00003.safetensors"),
Path("model-00002-of-00003.safetensors"),
Path("model-00003-of-00003.safetensors"),
]
weights = load_weights(sharded_paths)During serving, MAX finds and merges shards for you.
Inspect the module parameters
Each parameter name is the
attribute path from the root
Module to the parameter tensor, joined with dots. To list them, construct your
Module inside an F.lazy() block and iterate over its
parameters
property:
from max.dtype import DType
from max import functional as F
from max.tensor import default_dtype
# Construct the root module the way your _instantiate_module() does.
with F.lazy(), default_dtype(DType.bfloat16):
model = MyModel(config, kv_params)
for name, param in model.parameters:
shape = [int(d) for d in param.shape]
print(f"{name}: {shape} {param.dtype}")F.lazy() defers tensor realization, so constructing the module doesn't
materialize its initial parameter values.
default_dtype()
sets the dtype for parameters that don't declare one, as load_model() does
with the model config's dtype.
For an OLMo 3 model, the output may look similar to the following:
language_model.embed_tokens.weight: [vocab_size, hidden_size] DType.bfloat16
language_model.norm.weight: [hidden_size] DType.bfloat16
...
language_model.layers.0.self_attn.q_proj.weight: [hidden_size, hidden_size] DType.bfloat16
...
language_model.layers.0.self_attn.q_norm.weight: [hidden_size] DType.bfloat16
...The language_model. prefix comes from the root module. The Olmo3 root
module stores its text model in a self.language_model attribute, so every
parameter path starts there.
Name mapping
The most common job of the weight adapter is to map checkpoint parameter names
to the names expected by your MAX Module.
To do this, define a weight adapter function that:
- Accepts the checkpoint state dictionary.
- Iterates over the checkpoint state dictionary.
- Computes the parameter name to match your
Module. - Builds and returns a new state dictionary that maps the new parameter names to the original weight tensors.
In many cases, the mapping is straightforward because both Hugging Face and MAX derive parameter names from a module hierarchy. If the model architectures are similar, the adapter may only need to replace a common prefix or rename a small number of modules.
For example, Hugging Face safetensors checkpoints for OLMo 3 store the decoder
weights under a top-level model. prefix. The MAX Olmo3 module nests the same
layers under language_model.. The following adapter rewrites each prefix:
OLMO3_SAFETENSOR_MAP: dict[str, str] = {
"model.embed_tokens.": "language_model.embed_tokens.",
"model.norm.": "language_model.norm.",
"lm_head.": "language_model.lm_head.",
"model.layers.": "language_model.layers.",
}
def convert_safetensor_state_dict(
state_dict: dict[str, Weights],
huggingface_config: AutoConfig | None = None,
pipeline_config: PipelineConfig | None = None,
) -> dict[str, WeightData]:
"""Convert safetensor state dict to MAX format.
Args:
state_dict: Dictionary of weight tensors
Returns:
Dictionary of converted weight data
"""
new_state_dict: dict[str, WeightData] = {}
for weight_name, value in state_dict.items():
max_name: str = weight_name
for before, after in OLMO3_SAFETENSOR_MAP.items():
max_name = max_name.replace(before, after)
new_state_dict[max_name] = value.data()
return new_state_dictload_model() calls the adapter with the checkpoint state dictionary as the
first argument, plus huggingface_config and pipeline_config as keyword
arguments. An adapter that doesn't read them can accept **unused_kwargs
in their place.
Dtype casting
Every parameter in a Module has an expected dtype. You set that dtype when
you construct the Module, matching the dtype that the MAX kernel consuming
the weight requires. A checkpoint might store the same weight in a different
dtype than the kernel expects.
A pipeline model casts each weight your adapter returns between float32 and
bfloat16 to its parameter's dtype, unless MODULAR_AUTO_CAST_WEIGHTS is
false (environment variables). Cast any other
dtype mismatch in your adapter. Otherwise compile() raises a ValueError that
reports the loaded tensor's shape and dtype next to the parameter's. compile()
raises the same error when the shapes differ.
A cast that depends on the model config belongs in your pipeline model's
_prepare_state_dict() hook, which receives the state dictionary and the
config. Defer layer-level encoding decisions to
QuantFormat and
QuantConfig, which
configure the quantized layers in the module.
Weights the module does not use
A checkpoint and your Module don't always contain the same set of weights. A
mismatch can occur in either direction:
- The checkpoint contains weights that your
Moduledoesn't declare. Remove these in the weight adapter. - Your
Moduledeclares weights that aren't present in the checkpoint.compile()raises aKeyErrorthat names the first parameter it can't find. Fix it in the module definition or the name mapping.
compile() ignores state dictionary entries that match no parameter, and it
doesn't log a warning. Remove them in the adapter, so a
key comparison lists only naming
errors.
For example, FP8 and NVFP4 checkpoints often include .k_scale and .v_scale
tensors used to quantize the key and value caches. MAX reads KV cache scales
from a separate configuration path, so every DeepSeek-V3 adapter path skips
them, for example the bf16 path:
def _convert_bf16_state_dict(
state_dict: dict[str, Weights],
) -> dict[str, WeightData]:
"""Adapt a bf16 DeepseekV3 checkpoint to the V3 module hierarchy."""
out: dict[str, WeightData] = {}
for name, value in state_dict.items():
max_name = _remap_bf16_prefix(name)
max_name = _strip_weight_suffix_for_raw_mla(max_name)
# ...
if max_name.endswith((".k_scale", ".v_scale")):
continue
out[max_name] = value.data()
return outThe same adapter also deletes the multi-token prediction (MTP) layer, which the DeepSeek-V3 checkpoint stores as one extra decoder layer after the last one the module builds:
mtp_layer_idx = huggingface_config.num_hidden_layers
mtp_prefix = f"language_model.layers.{mtp_layer_idx}."
for key in list(new_state_dict.keys()):
if key.startswith(mtp_prefix):
del new_state_dict[key]A checkpoint can also store a value that MAX computes at runtime. GGUF
checkpoints bake the RoPE frequencies into a rope_freqs.weight tensor, so
the llama3_modulev3 GGUF adapter removes it:
# GGUF bakes `rope_freqs` into the weights file; we compute them at runtime.
new_state_dict.pop("rope_freqs.weight", None)Tied embeddings produce the same kind of extra entry. When
tie_word_embeddings is true, the Llama3TextModel and Olmo3TextModel
modules set self.lm_head = None. They compute logits from the embedding
table, as in h @ self.embed_tokens.weight.T. The module has no lm_head
parameter, so an lm_head.weight entry in the checkpoint matches nothing.
Tensor transformations
Transform tensor values only when the checkpoint representation differs from the
format expected by a MAX kernel. When possible, adapt your Module and leave
the checkpoint data unchanged. Tensor transformations encode assumptions about
a specific kernel implementation, and you may need to update them if the kernel
interface changes.
Common cases include:
- Partial-RoPE permutation. Some models use a partial-RoPE layout where only a subset of attention dimensions participate in rotary embeddings. If the checkpoint layout differs from the layout expected by the MAX kernel, the adapter must permute the query and key projection weights before loading.
- Quantization-specific transformations. Quantized checkpoints may store weights
in a layout that differs from the dequantization kernel's expected format. For
example, MXFP4 checkpoints require weight repacking, and GPTQ checkpoints
store a
g_idxpermutation that you must convert to the inverseperm_idxrepresentation expected by the kernel.
The following example permutes the rows of each attention head in the query and
key projection weights, the transformation a partial-RoPE layout needs. It
follows
convert_step3p5_state_dict.
NumPy has no bfloat16 type, so the adapter views each bfloat16
Buffer as uint16 while it
reorders rows, then views the result as bfloat16 again:
import numpy as np
from max.driver import Buffer
from max.dtype import DType
from max.graph import Shape
from max.graph.weights import WeightData, Weights
from transformers import AutoConfig
SAFETENSOR_MAPPING = {"model.": "language_model."}
def permute_head_rows(
weight: WeightData, perm: list[int], num_heads: int, head_dim: int
) -> WeightData:
buf = Buffer.from_dlpack(weight.data)
if weight.dtype == DType.bfloat16:
buf = buf.view(DType.uint16)
rows = buf.to_numpy().reshape(num_heads, head_dim, -1)
rows = np.ascontiguousarray(rows[:, perm, :]).reshape(buf.shape)
out = Buffer.from_numpy(rows).view(weight.dtype, list(rows.shape))
return WeightData(out, weight.name, weight.dtype, Shape(rows.shape))
def convert_safetensor_state_dict(
state_dict: dict[str, Weights],
huggingface_config: AutoConfig,
**unused_kwargs,
) -> dict[str, WeightData]:
num_heads = huggingface_config.num_attention_heads
head_dim = huggingface_config.hidden_size // num_heads
num_kv_heads = getattr(
huggingface_config, "num_key_value_heads", num_heads
)
rope_perm = build_rope_perm(head_dim) # architecture-specific helper
new_state_dict: dict[str, WeightData] = {}
for checkpoint_name, value in state_dict.items():
max_name = checkpoint_name
for before, after in SAFETENSOR_MAPPING.items():
max_name = max_name.replace(before, after)
data = value.data()
if max_name.endswith("q_proj.weight"):
data = permute_head_rows(data, rope_perm, num_heads, head_dim)
elif max_name.endswith("k_proj.weight"):
data = permute_head_rows(data, rope_perm, num_kv_heads, head_dim)
new_state_dict[max_name] = data
return new_state_dictCompare adapter keys with parameter names
Compare the keys produced by the weight adapter with the parameter names your
Module declares. The following script runs the adapter on a local checkpoint
and prints both differences:
from pathlib import Path
from max.dtype import DType
from max import functional as F
from max.tensor import default_dtype
from max.graph.weights import load_weights
from transformers import AutoConfig
from my_arch.weight_adapters import convert_safetensor_state_dict
weights = load_weights([Path("/path/to/model.safetensors")])
hf_config = AutoConfig.from_pretrained("org/model-id")
adapted = convert_safetensor_state_dict(
dict(weights.items()),
huggingface_config=hf_config,
)
with F.lazy(), default_dtype(DType.bfloat16):
model = MyModel(config, kv_params)
expected = {name for name, _ in model.parameters}
print("missing:", sorted(expected - adapted.keys()))
print("unused:", sorted(adapted.keys() - expected))Every name under missing makes compile() raise a KeyError. Fix those
first. Names under unused don't stop compilation, and most of them point to
a weight the adapter renames or removes. Expected entries include:
- A tied
lm_head.weight, when the module reuses the embedding weights. - Format-specific tensors that your
Moduledoesn't use.
Register the weight adapter
To wire the weight adapter function into the pipeline, add it to the
SupportedArchitecture
definition in arch.py, keyed by the corresponding
WeightsFormat. The
Qwen2 ModuleV3 architecture
(qwen2_modulev3_arch)
registers its safetensors adapter:
weight_adapters={
WeightsFormat.safetensors: weight_adapters.convert_safetensor_state_dict,
},An architecture that loads more than one format registers one adapter per
format. For example, llama3_modulev3/arch.py also maps WeightsFormat.gguf
to convert_gguf_state_dict().
Next steps
Once your adapter loads successfully, complete the remaining model bring-up steps:
- Implement the pipeline model: The pipeline model class connects weight loading, module construction, compilation, and execution.
- Configure quantization support: If your checkpoint
uses a quantized format, implement the corresponding
QuantConfigandQuantFormatintegration.
For the complete workflow, see Model bring-up workflow.