For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Supported layers and mechanisms
Every architecture you port differs from the last one in a handful of places:
a different rotary schedule, a norm with an offset, a fused checkpoint layout.
Most of those differences are constructor options on a layer in
max.nn. This page maps the common
mechanisms to the layer that implements them, so you can find the setting before
you write code.
Mechanisms and the built-in layersโ
Every row in the table is a constructor option on a built-in layer. Subclassing is for the cases no row covers.
A built-in layer runs on a device mesh without changes to the layer: its weights take the placement the mesh gives them, and the ops that read them plan around it. Tensor parallelism covers how MAX splits a served model across the GPUs you name.
| Mechanism | Layer | Option or input |
|---|---|---|
| Rotary embedding | RotaryEmbedding | theta, head_dim, max_seq_len |
| Rotate-half rotary | RotaryEmbedding | interleaved=False (default True) |
| YaRN-scaled rotary | YarnRotaryEmbedding | scaling_params |
| Banded scaling (Llama 3, LongRoPE) | RotaryEmbedding subclass | override freqs_cis_base() |
| No positional encoding (NoPE) | none | assemble attention without a rope layer |
| QK normalization | AttentionWithRope | use_qk_norm=True, rms_norm_eps |
| Fused QKV checkpoint | AttentionWithRope | stacked_qkv=True |
| QKV clipping | AttentionWithRope | clip_qkv, in weight units |
| Attention scale | AttentionWithRope | scale, defaulting to 1/sqrt(head_dim) |
| Projection bias | AttentionWithRope | has_bias=True |
| RMS normalization | RMSNorm | dim, eps |
| Offset RMS normalization | GemmaRMSNorm | dim, eps |
| Layer normalization | LayerNorm | dim, eps |
| Quantized projection | Linear | quant_config, static per-tensor FP8 |
| KV cache shape | AttentionWithRope | kv_params from max.nn.kv_cache |
| Tied input and output embedding | model, not a layer | logits computed against the embedding weight |
AttentionWithRope and the latent attention layers mask causally. A model
that needs another mask subclasses the attention layer.
The rotary layer builds its frequency table from theta and head_dim, and
caches it on first use. A model with a modified frequency schedule subclasses
the layer and overrides freqs_cis_base(), which leaves the rotation, the
pairing, and the parameter names intact. See
RotaryEmbedding
for the parameters that build the table.
When no option fitsโ
A mechanism that isn't in this table becomes code. Pick the smallest change that carries it:
- Subclass the built-in layer when you change one behavior inside an existing layer, such as a norm that multiplies before casting. See Subclass the closest layer.
- Write the layer when the mechanism is new, such as a router or a novel mixer. See Define a layer from scratch.
- Wrap a kernel when the computation exists as Mojo and you need it inside the module. See Wrap a custom op.
Each of those keeps the module contract:
named parameters, a forward()
that runs eagerly, a load_state_dict() that reports missing and unexpected
keys, and a compile() that captures the whole module as one graph.
Next stepsโ
- Layers: how to add a layer the built-in set doesn't cover.
- Modules: weights, tracing, and compilation.
- Quantize a model: what
quant_configaccepts. - Read a model config: where these options come from in a Hugging Face config.