IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Supported layers and mechanisms

Every architecture you port differs from the last one in a handful of places: a different rotary schedule, a norm with an offset, a fused checkpoint layout. Most of those differences are constructor options on a layer in max.nn. This page maps the common mechanisms to the layer that implements them, so you can find the setting before you write code.

Mechanisms and the built-in layersโ€‹

Every row in the table is a constructor option on a built-in layer. Subclassing is for the cases no row covers.

A built-in layer runs on a device mesh without changes to the layer: its weights take the placement the mesh gives them, and the ops that read them plan around it. Tensor parallelism covers how MAX splits a served model across the GPUs you name.

MechanismLayerOption or input
Rotary embeddingRotaryEmbeddingtheta, head_dim, max_seq_len
Rotate-half rotaryRotaryEmbeddinginterleaved=False (default True)
YaRN-scaled rotaryYarnRotaryEmbeddingscaling_params
Banded scaling (Llama 3, LongRoPE)RotaryEmbedding subclassoverride freqs_cis_base()
No positional encoding (NoPE)noneassemble attention without a rope layer
QK normalizationAttentionWithRopeuse_qk_norm=True, rms_norm_eps
Fused QKV checkpointAttentionWithRopestacked_qkv=True
QKV clippingAttentionWithRopeclip_qkv, in weight units
Attention scaleAttentionWithRopescale, defaulting to 1/sqrt(head_dim)
Projection biasAttentionWithRopehas_bias=True
RMS normalizationRMSNormdim, eps
Offset RMS normalizationGemmaRMSNormdim, eps
Layer normalizationLayerNormdim, eps
Quantized projectionLinearquant_config, static per-tensor FP8
KV cache shapeAttentionWithRopekv_params from max.nn.kv_cache
Tied input and output embeddingmodel, not a layerlogits computed against the embedding weight

AttentionWithRope and the latent attention layers mask causally. A model that needs another mask subclasses the attention layer.

The rotary layer builds its frequency table from theta and head_dim, and caches it on first use. A model with a modified frequency schedule subclasses the layer and overrides freqs_cis_base(), which leaves the rotation, the pairing, and the parameter names intact. See RotaryEmbedding for the parameters that build the table.

When no option fitsโ€‹

A mechanism that isn't in this table becomes code. Pick the smallest change that carries it:

  • Subclass the built-in layer when you change one behavior inside an existing layer, such as a norm that multiplies before casting. See Subclass the closest layer.
  • Write the layer when the mechanism is new, such as a router or a novel mixer. See Define a layer from scratch.
  • Wrap a kernel when the computation exists as Mojo and you need it inside the module. See Wrap a custom op.

Each of those keeps the module contract: named parameters, a forward() that runs eagerly, a load_state_dict() that reports missing and unexpected keys, and a compile() that captures the whole module as one graph.

Next stepsโ€‹

Was this page helpful?