For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Model pipeline
Implementing a model architecture in MAX involves more than defining the layers and the forward pass. To serve your model, you must also integrate it into a pipeline that manages the entire inference lifecycle.
A pipeline connects a model implementation to MAX's serving system. It's responsible for tasks such as preparing inputs, executing the model, managing inference state such as the KV cache, performing post-processing, and returning results.
A pipeline is built from components, each behind an interface. Most architectures reuse the standard components; this page shows how they work together and when to replace one.
The pipeline system
The pipeline system uses a registry pattern where model architectures register their capabilities, and the infrastructure handles the execution details. When you point MAX at a model, the registry looks up the architecture, validates compatibility, downloads weights, compiles the model, and returns a ready-to-use pipeline.
This separates concerns cleanly:
- Modules define model architectures and hold weights.
- Pipelines orchestrate the inference loop and manage state.
- Registry maps model identifiers to implementations.
- Compilation transforms your model into optimized executables for the target device.
Pipelines let you focus on model architecture while MAX handles the production infrastructure, including batching, caching, compilation, and serving.
The packages pipelines build on
Pipelines build on max.nn.legacy,
max.nn, and
max.pipelines.kv_cache.
Neural network modules
MAX architectures build their layers with
max.nn, the module API for
ModuleV3 architectures. It includes common components like:
- Core modules:
Module,Linear,Embedding,Sequential,ModuleList. - Normalization:
RMSNorm,GemmaRMSNorm, andLayerNorm. - Positional encodings:
RotaryEmbeddingandTransposedRotaryEmbedding. - Attention:
AttentionWithRope.
The max.nn.legacy package builds MAX graphs directly
and backs architectures written against it. To port one to ModuleV3, see
Migrate an existing architecture to ModuleV3.
The Module base class in
max.nn standardizes how modules manage weights and devices.
Here's an example of building a simple multi-layer perceptron with the
ModuleV3 API:
from max.driver import Accelerator
from max.nn import Linear, Module, module_dataclass
@module_dataclass
class MLP(Module):
fc1: Linear
fc2: Linear
def forward(self, x):
return self.fc2(self.fc1(x))
# Create a model with two linear layers
model = MLP(fc1=Linear(10, 20), fc2=Linear(20, 5))
# Weights are tracked automatically through the module hierarchy
for name, param in model.parameters:
print(f"{name}: {param.shape}")
# fc1.weight: [20, 10]
# fc1.bias: [20]
# fc2.weight: [5, 20]
# fc2.bias: [5]
# Move all parameters to an accelerator (GPU)
model.to(Accelerator())In this example, the Module base class automatically tracks all parameters
through the module hierarchy, letting you iterate over them or inspect them. The
to() method moves the model to a different device in one call. Fields
annotated
PinnedDeviceTensor,
such as a quantized layer's scales, stay on the device that created them.
The KV cache manager
The max.pipelines.kv_cache package provides
cache management for transformer inference. The main component is
PagedKVCacheManager,
which handles memory allocation for key-value pairs across generation steps.
For most use cases, you don't interact with the cache manager directly. The
pipeline builds the cache parameters with your architecture config's
get_kv_params(), and at startup the model worker measures peak memory during
warmups and sizes the cache from what remains on the device.
How modules and pipelines work together
Understanding the relationship between modules and pipelines is key to working
with MAX. When you build a model for the pipeline system, define your
architecture as a module, using either the graph API
(max.nn.legacy.Module) or the
ModuleV3 API
(max.nn.Module).
MAX then compiles your module into an optimized executable for the target
device, and the pipeline orchestrates execution.
Here's the workflow:
-
Define your model: Create a module that defines your model architecture. The module's computation method defines what happens when processing inputs:
__call__()for the graph API, orforward()for the ModuleV3 API. -
MAX compiles your model: MAX compiles the module into an optimized executable for the target device. This compilation happens once, and you can reuse the result for many inference calls.
-
Pipelines orchestrate execution: The pipeline receives pre-tokenized context objects (which the tokenizer creates), manages the KV cache, calls your model's
PipelineModel.execute()method, samples output tokens, and returns results.
This separation lets you work at the right level of abstraction: use
max.nn.legacy or
max.nn to define model
architectures, let MAX handle compilation, and rely on pipelines for
production serving.
Request execution example
The script below builds a PipelineConfig, retrieves the pipeline with
PIPELINE_REGISTRY.retrieve(),
and runs one request. Pipeline registry covers what the
registry does.
import asyncio
from max.pipelines.modeling.types import RequestID, TextGenerationInputs, TextGenerationRequest
from max.pipelines import PIPELINE_REGISTRY, PipelineArgs, PipelineConfig
from max.pipelines.context import TextContext
# 1. Configure and retrieve the pipeline and tokenizer
config = PipelineConfig.from_args(
PipelineArgs.from_flat_kwargs(
model_path="meta-llama/Llama-3.1-8B-Instruct",
max_length=512,
)
)
tokenizer, pipeline = PIPELINE_REGISTRY.retrieve(config)
# 2. Get the KV cache manager from the pipeline
kv_cache_manager = pipeline.kv_manager
# 3. Create a request with the text prompt
request = TextGenerationRequest(
request_id=RequestID(),
model_name=config.model.model_path,
prompt="Explain how neural networks work",
)
# 4. Create a context object (tokenization happens here)
context = asyncio.run(tokenizer.new_context(request))
# 5. Allocate space in the KV cache for this request
kv_cache_manager.claim(context)
# 6. Run the generation loop
generated_text = ""
while True:
# Allocate KV cache for the next token
kv_cache_manager.alloc(context)
# Execute the pipeline with the current context
inputs = TextGenerationInputs[TextContext](
batches=[[context]],
)
output = pipeline.execute(inputs)
# Decode and accumulate generated tokens
for token in output[context.request_id].tokens:
generated_text += asyncio.run(
tokenizer.decode(token, skip_special_tokens=True)
)
# Check if generation is complete
if output[context.request_id].is_done:
break
print(generated_text)The walkthrough below follows the script in execution order:
- The registry maps the model path to the appropriate tokenizer and compiled pipeline.
- The cache manager tracks memory allocation for the request's key-value pairs across generation steps.
- The
new_context()method handles tokenization internally and creates a context object that tracks the request's state throughout generation. - The pipeline processes tokens, the model executes, and the sampler selects new tokens until completion.
Request state lives in the context object and the KV cache manager;
Stateless orchestration covers that split.
Pipeline components
Interfaces every component implements
The max.pipelines.modeling.types
package defines the interfaces every pipeline component implements. These
abstractions enable MAX to work uniformly across different model architectures
and tasks.
The key interfaces are:
-
Pipeline: Abstract base class for all pipelines. Definesexecute()andrelease()methods that all pipeline implementations must provide. -
PipelineInputs: Base class for inputs to a pipeline, such as text generation requests or embeddings requests. -
PipelineOutput: Protocol for pipeline outputs. Must implementis_doneto signal when generation is complete. -
PipelineTokenizer: Interface for tokenizers that convert between text and token IDs, and create context objects for requests. -
PipelineModel: Abstract base class for model implementations. Defines methods likeexecute()and input preparation methods that all architectures must implement.
These interfaces are task-agnostic. Specialized variants like
TextGenerationInputs
and
TextGenerationOutput
extend them for specific use cases.
Pipeline registry
The
PIPELINE_REGISTRY
is a singleton that tracks all available model architectures. When you run the
max serve command with a model, the registry:
- Looks up the model's architecture from its Hugging Face config.
- Validates that it supports the requested encoding and settings.
- Returns the appropriate tokenizer and pipeline for that architecture.
You can interact with the registry directly to retrieve a model's tokenizer and compiled pipeline:
from max.pipelines import PIPELINE_REGISTRY, PipelineArgs, PipelineConfig
# Create configuration for a model
config = PipelineConfig.from_args(
PipelineArgs.from_flat_kwargs(
model_path="meta-llama/Llama-3.1-8B-Instruct",
)
)
# Retrieve tokenizer and compiled pipeline
tokenizer, pipeline = PIPELINE_REGISTRY.retrieve(config)
# Or get a factory for deferred compilation
retrieved = PIPELINE_REGISTRY.retrieve_factory(config)
pipeline = retrieved.factory() # Compile when readyIn this example, retrieve() returns a ready-to-use pipeline, while
retrieve_factory() returns a RetrievedPipeline with the tokenizer, a
factory that performs compilation when invoked, and the memory plan the
pipeline was sized against. The factory pattern is useful when you need to
pass the pipeline across process boundaries, since it avoids serializing the
compiled model.
Supported architecture
A
SupportedArchitecture
configuration defines each model architecture. This bridges the gap between
Hugging Face model conventions and MAX's execution system.
When you point MAX at a Hugging Face model (like
meta-llama/Llama-3.1-8B-Instruct), MAX downloads and reads the model's
config.json file. Inside that file is an architectures field listing the
model class name (like "LlamaForCausalLM"). The registry uses this name to
look up the corresponding SupportedArchitecture, which tells MAX which model
(which subclass of PipelineModel) to use, what quantization formats it
supports, how to load weights, and which tokenizer to instantiate.
Here's how you define an architecture:
from max.graph.weights import WeightsFormat
from max.pipelines.context import TextContext
from max.pipelines.modeling.types import PipelineTask
from max.pipelines.lib import (
SupportedArchitecture,
TextTokenizer,
)
llama_arch = SupportedArchitecture(
# Must match the Hugging Face model class name
name="LlamaForCausalLM",
# The type of task this architecture supports
task=PipelineTask.TEXT_GENERATION,
# Example models that use this architecture
example_repo_ids=[
"meta-llama/Llama-3.1-8B-Instruct",
"meta-llama/Llama-3.2-3B-Instruct",
],
# Quantization support
default_encoding="q4_k",
supported_encodings={
"bfloat16",
"q4_k",
},
# Implementation classes
pipeline_model=Llama3Model,
tokenizer=TextTokenizer,
context_type=TextContext,
# Weight handling
default_weights_format=WeightsFormat.safetensors,
weight_adapters={
WeightsFormat.safetensors: convert_safetensor_state_dict,
WeightsFormat.gguf: convert_gguf_state_dict,
},
# Architecture-specific settings
config=Llama3Config,
multi_gpu_supported=True,
)The name field must match the architectures field in the model's Hugging
Face config.json. Common architecture names include LlamaForCausalLM,
DeepseekV3ForCausalLM, Qwen3VLMoeForConditionalGeneration, and more.
However, if you are using a custom architecture, you will need to use a custom
name that is specific to your architecture.
The task field determines which Pipeline subclass MAX uses to orchestrate
execution. Different types of models serve different purposes, and each task
type has its own execution strategy:
TEXT_GENERATION: Autoregressive text generation for chat and completion use cases. MAX usesTextGenerationPipelineto handle the prefill and decode loop, KV cache management, and token sampling.EMBEDDINGS_GENERATION: Vector embeddings from text input. MAX usesEmbeddingsPipelineto produce dense vector representations suitable for semantic search and retrieval.
For example, if you are building a model for text generation, you will set
task=PipelineTask.TEXT_GENERATION.
Set the default_encoding to the quantization encoding you want to use by
default, and supported_encodings to the set of encodings your architecture
supports. For example, if your model supports q4_k
quantization, set supported_encodings={"q4_k"}. For more
information on quantization, see the
quantization guide.
The pipeline_model field is the class that builds and executes the model.
The tokenizer field is the class that handles text encoding and decoding.
Finally, the weight_adapters field is a dictionary of functions that convert
weights from different formats. You can use the weight_adapters field to
convert weights from different formats to the default format.
Set these fields to match your architecture. Each reference architecture sets a working combination for its model family.
Pipeline model
The
PipelineModel
abstract class defines the interface for model implementations. Every model
architecture implements execute(), and either implements
prepare_initial_token_inputs() directly or delegates input preparation to a
batch processor (see
input preparation):
from max.pipelines.lib import PipelineModel, ModelInputs, ModelOutputs
class MyModel(PipelineModel):
# Link to your architecture config class for max_seq_len and KV params
model_config_cls = MyModelConfig
def execute(self, model_inputs: ModelInputs) -> ModelOutputs:
"""Run inference on the compiled model."""
...
def prepare_initial_token_inputs(
self, context_batch, kv_cache_inputs=None, return_n_logits=1
) -> ModelInputs:
"""Prepare inputs for the first forward pass (prefill)."""
...The execute() method runs your model's forward pass on the compiled
executable. prepare_initial_token_inputs() builds the tensors for each
scheduled batch (prefill and decode steps share this path in the serving
pipeline):
- Prefill: Process the entire prompt at once, building up the KV cache.
- Decode: Generate tokens one at a time, reusing the cached keys/values.
Delegate input preparation to a batch processor
Every token-generation model needs a batch processor: a
BatchProcessor
subclass that builds the tensors for each scheduled batch and parses the model's
outputs. Register it through the batching field of
SupportedArchitecture:
at registration, MAX sets the processor as the model's batch_processor_cls,
and the base PipelineModel.prepare_initial_token_inputs() delegates to it. If
neither the field nor the model class's own batch_processor_cls provides a
processor, preparing the first batch raises NotImplementedError. For ragged
batches, subclass
RaggedBatchProcessor,
which receives a
BatchProcessorRuntime
describing the active devices and KV cache.
Reuse an existing processor when your model's batching matches a family that MAX
already supports. One batch processor, Llama3BatchProcessor, backs Llama 3 and
a set of similar architectures, such as OLMo, Granite, Qwen 2, and Phi-3. Qwen
3.5 subclasses it to adjust specific steps; see the
llama3 architecture
for a reference implementation. Create a batch_processor.py with your own
BatchProcessor subclass only when your model needs batching logic that no
existing processor covers.
Link to the architecture config
The model_config_cls class variable links your pipeline model to the
architecture configuration class. This connection enables the base
PipelineModel to delegate key behaviors to the config.
The base class delegates the following through model_config_cls:
-
Maximum sequence length: The architecture config's
calculate_max_seq_len()classmethod is the architecture's sequence-length policy (whether the checkpoint bound is a hard cap or a default).PipelineConfig.from_args()runs it once at construction, soconfig.model.max_lengthcarries the resolved value; the pipeline model reads that value, or the memory plan's clamped one, as the sequence length it plans against. -
KV cache parameters: The base class builds the cache parameters with
model_config_cls.construct_kv_params(), and the pipeline model'sget_kv_params()delegates to it. Architectures with multiple or hybrid caches (for example, separate sliding-window and full-attention caches) return aMultiKVCacheParamstree instead, which the cache manager consumes through the sharedKVCacheParamInterface.
You can override calculate_max_seq_len() (on the architecture config) or
get_kv_params() if needed.
When to customize ModelInputs
The default ModelInputs handles token IDs, position IDs, attention masks, and
KV cache inputs, which covers standard decoder-only models. Subclass it when the
forward pass needs tensors beyond those:
- models that process both text and image inputs, needing image tensors, pixel values, or image embeddings alongside text tokens;
- models that pass extra tensors through the batch, such as mixture-of-experts routing tensors or retrieval-augmented document embeddings.
Pipeline execution
A pipeline orchestrates the KV cache manager, which tracks key-value pairs
across generation steps; the model, which runs the forward pass; and the
sampler, which selects the next token from the model's output logits.
execute() ties them together in a generation loop.
Stateless orchestration
A core design principle of the MAX pipeline system is that pipelines are stateless orchestrators. The pipeline itself does not own or maintain per-request state. Instead, it operates on the state passed to it through inputs:
-
Context objects track all request-specific information (tokens, sampling parameters, generation status). You pass these into
execute(), and the pipeline updates them but doesn't store them internally. -
KV cache manager owns the allocation and lifecycle of cached key-value pairs across all requests. The pipeline uses the cache manager but doesn't own it.
Configuration flow
Now let's see how configuration flows from user input to a running pipeline. When you run a model, configuration flows through several layers:
-
Start with user arguments: MAX collects CLI or API arguments into a
PipelineArgsobject that specifies the model path, quantization settings, and runtime parameters. -
Resolve at construction:
PipelineConfig.from_args()reads the model's Hugging Face config to look up the architecture, then resolves and validates the quantization encoding, weight paths, and devices, and applies any defaults the architecture declares. A model whose architecture isn't registered fails here, before any weights are downloaded. -
Size memory: The model worker measures peak memory during startup warmups and sizes the KV cache from what remains on the devices.
-
Instantiate pipeline: Finally, the registry constructs and returns the tokenizer and compiled pipeline ready for inference.
from max.pipelines import PipelineArgs, PipelineConfig
config = PipelineConfig.from_args(
PipelineArgs.from_flat_kwargs(
# Model specification (Hugging Face repo ID or local path)
model_path="meta-llama/Llama-3.1-8B-Instruct",
# Sequence limits
max_length=4096,
# Batching
max_batch_size=32,
)
)The PipelineConfig consolidates all settings and provides defaults based on
the model and hardware. See the
PipelineConfig reference
for all available options.
Shard the model across devices
A pipeline model runs on one device until its architecture builds a device mesh.
The reference architectures do that in _instantiate_module(): they build a
DeviceMesh from
the devices the pipeline assigned, then enter
default_device(mesh)
so every weight is created with a placement.
Set multi_gpu_supported=True in arch.py so the registry advertises the
architecture for a multi-device pipeline. The ops inside the model read those
placements and run the collectives they need. Tensor
parallelism covers how MAX splits a
served model across the GPUs you name.
Next steps
Now that you understand the pipeline architecture, continue learning:
- Modules: Learn how to build model graphs using
MAX's
Moduleclass. - Serve custom model architectures: Implement a complete custom architecture with the pipeline system.
- Bring your own fine-tuned model to MAX pipelines: Use your own fine-tuned weights with MAX pipelines.