IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Model pipeline

Implementing a model architecture in MAX involves more than defining the layers and the forward pass. To serve your model, you must also integrate it into a pipeline that manages the entire inference lifecycle.

A pipeline connects a model implementation to MAX's serving system. It's responsible for tasks such as preparing inputs, executing the model, managing inference state such as the KV cache, performing post-processing, and returning results.

A pipeline is built from components, each behind an interface. Most architectures reuse the standard components; this page shows how they work together and when to replace one.

The pipeline system​

The pipeline system uses a registry pattern where model architectures register their capabilities, and the infrastructure handles the execution details. When you point MAX at a model, the registry looks up the architecture, validates compatibility, downloads weights, compiles the model, and returns a ready-to-use pipeline.

This separates concerns cleanly:

  • Modules define model architectures and hold weights.
  • Pipelines orchestrate the inference loop and manage state.
  • Registry maps model identifiers to implementations.
  • Compilation transforms your model into optimized executables for the target device.

Pipelines let you focus on model architecture while MAX handles the production infrastructure, including batching, caching, compilation, and serving.

The packages pipelines build on​

Pipelines build on max.nn.legacy, max.nn, and max.pipelines.kv_cache.

Neural network modules​

MAX architectures build their layers with max.nn, the module API for ModuleV3 architectures. It includes common components like:

The max.nn.legacy package builds MAX graphs directly and backs architectures written against it. To port one to ModuleV3, see Migrate an existing architecture to ModuleV3.

The Module base class in max.nn standardizes how modules manage weights and devices. Here's an example of building a simple multi-layer perceptron with the ModuleV3 API:

from max.driver import Accelerator
from max.nn import Linear, Module, module_dataclass

@module_dataclass
class MLP(Module):
    fc1: Linear
    fc2: Linear

    def forward(self, x):
        return self.fc2(self.fc1(x))

# Create a model with two linear layers
model = MLP(fc1=Linear(10, 20), fc2=Linear(20, 5))

# Weights are tracked automatically through the module hierarchy
for name, param in model.parameters:
    print(f"{name}: {param.shape}")
# fc1.weight: [20, 10]
# fc1.bias: [20]
# fc2.weight: [5, 20]
# fc2.bias: [5]

# Move all parameters to an accelerator (GPU)
model.to(Accelerator())

In this example, the Module base class automatically tracks all parameters through the module hierarchy, letting you iterate over them or inspect them. The to() method moves the model to a different device in one call. Fields annotated PinnedDeviceTensor, such as a quantized layer's scales, stay on the device that created them.

The KV cache manager​

The max.pipelines.kv_cache package provides cache management for transformer inference. The main component is PagedKVCacheManager, which handles memory allocation for key-value pairs across generation steps.

For most use cases, you don't interact with the cache manager directly. The pipeline builds the cache parameters with your architecture config's get_kv_params(), and at startup the model worker measures peak memory during warmups and sizes the cache from what remains on the device.

How modules and pipelines work together​

Understanding the relationship between modules and pipelines is key to working with MAX. When you build a model for the pipeline system, define your architecture as a module, using either the graph API (max.nn.legacy.Module) or the ModuleV3 API (max.nn.Module). MAX then compiles your module into an optimized executable for the target device, and the pipeline orchestrates execution.

Here's the workflow:

  1. Define your model: Create a module that defines your model architecture. The module's computation method defines what happens when processing inputs: __call__() for the graph API, or forward() for the ModuleV3 API.

  2. MAX compiles your model: MAX compiles the module into an optimized executable for the target device. This compilation happens once, and you can reuse the result for many inference calls.

  3. Pipelines orchestrate execution: The pipeline receives pre-tokenized context objects (which the tokenizer creates), manages the KV cache, calls your model's PipelineModel.execute() method, samples output tokens, and returns results.

This separation lets you work at the right level of abstraction: use max.nn.legacy or max.nn to define model architectures, let MAX handle compilation, and rely on pipelines for production serving.

Request execution example​

The script below builds a PipelineConfig, retrieves the pipeline with PIPELINE_REGISTRY.retrieve(), and runs one request. Pipeline registry covers what the registry does.

import asyncio
from max.pipelines.modeling.types import RequestID, TextGenerationInputs, TextGenerationRequest
from max.pipelines import PIPELINE_REGISTRY, PipelineArgs, PipelineConfig
from max.pipelines.context import TextContext

# 1. Configure and retrieve the pipeline and tokenizer
config = PipelineConfig.from_args(
    PipelineArgs.from_flat_kwargs(
        model_path="meta-llama/Llama-3.1-8B-Instruct",
        max_length=512,
    )
)
tokenizer, pipeline = PIPELINE_REGISTRY.retrieve(config)

# 2. Get the KV cache manager from the pipeline
kv_cache_manager = pipeline.kv_manager

# 3. Create a request with the text prompt
request = TextGenerationRequest(
    request_id=RequestID(),
    model_name=config.model.model_path,
    prompt="Explain how neural networks work",
)

# 4. Create a context object (tokenization happens here)
context = asyncio.run(tokenizer.new_context(request))

# 5. Allocate space in the KV cache for this request
kv_cache_manager.claim(context)

# 6. Run the generation loop
generated_text = ""
while True:
    # Allocate KV cache for the next token
    kv_cache_manager.alloc(context)

    # Execute the pipeline with the current context
    inputs = TextGenerationInputs[TextContext](
        batches=[[context]],
    )
    output = pipeline.execute(inputs)

    # Decode and accumulate generated tokens
    for token in output[context.request_id].tokens:
        generated_text += asyncio.run(
            tokenizer.decode(token, skip_special_tokens=True)
        )

    # Check if generation is complete
    if output[context.request_id].is_done:
        break

print(generated_text)

The walkthrough below follows the script in execution order:

  1. The registry maps the model path to the appropriate tokenizer and compiled pipeline.
  2. The cache manager tracks memory allocation for the request's key-value pairs across generation steps.
  3. The new_context() method handles tokenization internally and creates a context object that tracks the request's state throughout generation.
  4. The pipeline processes tokens, the model executes, and the sampler selects new tokens until completion.

Request state lives in the context object and the KV cache manager; Stateless orchestration covers that split.

Pipeline components​

Interfaces every component implements​

The max.pipelines.modeling.types package defines the interfaces every pipeline component implements. These abstractions enable MAX to work uniformly across different model architectures and tasks.

The key interfaces are:

  • Pipeline: Abstract base class for all pipelines. Defines execute() and release() methods that all pipeline implementations must provide.

  • PipelineInputs: Base class for inputs to a pipeline, such as text generation requests or embeddings requests.

  • PipelineOutput: Protocol for pipeline outputs. Must implement is_done to signal when generation is complete.

  • PipelineTokenizer: Interface for tokenizers that convert between text and token IDs, and create context objects for requests.

  • PipelineModel: Abstract base class for model implementations. Defines methods like execute() and input preparation methods that all architectures must implement.

These interfaces are task-agnostic. Specialized variants like TextGenerationInputs and TextGenerationOutput extend them for specific use cases.

Pipeline registry​

The PIPELINE_REGISTRY is a singleton that tracks all available model architectures. When you run the max serve command with a model, the registry:

  1. Looks up the model's architecture from its Hugging Face config.
  2. Validates that it supports the requested encoding and settings.
  3. Returns the appropriate tokenizer and pipeline for that architecture.

You can interact with the registry directly to retrieve a model's tokenizer and compiled pipeline:

from max.pipelines import PIPELINE_REGISTRY, PipelineArgs, PipelineConfig

# Create configuration for a model
config = PipelineConfig.from_args(
    PipelineArgs.from_flat_kwargs(
        model_path="meta-llama/Llama-3.1-8B-Instruct",
    )
)

# Retrieve tokenizer and compiled pipeline
tokenizer, pipeline = PIPELINE_REGISTRY.retrieve(config)

# Or get a factory for deferred compilation
retrieved = PIPELINE_REGISTRY.retrieve_factory(config)
pipeline = retrieved.factory()  # Compile when ready

In this example, retrieve() returns a ready-to-use pipeline, while retrieve_factory() returns a RetrievedPipeline with the tokenizer, a factory that performs compilation when invoked, and the memory plan the pipeline was sized against. The factory pattern is useful when you need to pass the pipeline across process boundaries, since it avoids serializing the compiled model.

Supported architecture​

A SupportedArchitecture configuration defines each model architecture. This bridges the gap between Hugging Face model conventions and MAX's execution system.

When you point MAX at a Hugging Face model (like meta-llama/Llama-3.1-8B-Instruct), MAX downloads and reads the model's config.json file. Inside that file is an architectures field listing the model class name (like "LlamaForCausalLM"). The registry uses this name to look up the corresponding SupportedArchitecture, which tells MAX which model (which subclass of PipelineModel) to use, what quantization formats it supports, how to load weights, and which tokenizer to instantiate.

Here's how you define an architecture:

from max.graph.weights import WeightsFormat
from max.pipelines.context import TextContext
from max.pipelines.modeling.types import PipelineTask
from max.pipelines.lib import (
    SupportedArchitecture,
    TextTokenizer,
)

llama_arch = SupportedArchitecture(
    # Must match the Hugging Face model class name
    name="LlamaForCausalLM",

    # The type of task this architecture supports
    task=PipelineTask.TEXT_GENERATION,

    # Example models that use this architecture
    example_repo_ids=[
        "meta-llama/Llama-3.1-8B-Instruct",
        "meta-llama/Llama-3.2-3B-Instruct",
    ],

    # Quantization support
    default_encoding="q4_k",
    supported_encodings={
        "bfloat16",
        "q4_k",
    },

    # Implementation classes
    pipeline_model=Llama3Model,
    tokenizer=TextTokenizer,
    context_type=TextContext,

    # Weight handling
    default_weights_format=WeightsFormat.safetensors,
    weight_adapters={
        WeightsFormat.safetensors: convert_safetensor_state_dict,
        WeightsFormat.gguf: convert_gguf_state_dict,
    },

    # Architecture-specific settings
    config=Llama3Config,
    multi_gpu_supported=True,
)

The name field must match the architectures field in the model's Hugging Face config.json. Common architecture names include LlamaForCausalLM, DeepseekV3ForCausalLM, Qwen3VLMoeForConditionalGeneration, and more. However, if you are using a custom architecture, you will need to use a custom name that is specific to your architecture.

The task field determines which Pipeline subclass MAX uses to orchestrate execution. Different types of models serve different purposes, and each task type has its own execution strategy:

  • TEXT_GENERATION: Autoregressive text generation for chat and completion use cases. MAX uses TextGenerationPipeline to handle the prefill and decode loop, KV cache management, and token sampling.
  • EMBEDDINGS_GENERATION: Vector embeddings from text input. MAX uses EmbeddingsPipeline to produce dense vector representations suitable for semantic search and retrieval.

For example, if you are building a model for text generation, you will set task=PipelineTask.TEXT_GENERATION.

Set the default_encoding to the quantization encoding you want to use by default, and supported_encodings to the set of encodings your architecture supports. For example, if your model supports q4_k quantization, set supported_encodings={"q4_k"}. For more information on quantization, see the quantization guide.

The pipeline_model field is the class that builds and executes the model. The tokenizer field is the class that handles text encoding and decoding.

Finally, the weight_adapters field is a dictionary of functions that convert weights from different formats. You can use the weight_adapters field to convert weights from different formats to the default format.

Set these fields to match your architecture. Each reference architecture sets a working combination for its model family.

Pipeline model​

The PipelineModel abstract class defines the interface for model implementations. Every model architecture implements execute(), and either implements prepare_initial_token_inputs() directly or delegates input preparation to a batch processor (see input preparation):

from max.pipelines.lib import PipelineModel, ModelInputs, ModelOutputs

class MyModel(PipelineModel):
    # Link to your architecture config class for max_seq_len and KV params
    model_config_cls = MyModelConfig

    def execute(self, model_inputs: ModelInputs) -> ModelOutputs:
        """Run inference on the compiled model."""
        ...

    def prepare_initial_token_inputs(
        self, context_batch, kv_cache_inputs=None, return_n_logits=1
    ) -> ModelInputs:
        """Prepare inputs for the first forward pass (prefill)."""
        ...

The execute() method runs your model's forward pass on the compiled executable. prepare_initial_token_inputs() builds the tensors for each scheduled batch (prefill and decode steps share this path in the serving pipeline):

  1. Prefill: Process the entire prompt at once, building up the KV cache.
  2. Decode: Generate tokens one at a time, reusing the cached keys/values.

Delegate input preparation to a batch processor​

Every token-generation model needs a batch processor: a BatchProcessor subclass that builds the tensors for each scheduled batch and parses the model's outputs. Register it through the batching field of SupportedArchitecture: at registration, MAX sets the processor as the model's batch_processor_cls, and the base PipelineModel.prepare_initial_token_inputs() delegates to it. If neither the field nor the model class's own batch_processor_cls provides a processor, preparing the first batch raises NotImplementedError. For ragged batches, subclass RaggedBatchProcessor, which receives a BatchProcessorRuntime describing the active devices and KV cache.

Reuse an existing processor when your model's batching matches a family that MAX already supports. One batch processor, Llama3BatchProcessor, backs Llama 3 and a set of similar architectures, such as OLMo, Granite, Qwen 2, and Phi-3. Qwen 3.5 subclasses it to adjust specific steps; see the llama3 architecture for a reference implementation. Create a batch_processor.py with your own BatchProcessor subclass only when your model needs batching logic that no existing processor covers.

The model_config_cls class variable links your pipeline model to the architecture configuration class. This connection enables the base PipelineModel to delegate key behaviors to the config.

The base class delegates the following through model_config_cls:

  • Maximum sequence length: The architecture config's calculate_max_seq_len() classmethod is the architecture's sequence-length policy (whether the checkpoint bound is a hard cap or a default). PipelineConfig.from_args() runs it once at construction, so config.model.max_length carries the resolved value; the pipeline model reads that value, or the memory plan's clamped one, as the sequence length it plans against.

  • KV cache parameters: The base class builds the cache parameters with model_config_cls.construct_kv_params(), and the pipeline model's get_kv_params() delegates to it. Architectures with multiple or hybrid caches (for example, separate sliding-window and full-attention caches) return a MultiKVCacheParams tree instead, which the cache manager consumes through the shared KVCacheParamInterface.

You can override calculate_max_seq_len() (on the architecture config) or get_kv_params() if needed.

When to customize ModelInputs​

The default ModelInputs handles token IDs, position IDs, attention masks, and KV cache inputs, which covers standard decoder-only models. Subclass it when the forward pass needs tensors beyond those:

  • models that process both text and image inputs, needing image tensors, pixel values, or image embeddings alongside text tokens;
  • models that pass extra tensors through the batch, such as mixture-of-experts routing tensors or retrieval-augmented document embeddings.

Pipeline execution​

A pipeline orchestrates the KV cache manager, which tracks key-value pairs across generation steps; the model, which runs the forward pass; and the sampler, which selects the next token from the model's output logits. execute() ties them together in a generation loop.

Stateless orchestration​

A core design principle of the MAX pipeline system is that pipelines are stateless orchestrators. The pipeline itself does not own or maintain per-request state. Instead, it operates on the state passed to it through inputs:

  • Context objects track all request-specific information (tokens, sampling parameters, generation status). You pass these into execute(), and the pipeline updates them but doesn't store them internally.

  • KV cache manager owns the allocation and lifecycle of cached key-value pairs across all requests. The pipeline uses the cache manager but doesn't own it.

Configuration flow​

Now let's see how configuration flows from user input to a running pipeline. When you run a model, configuration flows through several layers:

  1. Start with user arguments: MAX collects CLI or API arguments into a PipelineArgs object that specifies the model path, quantization settings, and runtime parameters.

  2. Resolve at construction: PipelineConfig.from_args() reads the model's Hugging Face config to look up the architecture, then resolves and validates the quantization encoding, weight paths, and devices, and applies any defaults the architecture declares. A model whose architecture isn't registered fails here, before any weights are downloaded.

  3. Size memory: The model worker measures peak memory during startup warmups and sizes the KV cache from what remains on the devices.

  4. Instantiate pipeline: Finally, the registry constructs and returns the tokenizer and compiled pipeline ready for inference.

from max.pipelines import PipelineArgs, PipelineConfig

config = PipelineConfig.from_args(
    PipelineArgs.from_flat_kwargs(
        # Model specification (Hugging Face repo ID or local path)
        model_path="meta-llama/Llama-3.1-8B-Instruct",

        # Sequence limits
        max_length=4096,

        # Batching
        max_batch_size=32,
    )
)

The PipelineConfig consolidates all settings and provides defaults based on the model and hardware. See the PipelineConfig reference for all available options.

Shard the model across devices​

A pipeline model runs on one device until its architecture builds a device mesh. The reference architectures do that in _instantiate_module(): they build a DeviceMesh from the devices the pipeline assigned, then enter default_device(mesh) so every weight is created with a placement.

Set multi_gpu_supported=True in arch.py so the registry advertises the architecture for a multi-device pipeline. The ops inside the model read those placements and run the collectives they need. Tensor parallelism covers how MAX splits a served model across the GPUs you name.

Next steps​

Now that you understand the pipeline architecture, continue learning:

Was this page helpful?