IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Structured output

MAX supports structured output using xgrammar for constrained decoding. Structured output, also called constrained decoding, lets you enforce specific output formats so a model returns structured, predictable responses.

When to use structured output​

If you want to structure a model's output when it responds to a user, then you should use a structured output response_format.

If you are connecting a model to tools, functions, data, or other systems, then you should use function calling instead of structured outputs.

Learn more about structured output in the LLM Inference Handbook.

How structured output works​

To use structured output, include the --enable-structured-output flag when serving your model with the max CLI.

max serve \
  --model "google/gemma-4-31B-it" \
  --enable-structured-output

You can tune structured output behavior with the following options:

  • --structured-output-backend: Selects the grammar backend for constrained decoding. xgrammar is the supported backend.
  • --structured-output-any-whitespace: Controls whether structured output grammars accept whitespace between JSON tokens. The default, false, constrains the model to compact JSON, which mitigates runaway output on some models. Pass true to allow whitespace-tolerant JSON.

Both the /chat/completions and /completions API endpoints are compatible with structured output.

You can define your structured output response format in two ways:

  1. JSON schema: Specify the schema directly in your request.
  2. Pydantic: Use Pydantic to define and validate your schema as a Python class.

We recommend testing your structured output responses thoroughly as they are sensitive to the way the model was trained.

JSON schema​

To specify structured output within your inference request, use the following format:

structured-image-description.py
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="google/gemma-4-31B-it",
    messages=[
        {
            "role": "system",
            "content": "You are an assistant that analyzes images and returns structured descriptions."
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Analyze this image and describe what you see."
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
                    }
                }
            ]
        }
    ],
    max_completion_tokens=300,
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "ImageAnalysis",
            "schema": {
                "type": "object",
                "properties": {
                    "description": {"type": "string"},
                    "subjects": {
                        "type": "array",
                        "items": {"type": "string"}
                    },
                    "colors": {
                        "type": "array",
                        "items": {"type": "string"}
                    },
                    "setting": {"type": "string"},
                    "mood": {"type": "string"}
                },
                "required": ["description", "subjects", "colors", "setting", "mood"],
                "additionalProperties": False
            }
        }
    }
)

print(response.choices[0].message.content)

Instead of a typical text response from the model, the response_format schema defined above results in a JSON-formatted structured output such as the following:

{
  "description": "A full-body shot of Peter Rabbit, the fictional character, standing on a dirt path. He is dressed in a blue jacket with brass buttons over a white shirt and a small yellow tie. He also wears brown pants and appears to be holding a small basket. The background consists of a rustic stone house with a thatched roof, a winding dirt road, green fields, and rolling hills under a bright sky. Wildflowers in shades of purple and white line the path in the foreground.",
  "subjects": [
    "rabbit",
    "house",
    "path",
    "fields",
    "hills",
    "flowers",
    "basket"
  ],
  "colors": [
    "blue",
    "brown",
    "green",
    "white",
    "yellow",
    "purple"
  ],
  "setting": "Rural countryside",
  "mood": "Whimsical, charming, idyllic"
}

Pydantic​

For production Python code, you can define your structured output using Pydantic. This gives you type-safe attribute access and automatic validation instead of manually parsing JSON strings.

Here's an example using a Pydantic BaseModel to analyze an image and return a validated response:

structured-image-analysis.py
from pydantic import BaseModel
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

class ImageAnalysis(BaseModel):
    description: str
    subjects: list[str]
    colors: list[str]
    setting: str
    mood: str

completion = client.chat.completions.parse(
    model="google/gemma-4-31B-it",
    messages=[
        {
            "role": "system",
            "content": "You are an assistant that analyzes images and returns structured descriptions."
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Analyze this image and describe what you see."
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
                    }
                }
            ]
        }
    ],
    max_completion_tokens=300,
    response_format=ImageAnalysis,
)

analysis = completion.choices[0].message.parsed
print(analysis)

Supported models​

All text generation models support structured output with MAX. As new models are added, they will also be compatible with structured output. This functionality is implemented at the pipeline level, ensuring consistency across different models.

However, structured output currently doesn't support CPU deployments, only GPU deployments.

Next steps​

Next, try processing local image files or deploying to the cloud.

Edit this page

Was this page helpful?

Models

Cookbook

Puzzles

Blog

Community