IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Quickstart

MAX provides a high-performance engine for serving built-in and custom AI models. This quickstart shows you how to start a local endpoint and send it a text generation request using the OpenAI SDK.

Get started​

Let's set up your system to run MAX. Check that you meet the system requirements. Then use pixi or uv to create a project.

  1. If needed, install pixi:

    curl -fsSL https://pixi.sh/install.sh | sh
  2. Create a project:

    pixi init max-quickstart \
      -c https://conda.modular.com/max-nightly/ -c conda-forge \
      && cd max-quickstart
  3. Install max-all:

    pixi add max-all

Start the server​

Start a local server with max serve:

  1. In your max-quickstart directory, start the virtual environment:

    pixi shell
  2. Start the server:

    max serve --model Qwen/Qwen3.8-27B
  3. Once the server starts, you'll see:

    πŸš€ Server ready on http://0.0.0.0:8000 (Press CTRL+C to quit)

Keep this terminal open while you send requests to the server.

Send a request​

Send a text generation request to the server.

  1. Open a new terminal, go to your max-quickstart directory, and start the virtual environment:

    pixi shell
  2. Save the following script as generate-text.py:

    generate-text.py
    from openai import OpenAI
    
    client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
    prompt = "In one sentence, what is the KV cache in LLM inference?"
    
    completion = client.chat.completions.create(
        model="Qwen/Qwen3.8-27B",
        messages=[{"role": "user", "content": prompt}],
    )
    
    print(completion.choices[0].message.content.strip())
  3. Run the script:

    python generate-text.py

You'll see a response similar to the following:

The KV cache stores the key and value vectors for previously processed tokens
so that autoregressive decoding can reuse them in attention without recomputing
them each step.

Benchmark the model​

To measure the performance of your served model, use max benchmark. The following command runs a benchmark against the chat completions endpoint using the Sonnet dataset:

max benchmark \
  --model Qwen/Qwen3.8-27B \
  --backend modular \
  --endpoint /v1/chat/completions \
  --dataset-name sonnet \
  --num-prompts 500 \
  --sonnet-input-len 550 \
  --output-lengths 256 \
  --sonnet-prefix-len 200 \
  --max-concurrency 32

When the benchmark finishes, MAX prints the results in your terminal.

To save the results as a JSON file, pass a path to --result-filename:

max benchmark \
  ... \
  --result-filename "results/quickstart-benchmark.json"

For a complete list of datasets and configuration options, see the max benchmark reference.

Next steps​

MAX offers so much more than serving text generation models:

Was this page helpful?