For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Speculative decoding

Speculative decoding accelerates LLM token generation without changing the model's outputs. A smaller, faster draft step proposes several candidate tokens, and the larger target model verifies them in one forward pass. MAX accepts the prefix the target agrees with and resamples at the first disagreement, so quality matches running the target alone.

The speedup comes from batching verification across K candidate positions. When the target accepts all K drafts, you get K+1 tokens per step instead of one, converting a memory-bandwidth-bound workload into one that better uses available compute.

Caution

Speculative decoding is in preview. APIs and behavior may change as the implementation matures.

We recommend running speculative decoding on an accelerator. For more information, see System requirements.

Supported methods

MAX supports common speculative-decoding methods like EAGLE, EAGLE3, MTP, and DFlash.

Method	Draft source	Supported targets and hardware
eagle	A trained EAGLE or EAGLE3 draft that shares the target's embedding and lm_head	Llama 3 (1 GPU), Kimi K2.5 and K2.6 (8× B200).
mtp	A native multi-token prediction head inside the target checkpoint	DeepSeek V3 and derivatives (8× B200), GLM-5.2 (8× B200), Gemma 4 (1 GPU), Inkling (8× B200).
dflash	A trained DFlash draft (a multi-layer block transformer) paired with the target	Llama 3 (1 GPU), Kimi K2.5 (8× B200).

Serve with speculative decoding

Pick a tab for the method you want to run. Each example starts max serve with the right target and draft, then sends a chat-completion request to the local endpoint from the OpenAI Python client.

To call the endpoint, install the OpenAI Python client:

pixi

pixi add openai

uv

uv add openai

pip

pip install openai

conda

conda install openai

EAGLE

Serve Llama 3.1 8B Instruct with a pretrained EAGLE checkpoint as the draft:

max serve \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --speculative-method eagle \
  --draft-model-path atomicapple0/EAGLE-LLaMA3.1-Instruct-8B \
  --num-speculative-tokens 2 \
  --devices gpu

We use atomicapple0/EAGLE-LLaMA3.1-Instruct-8B because it ships safetensors weights. For the full list of weight formats MAX supports, see WeightsFormat.

Once the server prints Server ready on http://0.0.0.0:8000, send a chat-completion request:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What are the benefits of speculative decoding?"},
    ],
    max_completion_tokens=500,
)

print(response.choices[0].message.content)

MTP

Serve DeepSeek V3 with its built-in MTP head. You don't pass a separate draft model because the MTP head already lives in the target checkpoint:

max serve \
  --model deepseek-ai/DeepSeek-V3 \
  --speculative-method mtp \
  --num-speculative-tokens 2 \
  --devices gpu:0,1,2,3,4,5,6,7

Once the server prints Server ready on http://0.0.0.0:8000, send a chat-completion request:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What are the benefits of speculative decoding?"},
    ],
    max_completion_tokens=500,
)

print(response.choices[0].message.content)

Interpret the startup architecture name

When you serve a model with speculative decoding, MAX shows a combined architecture name in the startup banner. It differs from the target architecture name because MAX resolves the target and its speculative mechanism into a dedicated speculative decoding architecture. For example, serving deepseek-ai/DeepSeek-V3 with MTP appears as UnifiedMTPDeepseekV3ForCausalLM.

Monitor acceptance rates

Once you send traffic, the scheduler logs per-batch acceptance stats in this format:

Draft Tokens: 145/160 (90.62%) accepted, Acceptance Len: 1.45 / 2 toks, Per-Pos: [95%, 86%] |

Each field means:

• Accepted / generated: tokens the target confirmed, over tokens the draft proposed.
• Acceptance length: average number of drafted tokens accepted per verification pass. A value of 1.45 / 2 means on average 1.45 of the 2 drafted tokens survive verification.
• Per-position: acceptance rate at each draft position, conditional on all earlier positions accepting. Later positions are always rarer.

Low acceptance rates (below roughly 50%) usually mean the draft doesn't match the target well. Try a smaller --num-speculative-tokens or a better-matched draft checkpoint.

Tune speculative decoding

The following flags control how MAX drafts tokens and how verification decides to accept them. The Config column shows the config class each flag maps to when you configure a pipeline programmatically.

Flag	Description	Config
--num-speculative-tokens	Number of tokens the draft proposes per step. Defaults to 2 for EAGLE and MTP. For DFlash-style block drafters, MAX reads the value from the draft checkpoint's trained width. If the checkpoint doesn't declare a width, you must set this flag explicitly. Larger values raise peak speedup but hurt acceptance at later positions.	SpeculativeConfig
--num-speculative-tokens-per-batch-size	How many of the drafted tokens the target verifies, set by decode batch-size range. Takes a JSON list of inclusive ranges, such as [{"batch_start": 1, "batch_end": 16, "num_tokens": 3}, {"batch_start": 17, "batch_end": 64, "num_tokens": 1}].	SpeculativeConfig
--rejection-sampling-strategy	How the target verifies drafted tokens. Accepts greedy, residual, typical-acceptance, or logit-comparison. When unset, EAGLE and MTP default to typical-acceptance.	SpeculativeConfig
--synthetic-acceptance-rate	Benchmarking-only knob that accepts each drafted token with a calibrated probability, ignoring real logits. Use this to model hypothetical speedups without changing the draft.	SpeculativeConfig
--draft-proposal	How the draft model proposes tokens. argmax (default) selects tokens deterministically. sampled draws from the draft model's distribution and preserves that distribution for verification. sampled requires a GPU, a static vocabulary size, and a serving architecture that supports it.	SpeculativeConfig
--use-relaxed-acceptance-for-thinking	Whether to accept a draft token inside a <think>...</think> block when it matches any of the target's top candidates within a probability threshold, instead of requiring an exact match. Outside the thinking span, the strict acceptance rule still applies. Requires --draft-proposal argmax and a serving architecture that supports relaxed acceptance. Defaults to false.	SpeculativeConfig
--relaxed-topk	The number of top candidates from the target distribution to consider when relaxed acceptance is active. Ignored when relaxed acceptance is off. Defaults to 10.	SpeculativeConfig
--relaxed-delta	The probability gap below the top candidate within which candidates remain eligible for relaxed acceptance. Ignored when relaxed acceptance is off. Defaults to 0.6.	SpeculativeConfig
--use-greedy-acceptance	Whether to use greedy (argmax) draft acceptance instead of the stochastic sampler, which lets MAX capture the fused speculative graph with CUDA graphs. Valid only for greedy serving (temperature 0, top-k 1), and incompatible with relaxed and synthetic acceptance. Requires a serving architecture that supports it. Defaults to false.	SpeculativeConfig
--draft-devices	Device list for the draft model. Useful when you want the draft and target on different GPUs.	Draft model config
--device-memory-utilization	Fraction of device memory MAX may use. Speculative decoding allocates KV cache for both the target and the draft, so leave more headroom than you would for single-model serving.	KVCacheConfig

For EAGLE and MTP, MAX auto-enables the overlap scheduler and device graph capture. Both reduce per-step latency and need no additional flags.

Compatibility and limits

When using speculative decoding:

• The --enable-echo flag isn't supported.
• The --max-length and --draft-max-length are both capped at the draft model's maximum supported length. MAX clamps both to that limit if either exceeds it.
• Structured output is supported. MAX applies the constraint bitmask to each speculated position during verification. This feature requires a GPU deployment.
• Repetition, frequency, and presence penalties aren't supported when using a separate draft model (--draft-model-path), including EAGLE. Repetition, frequency, and presence penalties are supported when using MTP.

Next steps

You can combine speculative decoding with prefix caching and with disaggregated inference. The following topics go deeper on performance and deployment:

• Prefix caching: Use prefix caching when serving a model with the MAX CLI.
• Benchmark MAX on NVIDIA or AMD GPUs: Learn how to use our benchmarking script to measure the performance of MAX.
• Deploy MAX on GPU with self-hosted endpoints: Learn how to deploy MAX pipelines to cloud.
• Speculative decoding: Learn more about it in the LLM Inference Handbook.