IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Speculative decoding

Speculative decoding accelerates LLM token generation without changing the model's outputs. A smaller, faster draft step proposes several candidate tokens, and the larger target model verifies them in one forward pass. MAX accepts the prefix the target agrees with and resamples at the first disagreement, so quality matches running the target alone.

The speedup comes from batching verification across K candidate positions. When the target accepts all K drafts, you get K+1 tokens per step instead of one, converting a memory-bandwidth-bound workload into one that better uses available compute.

Supported methods​

MAX supports common speculative-decoding methods like DFlash, DFlash2, DSpark, EAGLE, EAGLE3, and MTP.

MethodDraft sourceSupported targets and hardware
dflashA trained DFlash block draft paired with the targetLlama 3 (1 GPU), Gemma 4 (1 GPU), Kimi K2.5 (8× B200).
dflash2A trained DFlash2 block draft paired with the targetQwen3.8-27B (1× B200), Beam-1.0 (4× B200).
dsparkA trained DSpark block draft paired with the targetGemma 4 (1 GPU), Nemotron 3.5 Lightning.
eagleA trained EAGLE or EAGLE3 draft that shares the target's embedding and lm_headLlama 3 (1 GPU), Kimi K2.5 and K2.6 (8× B200).
mtpA multi-token prediction head, either inside the target checkpoint or as an assistant draftDeepSeek V3 and derivatives (8× B200), GLM-5.2 (8× B200), Gemma 4 (1 GPU), Inkling (8× B200), Qwen3.8-27B (1× B200).

Serve with speculative decoding​

Pick a tab for the method you want to run. Each example starts max serve with the right target and draft, then sends a chat-completion request to the local endpoint from the OpenAI Python client.

To call the endpoint, install the OpenAI Python client:

pixi add openai

Serve Llama 3.1 8B Instruct with a pretrained DFlash checkpoint as the draft:

max serve \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --speculative-method dflash \
  --draft-model-path z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat \
  --devices gpu

DFlash drafts are trained at a fixed block size, and MAX sets --num-speculative-tokens to the trained width, one less than the block size. If you pass a different value, MAX overrides it and logs a warning such as DFlash draft was trained at block_size=10, so num_speculative_tokens is being overridden from 2 to 9.

DSpark drafts are block drafts too. Serve them with --speculative-method dspark, and MAX sets the width from the draft checkpoint the same way.

The endpoint is ready when you see this message printed in your terminal:

Server ready on http://0.0.0.0:8000 (Press CTRL+C to quit)

Send a chat-completion request:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What are the benefits of speculative decoding?"},
    ],
    max_completion_tokens=500,
)

print(response.choices[0].message.content)

Supported model architectures​

When you serve a model with a draft, MAX loads a dedicated speculative decoding architecture for the pair and prints its name in the startup banner. For example, serving deepseek-ai/DeepSeek-V3 with --speculative-method mtp resolves to DeepseekV3ForCausalLM+mtp. The name comes from the target's architecture and the draft checkpoint's declared architecture:

TargetDraftResolved architecture
Llama 3EAGLE draftLlamaForCausalLM+eagle
Llama 3DFlash draftLlamaForCausalLM+dflash
Gemma 4Assistant draft (Gemma4AssistantForCausalLM)Gemma4ForConditionalGeneration+mtp
Gemma 4 31BDFlash draftGemma4ForConditionalGeneration+dflash
Gemma 4 31BDSpark draftGemma4ForConditionalGeneration+dspark
Gemma 4 12BDSpark draftGemma4UnifiedForConditionalGeneration+dspark
DeepSeek V3None (built-in MTP head)DeepseekV3ForCausalLM+mtp
GLM-5.2None (built-in MTP head)UnifiedMTPGlmMoeDsaForCausalLM
Kimi K2.5DFlash draftKimiK25ForConditionalGeneration+dflash
Qwen3.8-27BDFlash2 draftQwen3_5ForConditionalGeneration+dflash2
Beam-1.0DFlash2 draftBeamForCausalLM_ModuleV3+dflash2

Monitor acceptance rates​

Once you send traffic, the scheduler logs per-batch acceptance stats in this format:

Draft Tokens: 145/160 (90.62%) accepted, Acceptance Len: 1.45 / 2 toks (accepted drafts/step; 2.45 toks/step incl bonus), Per-Pos: [p0=95%, p1=86%] |

Each field means:

  • Accepted / generated: tokens the target confirmed, over tokens the draft proposed.
  • Acceptance length: average number of drafted tokens accepted per verification pass. A value of 1.45 / 2 means on average 1.45 of the 2 drafted tokens survive verification. The parenthetical counts the bonus token the target produces after them: 2.45 tokens per step including it.
  • Per-position: acceptance rate at each draft position, conditional on all earlier positions accepting. Later positions are always rarer.

Low acceptance rates (below roughly 50%) usually mean the draft doesn't match the target well. Try a smaller --num-speculative-tokens or a better-matched draft checkpoint.

Tune speculative decoding​

The following flags control how MAX drafts tokens and how verification decides to accept them. The Config column shows the config class each flag maps to when you configure a pipeline programmatically.

FlagDescriptionConfig
--num-speculative-tokensNumber of tokens the draft proposes per step. Defaults to 2 for EAGLE and MTP. For DFlash and DSpark drafts, MAX reads the value from the draft checkpoint's trained width, and you must set this flag when the checkpoint doesn't declare one. For dflash2, the checkpoint sets the width, and MAX stops at startup when this flag disagrees with it. Larger values raise peak speedup but hurt acceptance at later positions.SpeculativeConfig
--num-speculative-tokens-per-batch-sizeHow many of the drafted tokens the target verifies, set by decode batch-size range. Takes a JSON list of inclusive ranges, such as [{"batch_start": 1, "batch_end": 16, "num_tokens": 3}, {"batch_start": 17, "batch_end": 64, "num_tokens": 1}].SpeculativeConfig
--adaptive-speculative-widthsVerify widths to choose among at runtime by measured decode tokens per second. Takes a comma-separated list such as 1,3,5, or all for every width up to --num-speculative-tokens. Can't be combined with --num-speculative-tokens-per-batch-size. Graph capture probes every width at every batch size, so more widths mean longer startup. Unset by default.SpeculativeConfig
--adaptive-speculative-min-batch-sizeSmallest decode batch size that captures every adaptive verify width. Smaller batches capture only the widest, and MAX rounds a narrower width up to it. Raising this cuts startup time and graph memory. Requires --adaptive-speculative-widths. Defaults to 1.SpeculativeConfig
--enable-spec-decode-mixed-batchesKeeps draft verification active on a batch that mixes prefill and decode requests. Enabling this flag also enables in-flight batching. The serving architecture must support per-row mixed-batch commits. On any other architecture, MAX logs a warning and falls back to in-flight batching, where mixed batches advance one token at a time. Defaults to false.PipelineRuntimeConfig
--num-speculative-tokens-mixed-batchSets how many of the drafted tokens the target verifies on a batch that mixes prefill and decode requests. MAX caps the value at --num-speculative-tokens. A narrower width trades acceptance for the time to first token of the prefill requests sharing the step. When unset, mixed batches follow the --num-speculative-tokens-per-batch-size schedule. Requires --enable-spec-decode-mixed-batches.SpeculativeConfig
--synthetic-acceptance-rateBenchmarking-only knob that accepts each drafted token with a calibrated probability, ignoring real logits. Use this to model hypothetical speedups without changing the draft.SpeculativeConfig
--draft-proposalHow the draft model proposes tokens. argmax (default) selects tokens deterministically. sampled draws from the draft model's distribution and preserves that distribution for verification. sampled requires a GPU, a static vocabulary size, and a serving architecture that supports it.SpeculativeConfig
--draft-head-parallelismWhere a data-parallel block draft runs its sequential head. device0 (default) runs it on device 0 over the whole batch. replica runs it on each replica's own device over that replica's rows, trading device memory for a shorter draft step. No effect without --data-parallel-degree or on a draft whose head isn't split.SpeculativeConfig
--use-relaxed-acceptance-for-thinkingWhether to accept a draft token inside a <think>...</think> block when it matches any of the target's top candidates within a probability threshold, instead of requiring an exact match. Outside the thinking span, the strict acceptance rule still applies. Requires --draft-proposal argmax and a serving architecture that supports relaxed acceptance. Defaults to false.SpeculativeConfig
--relaxed-topkThe number of top candidates from the target distribution to consider when relaxed acceptance is active. Ignored when relaxed acceptance is off. Defaults to 10.SpeculativeConfig
--relaxed-deltaThe probability gap below the top candidate within which candidates remain eligible for relaxed acceptance. Ignored when relaxed acceptance is off. Defaults to 0.6.SpeculativeConfig
--use-greedy-acceptanceWhether to use greedy (argmax) draft acceptance instead of the stochastic sampler, which lets MAX capture the fused speculative graph with CUDA graphs. Valid only for greedy serving (temperature 0, top-k 1), and incompatible with relaxed and synthetic acceptance. Requires a serving architecture that supports it. Defaults to false.SpeculativeConfig
--draft-devicesDevice list for the draft model. Useful when you want the draft and target on different GPUs.Draft model config
--device-memory-utilizationFraction of device memory MAX may use. Speculative decoding allocates KV cache for both the target and the draft, so leave more headroom than you would for single-model serving.KVCacheConfig

MAX auto-enables the overlap scheduler for every speculative architecture. It also auto-enables device graph capture on the architectures that support it. Both reduce per-step latency and need no additional flags.

Compatibility and limits​

When using speculative decoding:

  • The --enable-echo flag isn't supported.
  • The --max-length and --draft-max-length are both capped at the draft model's maximum supported length. MAX clamps both to that limit if either exceeds it.
  • Structured output is supported. MAX applies the constraint bitmask to each speculated position during verification. This feature requires a GPU deployment.
  • Repetition, frequency, and presence penalties aren't supported when using a separate draft model (--draft-model-path), including EAGLE. Repetition, frequency, and presence penalties are supported when using MTP.
  • Prefix caching is off when the target checkpoint keeps recurrent state, which includes Qwen3.8-27B. MAX overrides the setting and logs a warning.

Next steps​

You can combine speculative decoding with disaggregated inference, and with prefix caching on every target whose checkpoint keeps no recurrent state. The following topics go deeper on performance and deployment:

Was this page helpful?