For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

SpeculativeConfig

SpeculativeConfig

class max.pipelines.SpeculativeConfig(*, config_file=None, section_name=None, speculative_method=None, num_speculative_tokens=None, num_speculative_tokens_per_batch_size=None, num_speculative_tokens_mixed_batch=None, rejection_sampling_strategy=None, synthetic_acceptance_rate=None, use_relaxed_acceptance_for_thinking=False, relaxed_topk=10, relaxed_delta=0.6, use_greedy_acceptance=False, draft_proposal='argmax')

source

Bases: ConfigFileModel

Configures speculative decoding for a pipeline.

Speculative decoding accelerates token generation by having a small draft step propose several candidate tokens that the larger target verifies in one forward pass. This class selects the method (speculative_method), how many tokens to draft per step (num_speculative_tokens), and the knobs that decide how the target verifies them (synthetic_acceptance_rate, use_greedy_acceptance).

The CLI surfaces these fields as --speculative-method, --num-speculative-tokens, --num-speculative-tokens-per-batch-size, --num-speculative-tokens-mixed-batch, --rejection-sampling-strategy, and --synthetic-acceptance-rate. Construct the config directly when configuring a pipeline programmatically:

from max.pipelines.speculative import SpeculativeConfig

spec = SpeculativeConfig(
    speculative_method="eagle",
    num_speculative_tokens=3,
)

Instances are immutable. Assigning a field after construction raises.

Parameters:

• config_file (str | None)
• section_name (str | None)
• speculative_method (Literal['eagle', 'mtp', 'dflash', 'dflash2'] | None)
• num_speculative_tokens (int | None)
• num_speculative_tokens_per_batch_size (list[VerifyWidthRange] | None)
• num_speculative_tokens_mixed_batch (int | None)
• rejection_sampling_strategy (Literal['greedy', 'residual', 'typical-acceptance', 'logit-comparison'] | None)
• synthetic_acceptance_rate (float | None)
• use_relaxed_acceptance_for_thinking (bool)
• relaxed_topk (int)
• relaxed_delta (float)
• use_greedy_acceptance (bool)
• draft_proposal (Literal['argmax', 'sampled'])

draft_proposal

draft_proposal: Literal['argmax', 'sampled']

source

draft_width

property draft_width: int

source

The number of tokens drafted per step.

Set for every config the pipeline builds: the architecture supplies it for checkpoints that fix it, and the rest take the default.

is_dflash()

is_dflash()

source

Returns whether the configured method is a DFlash block draft.

True for both "dflash" and "dflash2": v2 keeps v1’s fused graph shape and block-drafting contract, so consumers that only need “the draft arrives a block at a time” – pipeline-class selection, KV cache sizing – want both. Architecture selection is the exception: each fused graph is built for one drafter, so the v1 and Eagle arms in lib.config exclude v2 and a v2 target gets its own arm. Use is_dflash2() where they differ.

Return type:

bool

is_dflash2()

is_dflash2()

source

Returns whether the configured method is DFlash2 specifically.

Return type:

bool

is_eagle()

is_eagle()

source

Returns whether the configured method is EAGLE.

EAGLE drafts share the target’s embedding and lm_head layers and read the target’s hidden states.

Return type:

bool

is_mtp()

is_mtp()

source

Returns whether the configured method is multi-token prediction (MTP).

Return type:

bool

model_config

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True, 'strict': False}

source

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init()

model_post_init(context, /)

source

This function is meant to behave like a BaseModel method to initialise private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

• self (BaseModel) – The BaseModel instance.
• context (Any) – The context.

Return type:

None

num_speculative_tokens

num_speculative_tokens: int | None

source

The number of tokens the draft proposes per verification pass.

None means unset: eagle and mtp resolve it to 2 at construction, while dflash-style block drafts leave it for the architecture to resolve from the draft checkpoint’s trained width. Larger values can raise the average draft acceptance length and peak speedup, but they may hurt acceptance rates at later positions and increase kernel latencies from the additional tokens.

num_speculative_tokens_mixed_batch

num_speculative_tokens_mixed_batch: int | None

source

How many of the drafted tokens the target verifies on a mixed batch.

Narrows for a different reason than num_speculative_tokens_per_batch_size: that schedule trades acceptance for decode throughput at large batch sizes, while this trades it for the latency of the prefill rows sharing the step, which sit on the critical path of a prompt’s time to first token.

Capped at num_speculative_tokens where it is read, not here: dflash leaves that width for the architecture to resolve from the draft checkpoint, so the ceiling is not yet known at config-validation time.

None leaves mixed batches on the batch-size schedule, which is the behavior when the field is unset. Reachable only alongside --enable-spec-decode-mixed-batches; without it a mixed batch verifies nothing at all.

num_speculative_tokens_per_batch_size

num_speculative_tokens_per_batch_size: list[VerifyWidthRange] | None

source

How many of the drafted tokens the target verifies, by decode batch size.

A step always drafts num_speculative_tokens proposals; this narrows how many of them the target checks.

Ranges are inclusive on both ends. The first must start at batch size 1 so every runtime batch size resolves to a count; gaps and the tail past the final range carry the previous count forward, and every count is capped at num_speculative_tokens since a step cannot verify more drafts than it carries. None verifies every drafted token, which is the behavior when the field is unset.

Applies to every speculative method. Block drafters (dflash) still draft their whole checkpoint-fixed block every step; only how much of that block the target verifies narrows.

rejection_sampling_strategy

rejection_sampling_strategy: RejectionSamplingStrategy | None

source

The requested rejection sampling strategy for verifying drafted tokens.

Inert: see RejectionSamplingStrategy. The acceptance rule in effect is AcceptanceSampler.acceptance_rule, and the startup config dump reports it alongside the fields that decide it.

relaxed_delta

relaxed_delta: float

source

relaxed_topk

relaxed_topk: int

source

speculative_method

speculative_method: SpeculativeMethod | None

source

The speculative decoding method to use.

One of "eagle", "mtp", "dflash", or "dflash2". When None, speculative decoding is disabled.

synthetic_acceptance_rate

synthetic_acceptance_rate: float | None

source

A benchmarking-only override that accepts drafts with a calibrated probability, ignoring real logits.

Must be between 0.0 and 1.0. When set, each draft position is accepted with a probability calibrated so that the mean joint acceptance across num_speculative_tokens positions matches this value. Use it to model hypothetical speedups without changing the draft model; leave unset for real serving.

use_greedy_acceptance

use_greedy_acceptance: bool

source

use_relaxed_acceptance_for_thinking

use_relaxed_acceptance_for_thinking: bool

source

uses_greedy_rejection()

uses_greedy_rejection()

source

Returns whether the "greedy" rejection sampling strategy is selected.

Return type:

bool

uses_logit_comparison()

uses_logit_comparison()

source

Returns whether the "logit-comparison" strategy is selected.

Return type:

bool

uses_typical_acceptance()

uses_typical_acceptance()

source

Returns whether the "typical-acceptance" strategy is selected.

Return type:

bool

verify_width_schedule

property verify_width_schedule: list[tuple[int, int, int]] | None

source

The schedule as validated, sorted (start, end, count) triples.

None when no schedule was configured.