For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

SpeculativeConfig

SpeculativeConfig

class max.pipelines.SpeculativeConfig(*, config_file=None, section_name=None, speculative_method=None, num_speculative_tokens=None, num_speculative_tokens_per_batch_size=None, adaptive_speculative_widths=None, adaptive_speculative_min_batch_size=1, num_speculative_tokens_mixed_batch=None, synthetic_acceptance_rate=None, use_relaxed_acceptance_for_thinking=False, relaxed_topk=10, relaxed_delta=0.6, use_greedy_acceptance=False, draft_proposal='argmax', draft_head_parallelism='device0', draft_probs_layout='device0', logit_penalties=False)

source

Bases: ConfigFileModel

Configures speculative decoding for a pipeline.

Speculative decoding accelerates token generation by having a small draft step propose several candidate tokens that the larger target verifies in one forward pass. This class selects the method (speculative_method), how many tokens to draft per step (num_speculative_tokens), and the knobs that decide how the target verifies them (synthetic_acceptance_rate, use_greedy_acceptance).

The CLI surfaces these fields as --speculative-method, --num-speculative-tokens, --num-speculative-tokens-per-batch-size, --adaptive-speculative-widths, --adaptive-speculative-min-batch-size, --num-speculative-tokens-mixed-batch, and --synthetic-acceptance-rate. Construct the config directly when configuring a pipeline programmatically:

from max.pipelines.speculative import SpeculativeConfig

spec = SpeculativeConfig(
    speculative_method="eagle",
    num_speculative_tokens=3,
)

Instances are immutable. Assigning a field after construction raises.

Parameters:

• config_file (str | None)
• section_name (str | None)
• speculative_method (Literal['eagle', 'mtp', 'dflash', 'dflash2', 'dspark'] | None)
• num_speculative_tokens (int | None)
• num_speculative_tokens_per_batch_size (list[VerifyWidthRange] | None)
• adaptive_speculative_widths (str | None)
• adaptive_speculative_min_batch_size (int)
• num_speculative_tokens_mixed_batch (int | None)
• synthetic_acceptance_rate (float | None)
• use_relaxed_acceptance_for_thinking (bool)
• relaxed_topk (int)
• relaxed_delta (float)
• use_greedy_acceptance (bool)
• draft_proposal (Literal['argmax', 'sampled'])
• draft_head_parallelism (Literal['device0', 'replica'])
• draft_probs_layout (Literal['device0', 'replica'])
• logit_penalties (bool)

adaptive_speculative_min_batch_size

adaptive_speculative_min_batch_size: int

source

Decode batch size below which only the widest adaptive width is captured.

Small batches are memory-bound, so narrowing barely changes their step cost while each extra width multiplies the captured graphs. Verifying more drafts than asked is always correct, since every step drafts them all.

adaptive_speculative_widths

adaptive_speculative_widths: str | None

source

The verify widths a throughput-driven controller chooses among.

None keeps the width static. Cannot be combined with num_speculative_tokens_per_batch_size. Startup cost scales with how many widths are named.

draft_head_parallelism

draft_head_parallelism: Literal['device0', 'replica']

source

draft_probs_layout

draft_probs_layout: Literal['device0', 'replica']

source

draft_proposal

draft_proposal: Literal['argmax', 'sampled']

source

draft_width

property draft_width: int

source

The number of tokens drafted per step.

Set for every config the pipeline builds: the architecture supplies it for checkpoints that fix it, and the rest take the default.

is_block_draft()

is_block_draft()

source

Returns whether the draft proposes a block of tokens per step.

True for every method outside the one-token-per-step methods ("eagle", "mtp"): "dflash", "dflash2" and "dspark". Pipeline-class selection and KV cache sizing branch on this; selecting a fused graph branches on the method itself.

Return type:

bool

is_dflash()

is_dflash()

source

Returns whether the configured method is a DFlash block draft.

True for both "dflash" and "dflash2": v2 keeps v1’s fused graph shape and block-drafting contract. Each fused graph is built for one drafter, though, so the v1 and Eagle arms in lib.config exclude v2 and a v2 target gets its own arm. Use is_dflash2() where they differ, and is_block_draft() where only “the draft arrives a block at a time” matters.

Return type:

bool

is_dflash2()

is_dflash2()

source

Returns whether the configured method is DFlash2 specifically.

Return type:

bool

is_dspark()

is_dspark()

source

Returns whether the configured method is a DSpark block draft.

DSpark drafts a block at a time like DFlash, but its drafts run a different step, so it is a method of its own rather than a DFlash variant.

Return type:

bool

is_eagle()

is_eagle()

source

Returns whether the configured method is EAGLE.

EAGLE drafts share the target’s embedding and lm_head layers and read the target’s hidden states.

Return type:

bool

is_mtp()

is_mtp()

source

Returns whether the configured method is multi-token prediction (MTP).

Return type:

bool

logit_penalties

logit_penalties: bool

source

model_config

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True, 'strict': False}

source

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init()

model_post_init(context, /)

source

This function is meant to behave like a BaseModel method to initialise private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

• self (BaseModel) – The BaseModel instance.
• context (Any) – The context.

Return type:

None

num_speculative_tokens

num_speculative_tokens: int | None

source

The number of tokens the draft proposes per verification pass.

None means unset: eagle and mtp resolve it to 2 at construction, while dflash-style block drafts leave it for the architecture to resolve from the draft checkpoint’s trained width. Larger values can raise the average draft acceptance length and peak speedup, but they may hurt acceptance rates at later positions and increase kernel latencies from the additional tokens.

num_speculative_tokens_mixed_batch

num_speculative_tokens_mixed_batch: int | None

source

How many of the drafted tokens the target verifies on a mixed batch.

Narrows for a different reason than num_speculative_tokens_per_batch_size: that schedule trades acceptance for decode throughput at large batch sizes, while this trades it for the latency of the prefill rows sharing the step, which sit on the critical path of a prompt’s time to first token.

Capped at num_speculative_tokens where it is read, not here: a block draft leaves that width for the architecture to resolve from the draft checkpoint, so the ceiling is not yet known at config-validation time.

None leaves mixed batches on the batch-size schedule, which is the behavior when the field is unset. Reachable only alongside --enable-spec-decode-mixed-batches; without it a mixed batch verifies nothing at all.

num_speculative_tokens_per_batch_size

num_speculative_tokens_per_batch_size: list[VerifyWidthRange] | None

source

How many of the drafted tokens the target verifies, by decode batch size.

A step always drafts num_speculative_tokens proposals; this narrows how many of them the target checks.

Ranges are inclusive on both ends. The first must start at batch size 1 so every runtime batch size resolves to a count; gaps and the tail past the final range carry the previous count forward, and every count is capped at num_speculative_tokens since a step cannot verify more drafts than it carries. None verifies every drafted token, which is the behavior when the field is unset.

Applies to every speculative method. A block drafter (dflash, dspark) still drafts its whole checkpoint-fixed block, and narrowing changes only how much of that block the target verifies. The exception is a count of 0, where a draft that declares supports_zero_draft_rows skips the draft forward too.

relaxed_delta

relaxed_delta: float

source

relaxed_topk

relaxed_topk: int

source

speculative_method

speculative_method: SpeculativeMethod | None

source

The speculative decoding method to use.

One of "eagle", "mtp", "dflash", "dflash2", or "dspark". When None, speculative decoding is disabled.

synthetic_acceptance_rate

synthetic_acceptance_rate: float | None

source

A benchmarking-only override that accepts drafts with a calibrated probability, ignoring real logits.

Must be between 0.0 and 1.0. When set, each draft position is accepted with a probability calibrated so that the mean joint acceptance across num_speculative_tokens positions matches this value. Use it to model hypothetical speedups without changing the draft model; leave unset for real serving.

use_greedy_acceptance

use_greedy_acceptance: bool

source

use_relaxed_acceptance_for_thinking

use_relaxed_acceptance_for_thinking: bool

source

verify_width_schedule

property verify_width_schedule: list[tuple[int, int, int]] | None

source

The schedule as validated, sorted (start, end, count) triples.

None when no schedule was configured.