IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python module

max.pipelines.speculative.driver

Model-agnostic driver for sequential (eagle / MTP) speculative decoding.

Every sequential unified spec-decode module in the tree runs the same seven phases – merge, verify, mask, accept, shift, propose, pack – and differs only in how it calls its target, how it calls its draft, and what per-step cache bookkeeping the draft needs. SequentialDriver owns the phases; a model contributes a verify callable and a SequentialProposer.

The two adapters are deliberately not Module subclasses. The driver registers the target and draft modules under the same attribute names the hand-written modules used, so weight loading, state_dict keys and the weights registry are unchanged.

The driver runs on graph values, but nothing outside it has to. A caller may pass experimental Tensor inputs, one per device, and gets Tensor outputs back; a proposer that subclasses TensorAdapter, and a verify wrapped in tensor_compat_verify(), read and return Tensor values too. Both convert at the boundary, so a ModuleV3 model never reaches for the graph API.

Driver​

SequentialDriverDrives the sequential speculative-decoding loop.
SequentialProposerA draft that emits one token per step, K steps deep.

Graph inputs​

DistributedInputsThe CPU-side inputs only a distributed=True signature declares.
SequentialBatchOne spec-decode iteration's graph inputs, merged and broadcast.

Draft state​

CarryDimNamesHow the driver names the carried hidden state's batch dim per step.
DraftStepInputLoop-carried state between draft steps.
ProposedOne draft invocation's contribution.
ReuseSpecA per-step result the draft carries beside its hidden state.

Draft cache​

DecodeKVSwapA single per-step edit to the draft's PagedCacheValues.
DraftCacheWhich cache the draft writes, and so what the driver advances.

Experimental tensor adapters​

TensorAdapterMarks a proposer that runs on experimental Tensor values.

Functions​

gather_variable_rowsGathers each request's last return_n_logits rows of h.
tensor_compat_verifyAdapts a verify that runs on experimental Tensor values.