IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python module

max.pipelines.speculative.spec_adapters

How spec decoding calls each target architecture’s module.

The modules MAX architectures build share no calling convention: each takes its inputs in an order of its own and returns its outputs by position. A driver can’t call a target without knowing which architecture built it, so each target’s package holds a target adapter, its spec_adapters module, that holds its call. _TARGET_ADAPTERS names the module of each target architecture’s TargetAdapter, which takes what it leaves out from Llama 3’s and holds a target’s differences under one speculative method in by_method. A target whose ModuleV3 variant builds a module of another kind names that variant’s adapter under the variant’s own name.

A module is imported only when one of its targets runs, so spec decoding imports no architecture it doesn’t run, and a build that doesn’t ship a private target’s package never imports it.

HISPARSE_INPUTS​

max.pipelines.speculative.spec_adapters.HISPARSE_INPUTS = 'hisparse_inputs'

source

The key of a driver batch’s extra that holds the target’s HiSparse graph inputs: the host and hot tables per device, then the host mirror, in the order the signature declares them.

Only a target whose graph declares them reads the key; the draft never does, because its latent and indexer stay fully resident.

VerifyFn​

max.pipelines.speculative.spec_adapters.VerifyFn

source

Verifies a driver batch on the target.

alias of Callable[[SequentialBatch | BlockBatch], Verified[Any, TensorValue]]

batch_distributed_buffers()​

max.pipelines.speculative.spec_adapters.batch_distributed_buffers(target_arch, target_inputs)

source

Returns the per-device context lengths and the prepared buffers.

The buffers are the ones in ep_input_types() that the target’s batch processor prepared.

None context lengths when it prepares none.

Parameters:

Return type:

tuple[Sequence[Buffer] | None, Sequence[Buffer]]

batch_replay_buffers()​

max.pipelines.speculative.spec_adapters.batch_replay_buffers(target_arch, method, target_inputs)

source

Returns the replay inputs that the target prepared.

They are the batch’s values of replay_input_types(), and empty for the full graph.

Parameters:

Return type:

list[Buffer]

batch_trailing_buffers()​

max.pipelines.speculative.spec_adapters.batch_trailing_buffers(target_arch, method, target_inputs)

source

Returns the trailing inputs that the target prepared.

They are the batch’s values of trailing_input_types(), for a target whose batch processor prepares the pairing’s batches.

Parameters:

Return type:

list[Buffer]

block_proposer()​

max.pipelines.speculative.spec_adapters.block_proposer(target_arch, verify, draft, draft_config, speculative)

source

Returns the proposer of a draft that runs on its target’s own state.

None for every other draft, which its method’s proposer runs.

Parameters:

Return type:

BlockProposer[Any, Any, TensorValue, BufferValue, Value[Any]] | None

check()​

max.pipelines.speculative.spec_adapters.check(target_arch, speculative, structured_output)

source

Raises if the target can’t verify under the speculative config.

Parameters:

Return type:

None

ep_input_tree()​

max.pipelines.speculative.spec_adapters.ep_input_tree(target_arch, target)

source

Returns the group that names a target’s expert-parallel inputs.

It is None when no group names them.

Parameters:

Return type:

object | None

ep_input_types()​

max.pipelines.speculative.spec_adapters.ep_input_types(target_arch, target)

source

Returns the expert-parallel inputs of a built target module.

They are the leaves of ep_input_tree() when a group names them.

Parameters:

Return type:

Sequence[TensorType | BufferType]

ep_tail()​

max.pipelines.speculative.spec_adapters.ep_tail(target_arch, target)

source

Returns whether the target declares them on an undistributed signature.

Parameters:

Return type:

bool

export_contract()​

max.pipelines.speculative.spec_adapters.export_contract(target_arch, speculative, draft_config, target_config, kv_params, input_types, structured_output)

source

Returns the record a fused-graph engine checks its inputs against.

Only a target whose engine keeps such a record has one.

An exporter writes it to the graph’s sidecar.

Parameters:

Return type:

Mapping[str, object] | None

extra()​

max.pipelines.speculative.spec_adapters.extra(target_arch, method, target_config, trailing)

source

Returns what the target’s call reads of the trailing graph inputs.

The driver hands it to the verify and the proposer as the batch’s extra.

Parameters:

Return type:

dict[str, Any]

fold_index_glue()​

max.pipelines.speculative.spec_adapters.fold_index_glue(target_arch)

source

Returns whether the target’s accept path folds its index glue.

A target opts into the folded single-launch ops; the default keeps the whole-tensor op chains.

Parameters:

target_arch (str)

Return type:

bool

has_adapter()​

max.pipelines.speculative.spec_adapters.has_adapter(target_arch)

source

Returns whether spec decoding holds a call under exactly this name.

A ModuleV3 variant counts only when it names an adapter of its own.

Parameters:

target_arch (str)

Return type:

bool

hisparse_input_types()​

max.pipelines.speculative.spec_adapters.hisparse_input_types(target_arch, method, target_config)

source

Returns the target’s HiSparse inputs, which follow the EP inputs.

Empty when the target keeps every latent page resident.

Parameters:

  • target_arch (str)
  • method (str)
  • target_config (Any)

Return type:

Sequence[TensorType | BufferType]

prepares_every_batch()​

max.pipelines.speculative.spec_adapters.prepares_every_batch(target_arch)

source

Returns whether the target prepares every batch of a pairing.

The inputs past the canonical tail count.

Parameters:

target_arch (str)

Return type:

bool

primary_kv()​

max.pipelines.speculative.spec_adapters.primary_kv(target_arch, method)

source

Returns the name of the KV leaf a driver dispatches the target over.

The target’s other leaves ride in the driver batch’s passthrough_kv, under their names in the KV tree. A target with one leaf has the name "".

Parameters:

  • target_arch (str)
  • method (str)

Return type:

str

relaxed_thinking_acceptance()​

max.pipelines.speculative.spec_adapters.relaxed_thinking_acceptance(target_arch)

source

Returns whether a reasoning model’s drafts may be accepted.

The relaxed in-thinking rule decides it.

Parameters:

target_arch (str)

Return type:

bool

replay_enabled()​

max.pipelines.speculative.spec_adapters.replay_enabled(target_arch, method, pipeline_config, target_config)

source

Returns whether the pairing compiles a replay graph beside the full one.

Parameters:

  • target_arch (str)
  • method (str)
  • pipeline_config (Any)
  • target_config (Any)

Return type:

bool

replay_extra()​

max.pipelines.speculative.spec_adapters.replay_extra(target_arch, method, target_config, replay)

source

Returns what the target’s call reads of the replay graph’s inputs.

Parameters:

Return type:

dict[str, Any]

replay_input_types()​

max.pipelines.speculative.spec_adapters.replay_input_types(target_arch, method, target_config)

source

Returns the inputs only the replay graph takes, last of all.

Parameters:

  • target_arch (str)
  • method (str)
  • target_config (Any)

Return type:

Sequence[TensorType | BufferType]

sequential_proposer()​

max.pipelines.speculative.spec_adapters.sequential_proposer(target_arch, verify, draft, draft_config, speculative)

source

Returns the proposer of a one-token-per-step draft.

The draft runs on its target’s own module.

None for every other draft, which its method’s proposer runs.

Parameters:

Return type:

SequentialProposer[Any, TensorValue, BufferValue, Value[Any]] | None

signal_buffer_index()​

max.pipelines.speculative.spec_adapters.signal_buffer_index(target_arch)

source

Returns where a fused graph takes its signal buffers.

The result is fixed across device counts, or none when the graph does not fix it.

Parameters:

target_arch (str)

Return type:

int | None

speculates_on_last_stage()​

max.pipelines.speculative.spec_adapters.speculates_on_last_stage(target_arch, method)

source

Returns whether method builds as the last pipeline stage.

The answer covers the target’s fused graph.

Parameters:

  • target_arch (str)
  • method (str)

Return type:

bool

successor_replay()​

max.pipelines.speculative.spec_adapters.successor_replay(target_arch, method)

source

Returns whether the draft head reads each position’s successor.

An engine then replays a cached prompt row’s boundary token, as Mach’s artifact boot decides from the export’s record.

Parameters:

  • target_arch (str)
  • method (str)

Return type:

bool

trailing_buffers()​

max.pipelines.speculative.spec_adapters.trailing_buffers(target_arch, method, target_config, contexts, devices)

source

Returns the trailing inputs that no KV child holds.

They are a batch’s values of trailing_input_types().

Parameters:

  • target_arch (str) – The target checkpoint’s architecture name.
  • method (str) – The speculative method.
  • target_config (Any) – The config the target was built from.
  • contexts (Sequence[TextContext]) – The batch, in the order its tokens are merged in.
  • devices (Sequence[Device]) – The devices the target runs on.

Return type:

list[Buffer]

trailing_input_tree()​

max.pipelines.speculative.spec_adapters.trailing_input_tree(target_arch, method, target_config)

source

Returns the groups that name trailing_input_types().

None for a target whose trailing inputs an exported graph leaves unnamed.

Parameters:

  • target_arch (str)
  • method (str)
  • target_config (Any)

Return type:

Mapping[str, object] | None

trailing_input_types()​

max.pipelines.speculative.spec_adapters.trailing_input_types(target_arch, method, target_config)

source

Returns the inputs the target’s call takes past the canonical tail.

Parameters:

  • target_arch (str)
  • method (str)
  • target_config (Any)

Return type:

Sequence[TensorType | BufferType]

verify_commits_long_rows()​

max.pipelines.speculative.spec_adapters.verify_commits_long_rows(target_arch, method)

source

Returns whether the target’s verify commits a long row.

A long row is longer than one token and its drafts, and the commit must happen in every recurrent leaf under method. Mach’s gen_mef writes the answer to the export’s sidecar.

Parameters:

  • target_arch (str)
  • method (str)

Return type:

bool

verify_for()​

max.pipelines.speculative.spec_adapters.verify_for(target_arch, target, target_config, hidden_states, draft_config)

source

Returns how a built target module verifies under a spec-decode driver.

Parameters:

  • target_arch (str) – The target checkpoint’s architecture name.
  • target (Module) – The module the target’s pipeline model built.
  • target_config (Any) – The config it was built from.
  • hidden_states (ReturnHiddenStates) – The hidden states it was built to return.
  • draft_config (Any) – The config the draft was built from.

Return type:

Callable[[SequentialBatch | BlockBatch], Verified[Any, TensorValue]]

verify_takes_prefill_rows()​

max.pipelines.speculative.spec_adapters.verify_takes_prefill_rows(target_arch, method, target_config)

source

Returns whether the verify commits a prefill row in a verify step.

An engine may then mix a prompt row into the step, as Mach’s artifact boot decides from the export’s record.

Parameters:

  • target_arch (str)
  • method (str)
  • target_config (Any)

Return type:

bool