IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

SpecWidthPolicy

SpecWidthPolicy​

class max.pipelines.speculative.spec_width_policy.SpecWidthPolicy(num_speculative_tokens, by_batch_size=None, mixed=None, adaptive=None, adaptive_min_batch_size=1, skippable_draft=False, skips_draft=False)

source

Bases: object

How many drafts a speculative step produces and verifies.

Resolved once from the config, so every setting that bears on a step’s widths is combined in one place.

Parameters:

adaptive​

adaptive: Sequence[int] | None = None

source

The widths a decode step chooses among by measured tokens per second, ascending. None keeps the width static.

adaptive_min_batch_size​

adaptive_min_batch_size: int = 1

source

Smallest decode batch size that captures every adaptive width. Smaller batches capture only the widest.

by_batch_size​

by_batch_size: Sequence[int] | None = None

source

Dense batch_size -> width, index 0 unused. None verifies every carried draft at every batch size.

captured_adaptive_widths()​

captured_adaptive_widths(batch_size)

source

The adaptive widths graph capture records at a decode batch size.

Small batches are memory-bound, so a narrower width barely saves time there but would still cost a captured graph per width.

Parameters:

batch_size (int) – The per-replica decode batch size, where 0 takes the widths of 1.

Returns:

The widths, ascending.

Return type:

list[int]

decode_widths()​

decode_widths(max_batch_size)

source

The widths a pure decode step up to max_batch_size verifies.

Parameters:

max_batch_size (int) – The largest decode batch size to cover.

Returns:

The distinct widths, sorted.

Return type:

list[int]

draft_width()​

draft_width(batch_size)

source

How many drafts a step produces for the next step to verify.

A step’s drafts are consumed by the next step, and this step’s batch size is the best estimate available for the next step’s. Guessing wrong costs only one step of speculation: drafts that were never produced arrive as MAGIC_DRAFT_TOKEN_ID, which the acceptance test rejects.

Parameters:

batch_size (int) – The per-replica decode batch size.

Returns:

The full draft depth, or the schedule’s width when skips_draft so that a width of 0 skips the drafter.

Return type:

int

from_config()​

classmethod from_config(spec_config, num_speculative_tokens, max_batch_size, *, mixed_steps_verify, skippable_draft=False)

source

Resolves the policy for a pipeline.

Parameters:

  • spec_config (SpeculativeConfig | None) – The pipeline’s speculative config, which carries the schedule or the adaptive widths, and the mixed-batch width.
  • num_speculative_tokens (int) – The configured draft depth, which caps every width. For block drafters this is the checkpoint’s fixed block width. The schedule narrows only how much of that block the target verifies, not how much the draft produces.
  • max_batch_size (int) – Largest decode batch size the schedule must cover.
  • mixed_steps_verify (bool) – Whether a mixed prefill+decode step verifies drafts at all.
  • skippable_draft (bool) – Whether the model’s graph takes its drafter’s row count as an input, from declares_skippable_draft().

Returns:

The resolved policy.

Return type:

SpecWidthPolicy

log_summary()​

log_summary()

source

Logs every setting that narrows a step below the full draft depth.

Return type:

None

max_mixed_width​

property max_mixed_width: int

source

The most drafts a mixed step verifies per decode row.

mixed unset follows the schedule, which the draft depth caps.

mixed​

mixed: int | None = None

source

The width a mixed prefill+decode step verifies: 0 when mixed steps do not verify, None to follow the batch-size schedule.

num_speculative_tokens​

num_speculative_tokens: int

source

The configured draft depth, which caps every width.

reachable_widths()​

reachable_widths()

source

Every width any step could verify, sorted.

Includes the mixed-batch width, which is reachable at any batch size. The constrained-decoding bitmask allocates one buffer per width from this, and a width missing here fails the step with “no allocated buffer” rather than degrading.

Return type:

list[int]

scheduled_width()​

scheduled_width(batch_size)

source

The schedule’s width at a decode batch size.

Parameters:

batch_size (int) – The per-replica decode batch size. Sizes past the end of the schedule take its last width, and 0 takes the width of 1.

Returns:

The width.

Return type:

int

skippable_draft​

skippable_draft: bool = False

source

Whether the graph takes its drafter’s row count as an input, which the pipeline then binds on every step. See declares_skippable_draft().

skips_draft​

skips_draft: bool = False

source

Whether a step whose drafts nothing would verify hands the drafter zero rows.

verifies_mixed_steps​

property verifies_mixed_steps: bool

source

Whether a mixed prefill+decode step verifies any drafts.

verify_width()​

verify_width(batch_size, *, mixed_step)

source

The width a step verifies, once it is known to verify at all.

Parameters:

  • batch_size (int) – The per-replica decode batch size.
  • mixed_step (bool) – Whether the step is a mixed prefill+decode step, which takes mixed over the schedule when it is set.

Returns:

The width.

Return type:

int

widths_by_batch_size()​

widths_by_batch_size(max_batch_size)

source

Dense batch_size -> widths a pure decode step may verify.

The table graph capture records: each batch size captures exactly its own row, which holds every adaptive width at or above adaptive_min_batch_size and only the widest below it. Index 0 is unused.

Parameters:

max_batch_size (int) – The largest decode batch size to cover.

Returns:

A list of length max_batch_size + 1.

Return type:

list[list[int]]