For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python class
SpecWidthPolicy
SpecWidthPolicy
class max.pipelines.speculative.spec_width_policy.SpecWidthPolicy(num_speculative_tokens, by_batch_size=None, mixed=None, adaptive=None, adaptive_min_batch_size=1, skippable_draft=False, skips_draft=False)
Bases: object
How many drafts a speculative step produces and verifies.
Resolved once from the config, so every setting that bears on a step’s widths is combined in one place.
-
Parameters:
adaptive
The widths a decode step chooses among by measured tokens per second,
ascending. None keeps the width static.
adaptive_min_batch_size
adaptive_min_batch_size: int = 1
Smallest decode batch size that captures every adaptive width. Smaller batches capture only the widest.
by_batch_size
Dense batch_size -> width, index 0 unused. None verifies every
carried draft at every batch size.
captured_adaptive_widths()
captured_adaptive_widths(batch_size)
The adaptive widths graph capture records at a decode batch size.
Small batches are memory-bound, so a narrower width barely saves time there but would still cost a captured graph per width.
decode_widths()
decode_widths(max_batch_size)
The widths a pure decode step up to max_batch_size verifies.
draft_width()
draft_width(batch_size)
How many drafts a step produces for the next step to verify.
A step’s drafts are consumed by the next step, and this step’s batch
size is the best estimate available for the next step’s. Guessing
wrong costs only one step of speculation: drafts that were never
produced arrive as MAGIC_DRAFT_TOKEN_ID, which the acceptance test
rejects.
-
Parameters:
-
batch_size (int) – The per-replica decode batch size.
-
Returns:
-
The full draft depth, or the schedule’s width when
skips_draftso that a width of 0 skips the drafter. -
Return type:
from_config()
classmethod from_config(spec_config, num_speculative_tokens, max_batch_size, *, mixed_steps_verify, skippable_draft=False)
Resolves the policy for a pipeline.
-
Parameters:
-
- spec_config (SpeculativeConfig | None) – The pipeline’s speculative config, which carries the schedule or the adaptive widths, and the mixed-batch width.
- num_speculative_tokens (int) – The configured draft depth, which caps every width. For block drafters this is the checkpoint’s fixed block width. The schedule narrows only how much of that block the target verifies, not how much the draft produces.
- max_batch_size (int) – Largest decode batch size the schedule must cover.
- mixed_steps_verify (bool) – Whether a mixed prefill+decode step verifies drafts at all.
- skippable_draft (bool) – Whether the model’s graph takes its drafter’s
row count as an input, from
declares_skippable_draft().
-
Returns:
-
The resolved policy.
-
Return type:
log_summary()
log_summary()
Logs every setting that narrows a step below the full draft depth.
-
Return type:
-
None
max_mixed_width
property max_mixed_width: int
The most drafts a mixed step verifies per decode row.
mixed unset follows the schedule, which the draft depth caps.
mixed
The width a mixed prefill+decode step verifies: 0 when mixed steps
do not verify, None to follow the batch-size schedule.
num_speculative_tokens
num_speculative_tokens: int
The configured draft depth, which caps every width.
reachable_widths()
reachable_widths()
Every width any step could verify, sorted.
Includes the mixed-batch width, which is reachable at any batch size. The constrained-decoding bitmask allocates one buffer per width from this, and a width missing here fails the step with “no allocated buffer” rather than degrading.
scheduled_width()
scheduled_width(batch_size)
The schedule’s width at a decode batch size.
skippable_draft
skippable_draft: bool = False
Whether the graph takes its drafter’s row count as an input, which the
pipeline then binds on every step. See declares_skippable_draft().
skips_draft
skips_draft: bool = False
Whether a step whose drafts nothing would verify hands the drafter zero rows.
verifies_mixed_steps
property verifies_mixed_steps: bool
Whether a mixed prefill+decode step verifies any drafts.
verify_width()
verify_width(batch_size, *, mixed_step)
The width a step verifies, once it is known to verify at all.
widths_by_batch_size()
widths_by_batch_size(max_batch_size)
Dense batch_size -> widths a pure decode step may verify.
The table graph capture records: each batch size captures exactly its
own row, which holds every adaptive width at or above
adaptive_min_batch_size and only the widest below it. Index 0
is unused.