For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python class
CrossAttentionMTPProposer
CrossAttentionMTPProposer
class max.pipelines.speculative.cross_attention.CrossAttentionMTPProposer(draft, hidden_dim)
Bases: object
Drafts one token a step, attending into the target’s caches.
-
Parameters:
-
- draft (CrossAttentionDraft)
- hidden_dim (DimLike)
carry_dim_names
carry_dim_names = CarryDimNames(prefix='mtp_step', per_device=False)
decode_swaps
decode_swaps: tuple[DecodeKVSwap, ...] = (DecodeKVSwap.MAX_PROMPT_LENGTH_ONE,)
draft_cache
draft_cache = 'target'
passthrough_decode_swaps
passthrough_decode_swaps: tuple[str, ...] = ('full_attention',)
prefill()
prefill(batch, tokens, target_hidden)
Runs the first draft step over the target’s hidden states.
-
Parameters:
-
- batch (SequentialBatch)
- tokens (TensorValue)
- target_hidden (list[TensorValue])
-
Return type:
reuse
split_prefix
split_prefix = 'mtp'
step()
step(batch, draft_input, index)
Runs one later draft step on the previous step’s output.
-
Parameters:
-
- batch (SequentialBatch)
- draft_input (DraftStepInput)
- index (int)
-
Return type:
step_hidden_mode
step_hidden_mode = 'last_per_device'
supports_zero_draft_rows
supports_zero_draft_rows = True
takes_distributed_inputs
takes_distributed_inputs = True
uses_thinking_phase
uses_thinking_phase = True