For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python class
CrossAttentionModuleV3MTPProposer
CrossAttentionModuleV3MTPProposer
class max.pipelines.speculative.cross_attention.CrossAttentionModuleV3MTPProposer(draft, hidden_dim)
Bases: TensorAdapter
Drafts one token a step on ModuleV3, attending into the target’s caches.
The ModuleV3 counterpart of CrossAttentionMTPProposer: the same
swaps, carries and outputs, but its draft takes Tensor values and its
modes as arguments rather than as attributes it holds.
-
Parameters:
-
- draft (CrossAttentionModuleV3Draft)
- hidden_dim (DimLike)
carry_dim_names
carry_dim_names = CarryDimNames(prefix='mtp_step', per_device=False)
decode_swaps
decode_swaps: tuple[DecodeKVSwap, ...] = (DecodeKVSwap.MAX_PROMPT_LENGTH_ONE,)
draft_cache
draft_cache = 'target'
passthrough_decode_swaps
passthrough_decode_swaps: tuple[str, ...] = ('full_attention',)
prefill()
prefill(batch, tokens, target_hidden)
Runs the first draft step over the target’s hidden states.
reuse
split_prefix
split_prefix = 'mtp'
step()
step(batch, draft_input, index)
Runs one later draft step on the previous step’s output.
-
Parameters:
-
- batch (SequentialBatch[Tensor, Tensor, Tensor])
- draft_input (DraftStepInput[Tensor])
- index (int)
-
Return type:
step_hidden_mode
step_hidden_mode = 'last_per_device'
supports_zero_draft_rows
supports_zero_draft_rows = True
takes_distributed_inputs
takes_distributed_inputs = True
uses_thinking_phase
uses_thinking_phase = True