For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

DecoderInputs

DecoderInputs

class max.pipelines.lib.role_contracts.DecoderInputs(*, tokens, vision=None, input_row_offsets, token_positions=None, host_input_row_offsets=None, return_n_logits, data_parallel_splits=None, signals=None, kv, batch_context_lengths=None, ep=None, lora=None, constants=None, state=None, scratch=None, stage_boundary=None)

source

Bases: Generic[TensorT, _SignalT]

The inputs of a DECODER module’s forward.

The decoder graph takes the fields in this order, which is the canonical order of the design’s role contract. A field that is None is not a graph input: the framework sets each optional field only when the deployment needs it.

Parameters:

• tokens (TensorT)
• vision (VisionInputs[TensorT] | None)
• input_row_offsets (TensorT)
• token_positions (TensorT | None)
• host_input_row_offsets (TensorT | None)
• return_n_logits (TensorT)
• data_parallel_splits (TensorT | None)
• signals (tuple[_SignalT, ...] | None)
• kv (Any)
• batch_context_lengths (tuple[TensorT, ...] | None)
• ep (Any)
• lora (tuple[TensorT, ...] | None)
• constants (Any)
• state (tuple[Any, ...] | None)
• scratch (tuple[Any, ...] | None)
• stage_boundary (Any)

batch_context_lengths

batch_context_lengths: tuple[TensorT, ...] | None = None

source

Each data-parallel replica’s page-aligned KV context length, a CPU int32 tensor of shape [1] per replica, or one per device for a decoder that sets device_context_length_dtype.

Set when the config sets takes_batch_context_lengths, as the MLA family does to size its prefill buffers without a device-to-host copy.

constants

constants: Any = None

source

The checkpoint-backed deployment inputs, held for the model lifetime.

data_parallel_splits

data_parallel_splits: TensorT | None = None

source

Each replica’s first sequence, a CPU int64 tensor of shape [data_parallel_degree + 1], under data parallelism.

Replica i serves sequences data_parallel_splits[i] up to data_parallel_splits[i + 1].

ep

ep: Any = None

source

The expert-parallel communication buffers, under expert parallelism, in the tree input_tree() declares.

host_input_row_offsets

host_input_row_offsets: TensorT | None = None

source

A CPU copy of input_row_offsets, under data parallelism.

It bounds each replica’s slice of the batch without a device-to-host copy inside the graph.

input_row_offsets

input_row_offsets: TensorT

source

Each sequence’s start in tokens, shape [batch_size + 1].

kv

kv: Any

source

The KV cache inputs, in the tree the config’s KV params declare.

lora

lora: tuple[TensorT, ...] | None = None

source

The LoRA routing and adapter inputs, when serving LoRA adapters.

The framework binds them into the decoder’s LoRA layers before it calls the decoder, so a decoder doesn’t read them.

return_n_logits

return_n_logits: TensorT

source

How many logits per sequence to return, a CPU tensor of shape [1].

scratch

scratch: tuple[Any, ...] | None = None

source

The workspace buffers the decoder declares with scratch_types, in its order (see decoder_scratch). The same buffers come every step, and nothing in them outlives one.

signals

signals: tuple[_SignalT, ...] | None = None

source

The allreduce signal buffers, one per device, of a graph-API decoder.

A multi-device graph-API decoder takes them, and so does one that sets always_signal_buffers. A ModuleV3 decoder never sees them: its compiled graph adds its own, where its input_layout names them. Runtime inputs hold driver buffers; graph-API inputs hold buffer values by default.

stage_boundary

stage_boundary: Any = None

source

What the previous pipeline stage’s decoder returned, in the tree it returned it, on a stage after the first (see boundary_types). A stage before the last returns such a tree in place of DecoderOutputs.

state

state: tuple[Any, ...] | None = None

source

The decoder’s per-request state, in the order its RequestStateStore declares: tensors, or buffers for state the graph updates in place. The architecture defines what each one holds.

token_positions

token_positions: TensorT | None = None

source

Each ragged token’s absolute sequence position, when declared.

tokens

tokens: TensorT

source

The batch’s ragged tokens, shape [total_seq_len].

vision

vision: VisionInputs[TensorT] | None = None

source

The batch’s image embeddings, when a VISION module is registered.