For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

DecoderModelInputs

DecoderModelInputs

class max.pipelines.lib.module_registration.DecoderModelInputs(*, kv_cache_inputs=None, lora_buffers=(), vision_embeddings=<factory>, vision_scatter_indices=<factory>, hidden_states=None, tokens, input_row_offsets, return_n_logits, token_positions=None, host_input_row_offsets=None, data_parallel_splits=None, batch_context_lengths=None, ep_inputs=(), signal_buffers=(), constants=None, state=(), scratch=(), stage_boundary=None, request_ids=(), input_tree_def=None)

source

Bases: ModelInputs

The prepared inputs of a registered DECODER.

Parameters:

• kv_cache_inputs (KVCacheInputs[Buffer, Buffer] | None)
• lora_buffers (tuple[Buffer, ...])
• vision_embeddings (list[Buffer])
• vision_scatter_indices (list[Buffer])
• hidden_states (Buffer | list[Buffer] | None)
• tokens (Buffer)
• input_row_offsets (Buffer)
• return_n_logits (Buffer)
• token_positions (Buffer | None)
• host_input_row_offsets (Buffer | None)
• data_parallel_splits (Buffer | None)
• batch_context_lengths (list[Buffer] | None)
• ep_inputs (tuple[Buffer, ...])
• signal_buffers (tuple[Buffer, ...])
• constants (tree.Tree[Buffer] | None)
• state (tuple[Buffer, ...])
• scratch (tuple[Buffer, ...])
• stage_boundary (tree.Tree[Buffer] | None)
• request_ids (tuple[RequestID, ...])
• input_tree_def (tree.TreeDef | None)

batch_context_lengths

batch_context_lengths: list[Buffer] | None = None

source

Each replica’s page-aligned KV context length, when the graph takes them.

buffers

property buffers: tuple[Buffer, ...]

source

The step’s buffers, in the graph’s input order.

constants

constants: tree.Tree[Buffer] | None = None

source

The checkpoint-backed inputs, retained across steps.

data_parallel_splits

data_parallel_splits: Buffer | None = None

source

Each replica’s first sequence, under data parallelism.

ep_inputs

ep_inputs: tuple[Buffer, ...] = ()

source

The expert-parallel communication buffers, under expert parallelism.

host_input_row_offsets

host_input_row_offsets: Buffer | None = None

source

A CPU copy of input_row_offsets, under data parallelism.

input_fields()

input_fields()

source

Returns the step’s buffers as the shared role tree dataclass.

Return type:

DecoderInputs[tree.Tree[Buffer], Buffer]

input_row_offsets

input_row_offsets: Buffer

source

Each sequence’s start in tokens.

input_tree_def

input_tree_def: tree.TreeDef | None = None

source

The graph’s semantic input order, bound by its batch processor.

request_ids

request_ids: tuple[RequestID, ...] = ()

source

The step’s requests in batch order, when the decoder keeps state for them.

return_n_logits

return_n_logits: Buffer

source

How many logits per sequence to return.

scratch

scratch: tuple[Buffer, ...] = ()

source

The decoder’s workspace buffers, when it declares any.

signal_buffers

signal_buffers: tuple[Buffer, ...] = ()

source

The graph-API decoder’s allreduce signal buffers, when required.

stage_boundary

stage_boundary: tree.Tree[Buffer] | None = None

source

The preceding stage’s forwarded tensors, in its declared tree.

state

state: tuple[Buffer, ...] = ()

source

The decoder’s per-request state inputs, when it declares any.

token_positions

token_positions: Buffer | None = None

source

Each ragged token’s absolute sequence position, when declared.

tokens

tokens: Buffer

source

The batch’s ragged tokens.