For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

DecoderBatchProcessor

DecoderBatchProcessor

class max.pipelines.lib.module_registration.DecoderBatchProcessor(config, runtime, inputs=())

source

Bases: BatchProcessor[TextContext, DecoderModelInputs, ArchConfig]

Batches ragged text for a registered DECODER.

Under data parallelism it concatenates every replica’s batch, in replica order, and records where each replica’s sequences start. It also fills the optional inputs the bound graph takes: each replica’s context length and the expert-parallel buffers.

Parameters:

• config (ArchConfigT_co)
• runtime (BatchProcessorRuntime)
• inputs (Sequence[InputDescriptor])

bind_ep_comm_initializer()

bind_ep_comm_initializer(initializer)

source

Binds the expert-parallel buffers each batch passes the graph.

Parameters:

initializer (ExpertParallelExchange | None)

Return type:

None

bind_input_tree()

bind_input_tree(input_tree)

source

Binds the graph’s inputs, and sizes the ones this fills.

Parameters:

input_tree (OrderedDict[str, object])

Return type:

None

constant_inputs

constant_inputs: ConstantInputs | None = None

source

The deployment’s constant input trees, owned for its lifetime.

get_symbolic_inputs()

get_symbolic_inputs(*, kv_params, device_refs)

source

Returns the bound graph’s inputs, or its defaults before binding.

Parameters:

• kv_params (KVCacheParamInterface)
• device_refs (list[DeviceRef])

Return type:

list[Any]

max_dims()

max_dims(dims, kv_params)

source

Returns the largest size each per-step input dim can take.

The state store bounds the dims its inputs introduce, and the runtime the rest.

Raises:

ValueError – If neither bounds a dim.

Parameters:

• dims (Iterable[str])
• kv_params (KVCacheParamInterface)

Return type:

dict[str, int]

prepare_initial_token_inputs()

prepare_initial_token_inputs(replica_batches, kv_cache_inputs=None, return_n_logits=1)

source

Packs the replicas’ contexts into one ragged batch.

Raises:

ValueError – If there is a batch for other than one replica per data-parallel rank.

Parameters:

• replica_batches (Sequence[Sequence[TextContext]])
• kv_cache_inputs (KVCacheInputs[Buffer, Buffer] | None)
• return_n_logits (int)

Return type:

DecoderModelInputs

process_outputs()

process_outputs(outputs)

source

Maps raw execution buffers to ModelOutputs.

Parameters:

outputs (Sequence[Buffer | object])

Return type:

ModelOutputs

scratch

scratch: DecoderScratch | None = None

source

The decoder’s workspace, which every batch passes. The pipeline model sets it once it allocates the buffers.

state_store

state_store: RequestStateStore | None = None

source

The decoder’s per-request state store, which fills the batch’s state inputs. The pipeline model sets it once it builds the store.