For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python class
BatchProcessorRuntime
BatchProcessorRuntime
class max.pipelines.lib.interfaces.BatchProcessorRuntime(pipeline_config, devices, return_logits, max_batch_size, max_seq_len, max_batch_input_tokens, enable_chunked_prefill=True, data_parallel_degree=1, return_hidden_states=ReturnHiddenStates.NONE, signal_buffers=(), lora_manager=None, pad_token_id=0)
Bases: object
Runtime dependencies shared by batch processors.
-
Parameters:
-
- pipeline_config (PipelineConfig)
- devices (list[Device])
- return_logits (ReturnLogits)
- max_batch_size (int)
- max_seq_len (int)
- max_batch_input_tokens (int)
- enable_chunked_prefill (bool)
- data_parallel_degree (int)
- return_hidden_states (ReturnHiddenStates)
- signal_buffers (Sequence[Buffer])
- lora_manager (LoRAManagerV3 | None)
- pad_token_id (int)
data_parallel_degree
data_parallel_degree: int = 1
Replicas the scheduler fills, each to its own batch size and budget.
A processor stages every replica’s contexts into one ragged stream, so the whole product bounds the buffers it stages into.
devices
enable_chunked_prefill
enable_chunked_prefill: bool = True
Whether the scheduler splits a long prompt to stay inside that budget.
Mirrors pipeline_config.runtime.enable_chunked_prefill, which the
scheduler reads as target_tokens_per_batch_ce’s chunking policy.
lora_manager
lora_manager: LoRAManagerV3 | None = None
max_batch_active_tokens
property max_batch_active_tokens: int
Most active tokens one batch’s ragged token stream can carry.
Every context contributes its active window, and chunked prefill
holds a replica’s sum under max_batch_input_tokens by
splitting a prompt that would overflow it. With chunked prefill off
that budget is only a soft limit – the scheduler admits an
oversized request whole rather than dropping it – so one
full-length sequence can land on top of an already-full budget.
Neither case can exceed a full batch of full-length sequences, and
every batch carries at least one token per context. Each replica
gets its own budget, and they are staged into one stream.
max_batch_input_tokens
max_batch_input_tokens: int
The scheduler’s per-step budget for active (unencoded) tokens.
max_batch_size
max_batch_size: int
Most contexts the scheduler puts in one replica’s batch.
max_dims()
max_dims(dims, kv_params)
Returns the largest size each per-step input dim can take.
A serving engine allocates each per-step input once at these sizes and slices it per batch, so each bound comes from the code that sizes the same input for MAX’s own staging.
-
Parameters:
-
- dims (Iterable[str]) – Symbolic dims of a graph’s per-step inputs.
- kv_params (KVCacheParamInterface) – The KV cache parameters, which bound a lookup table’s pages per context.
-
Returns:
-
Each dim’s maximum, keyed by name.
-
Raises:
-
ValueError – If a dim is not one this runtime bounds, or is an image row dim while the vision cache is off.
-
Return type:
max_global_batch_size
property max_global_batch_size: int
Contexts in the largest batch a processor stages.
max_batch_size bounds one replica; this bounds the batch a
processor actually sees, which is every replica’s contexts staged
into one ragged stream.
max_seq_len
max_seq_len: int
Longest a single sequence can grow, after memory planning clamped it.
pad_token_id
pad_token_id: int = 0
pipeline_config
pipeline_config: PipelineConfig
return_hidden_states
return_hidden_states: ReturnHiddenStates = 'none'
return_logits
return_logits: ReturnLogits