For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python class
PipelineRuntimeConfig
PipelineRuntimeConfig
class max.pipelines.lib.PipelineRuntimeConfig(*, config_file=None, section_name=None, pipeline_role='prefill_and_decode', max_batch_size=None, precompiled_mefs=None, export_mefs=None, max_queue_size_tg=None, min_batch_size_tg=None, ep_size=1, ep_use_allreduce=False, eplb_profile=<factory>, ce_delay_ms=0.0, enable_prioritize_first_decode=False, enable_chunked_prefill=True, chunked_prefill_min_chunk_size=0, enable_in_flight_batching=False, eplb_replicas_per_gpu=0, max_batch_input_tokens=8192, use_experimental_kernels='false', use_vendor_blas='false', use_vendor_ccl='false', custom_architectures=<factory>, execute_empty_batches=False, max_batch_total_tokens=None, device_graph_capture=None, fold_sampler_into_graph=True, force=False, max_pending_futures=1, decode_stall_timeout_s=None, decode_request_ttl_s=None, enable_overlap_scheduler=False, dp_ce_balance_timeout_ms=-1.0, dp_ce_balance_threshold=0.8, dp_ce_balance_enable_dynamic_chunk_size=False, allow_unsupported_logprobs=False, allow_extra_request_fields=False, prefer_module_v3=False, reasoning_parser=None, tool_parser=None, emit_reasoning_content=False, temperature=None, top_k=None, thinking_temperature=None, max_vision_cache_entries=256, denoising_cache=<factory>)
Bases: ConfigFileModel
Model-agnostic runtime settings for pipeline execution.
Contains batching, scheduling, and execution configuration that is independent of any particular model architecture.
-
Parameters:
-
- config_file (str | None)
- section_name (str | None)
- pipeline_role (Literal['prefill_and_decode', 'prefill_only', 'decode_only'])
- max_batch_size (int | None)
- precompiled_mefs (str | None)
- export_mefs (str | None)
- max_queue_size_tg (int | None)
- min_batch_size_tg (int | None)
- ep_size (int)
- ep_use_allreduce (bool)
- eplb_profile (bool)
- ce_delay_ms (float)
- enable_prioritize_first_decode (bool)
- enable_chunked_prefill (bool)
- chunked_prefill_min_chunk_size (int)
- enable_in_flight_batching (bool)
- eplb_replicas_per_gpu (int)
- max_batch_input_tokens (int)
- use_experimental_kernels (str)
- use_vendor_blas (str)
- use_vendor_ccl (str)
- custom_architectures (list[str])
- execute_empty_batches (bool)
- max_batch_total_tokens (int | None)
- device_graph_capture (bool | None)
- fold_sampler_into_graph (bool)
- force (bool)
- max_pending_futures (int)
- decode_stall_timeout_s (float | None)
- decode_request_ttl_s (float | None)
- enable_overlap_scheduler (bool)
- dp_ce_balance_timeout_ms (float)
- dp_ce_balance_threshold (float)
- dp_ce_balance_enable_dynamic_chunk_size (bool)
- allow_unsupported_logprobs (bool)
- allow_extra_request_fields (bool)
- prefer_module_v3 (bool)
- reasoning_parser (str | None)
- tool_parser (str | None)
- emit_reasoning_content (bool)
- temperature (float | None)
- top_k (int | None)
- thinking_temperature (float | None)
- max_vision_cache_entries (int)
- denoising_cache (DenoisingCacheConfig)
allow_extra_request_fields
allow_extra_request_fields: bool
allow_unsupported_logprobs
allow_unsupported_logprobs: bool
ce_delay_ms
ce_delay_ms: float
chunked_prefill_min_chunk_size
chunked_prefill_min_chunk_size: int
Minimum tokens in any chunk created by chunked prefill (0 = off).
custom_architectures
decode_request_ttl_s
decode_stall_timeout_s
denoising_cache
denoising_cache: DenoisingCacheConfig
device_graph_capture
dp_ce_balance_enable_dynamic_chunk_size
dp_ce_balance_enable_dynamic_chunk_size: bool
Whether below-threshold CE steps run at a reduced chunk size.
dp_ce_balance_threshold
dp_ce_balance_threshold: float
Occupancy threshold (0-1) that schedules CE work without deferral.
dp_ce_balance_timeout_ms
dp_ce_balance_timeout_ms: float
Deferral deadline for DP-balanced CE scheduling (-1 = disabled).
emit_reasoning_content
emit_reasoning_content: bool
enable_chunked_prefill
enable_chunked_prefill: bool
enable_in_flight_batching
enable_in_flight_batching: bool
enable_overlap_scheduler
enable_overlap_scheduler: bool
enable_prioritize_first_decode
enable_prioritize_first_decode: bool
ep_size
ep_size: int
ep_use_allreduce
ep_use_allreduce: bool
eplb_profile
eplb_profile: bool
eplb_replicas_per_gpu
eplb_replicas_per_gpu: int
execute_empty_batches
execute_empty_batches: bool
export_mefs
fold_sampler_into_graph
fold_sampler_into_graph: bool
force
force: bool
max_batch_input_tokens
max_batch_input_tokens: int
max_batch_size
max_batch_total_tokens
max_pending_futures
max_pending_futures: int
max_queue_size_tg
max_vision_cache_entries
max_vision_cache_entries: int
min_batch_size_tg
model_config
model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'strict': False}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
model_post_init()
model_post_init(context, /)
This function is meant to behave like a BaseModel method to initialise private attributes.
It takes context as an argument since that’s what pydantic-core passes when calling it.
-
Parameters:
-
- self (BaseModel) – The BaseModel instance.
- context (Any) – The context.
-
Return type:
-
None
pipeline_role
pipeline_role: PipelineRole
precompiled_mefs
prefer_module_v3
prefer_module_v3: bool
reasoning_parser
temperature
thinking_temperature
tool_parser
top_k
use_experimental_kernels
use_experimental_kernels: str
use_vendor_blas
use_vendor_blas: str
use_vendor_ccl
use_vendor_ccl: str