For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python class
PipelineRuntimeConfig
PipelineRuntimeConfig
class max.pipelines.PipelineRuntimeConfig(*, config_file=None, section_name=None, pipeline_role='prefill_and_decode', max_batch_size=None, precompiled_mefs=None, export_mefs=None, max_queue_size_tg=None, min_batch_size_tg=None, ep_size=1, ep_use_allreduce=False, ep_fuse_ffn_combine_send=None, eplb_profile=<factory>, ce_delay_ms=0.0, enable_prioritize_first_decode=False, enable_chunked_prefill=True, chunked_prefill_min_chunk_size=0, enable_in_flight_batching=False, enable_spec_decode_mixed_batches=False, prefill_coalesce_min_pending=0, prefill_coalesce_max_held_steps=0, prefill_schedule_interval=1, eplb_replicas_per_gpu=0, max_batch_input_tokens=8192, max_request_input_tokens=0, use_experimental_kernels='false', use_vendor_blas='false', use_vendor_ccl='false', custom_architectures=<factory>, execute_empty_batches=False, max_batch_total_tokens=None, device_graph_capture=None, experimental_device_graph_synthesis=False, force=False, decode_stall_timeout_s=None, decode_request_ttl_s=None, enable_overlap_scheduler=False, disable_overlap=<factory>, disable_vendor_blas_fallback=<factory>, dp_ce_balance_timeout_ms=-1.0, dp_ce_balance_threshold=0.8, dp_ce_balance_enable_dynamic_chunk_size=False, allow_unsupported_logprobs=False, allow_extra_request_fields=False, prefer_module_v3=False, eager_usage_validator=EagerValidatorMode.INIT_ONLY, reasoning_parser=None, tool_parser=None, emit_reasoning_content=False, temperature=None, top_k=None, thinking_temperature=None, vision_cache_utilization=0.05, max_vision_preprocess_cache_bytes=10737418240, max_video_preprocess_cache_bytes=10737418240, max_media_preprocess_cache_idle_seconds=300.0, denoising_cache=DenoisingCacheConfig(config_file=None, section_name=None, first_block_caching=False, taylorseer=False, taylorseer_cache_interval=5, taylorseer_warmup_steps=9, taylorseer_max_order=1))
Bases: ConfigFileModel
Model-agnostic runtime settings for pipeline execution.
Contains batching, scheduling, and execution configuration that is independent of any particular model architecture.
-
Parameters:
-
- config_file (str | None)
- section_name (str | None)
- pipeline_role (Literal['prefill_and_decode', 'prefill_only', 'decode_only'])
- max_batch_size (int | None)
- precompiled_mefs (str | None)
- export_mefs (str | None)
- max_queue_size_tg (int | None)
- min_batch_size_tg (int | None)
- ep_size (int)
- ep_use_allreduce (bool)
- ep_fuse_ffn_combine_send (bool | None)
- eplb_profile (bool)
- ce_delay_ms (float)
- enable_prioritize_first_decode (bool)
- enable_chunked_prefill (bool)
- chunked_prefill_min_chunk_size (int)
- enable_in_flight_batching (bool)
- enable_spec_decode_mixed_batches (bool)
- prefill_coalesce_min_pending (int)
- prefill_coalesce_max_held_steps (int)
- prefill_schedule_interval (int)
- eplb_replicas_per_gpu (int)
- max_batch_input_tokens (int)
- max_request_input_tokens (int)
- use_experimental_kernels (str)
- use_vendor_blas (str)
- use_vendor_ccl (str)
- custom_architectures (list[str])
- execute_empty_batches (bool)
- max_batch_total_tokens (int | None)
- device_graph_capture (bool | None)
- experimental_device_graph_synthesis (bool)
- force (bool)
- decode_stall_timeout_s (float | None)
- decode_request_ttl_s (float | None)
- enable_overlap_scheduler (bool)
- disable_overlap (bool)
- disable_vendor_blas_fallback (bool)
- dp_ce_balance_timeout_ms (float)
- dp_ce_balance_threshold (float)
- dp_ce_balance_enable_dynamic_chunk_size (bool)
- allow_unsupported_logprobs (bool)
- allow_extra_request_fields (bool)
- prefer_module_v3 (bool)
- eager_usage_validator (EagerValidatorMode)
- reasoning_parser (str | None)
- tool_parser (str | None)
- emit_reasoning_content (bool)
- temperature (float | None)
- top_k (int | None)
- thinking_temperature (float | None)
- vision_cache_utilization (float)
- max_vision_preprocess_cache_bytes (int)
- max_video_preprocess_cache_bytes (int)
- max_media_preprocess_cache_idle_seconds (float)
- denoising_cache (DenoisingCacheConfig)
allow_extra_request_fields
allow_extra_request_fields: bool
allow_unsupported_logprobs
allow_unsupported_logprobs: bool
ce_delay_ms
ce_delay_ms: float
chunked_prefill_min_chunk_size
chunked_prefill_min_chunk_size: int
Minimum tokens in any chunk created by chunked prefill (0 = off).
custom_architectures
decode_request_ttl_s
decode_stall_timeout_s
denoising_cache
denoising_cache: DenoisingCacheConfig
device_graph_capture
disable_overlap
disable_overlap: bool
disable_vendor_blas_fallback
disable_vendor_blas_fallback: bool
dp_ce_balance_enable_dynamic_chunk_size
dp_ce_balance_enable_dynamic_chunk_size: bool
Whether below-threshold CE steps run at a reduced chunk size.
dp_ce_balance_threshold
dp_ce_balance_threshold: float
Occupancy threshold (0-1) that schedules CE work without deferral.
dp_ce_balance_timeout_ms
dp_ce_balance_timeout_ms: float
Deferral deadline for DP-balanced CE scheduling (-1 = disabled).
eager_usage_validator
eager_usage_validator: EagerValidatorMode
emit_reasoning_content
emit_reasoning_content: bool
enable_chunked_prefill
enable_chunked_prefill: bool
enable_in_flight_batching
enable_in_flight_batching: bool
enable_overlap_scheduler
enable_overlap_scheduler: bool
enable_prioritize_first_decode
enable_prioritize_first_decode: bool
enable_spec_decode_mixed_batches
enable_spec_decode_mixed_batches: bool
ep_fuse_ffn_combine_send
ep_size
ep_size: int
ep_use_allreduce
ep_use_allreduce: bool
eplb_profile
eplb_profile: bool
eplb_replicas_per_gpu
eplb_replicas_per_gpu: int
execute_empty_batches
execute_empty_batches: bool
experimental_device_graph_synthesis
experimental_device_graph_synthesis: bool
export_mefs
force
force: bool
is_disaggregated
property is_disaggregated: bool
Whether this worker is part of a disaggregated prefill/decode deployment.
max_batch_input_tokens
max_batch_input_tokens: int
max_batch_size
max_batch_total_tokens
max_media_preprocess_cache_idle_seconds
max_media_preprocess_cache_idle_seconds: float
max_num_input_tokens
Tokens one forward encodes, or None when a prompt is encoded whole.
Chunked prefill cuts a CE batch at max_batch_input_tokens, which is
also what a request has to be allocated for. Without it a prompt is
admitted in full, so nothing bounds the allocation.
max_queue_size_tg
max_request_input_tokens
max_request_input_tokens: int
Per-request ceiling on prefill tokens per batch (0 = off).
max_video_preprocess_cache_bytes
max_video_preprocess_cache_bytes: int
max_vision_preprocess_cache_bytes
max_vision_preprocess_cache_bytes: int
min_batch_size_tg
model_config
model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True, 'strict': False}
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
model_post_init()
model_post_init(context, /)
This function is meant to behave like a BaseModel method to initialise private attributes.
It takes context as an argument since that’s what pydantic-core passes when calling it.
-
Parameters:
-
- self (BaseModel) – The BaseModel instance.
- context (Any) – The context.
-
Return type:
-
None
pipeline_role
pipeline_role: PipelineRole
precompiled_mefs
prefer_module_v3
prefer_module_v3: bool
prefill_coalesce_max_held_steps
prefill_coalesce_max_held_steps: int
prefill_coalesce_min_pending
prefill_coalesce_min_pending: int
prefill_schedule_interval
prefill_schedule_interval: int
reasoning_parser
temperature
thinking_temperature
tool_parser
top_k
use_experimental_kernels
use_experimental_kernels: str
use_vendor_blas
use_vendor_blas: str
use_vendor_ccl
use_vendor_ccl: str
vision_cache_utilization
vision_cache_utilization: float