IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

PipelineRuntimeConfig

PipelineRuntimeConfig​

class max.pipelines.lib.PipelineRuntimeConfig(*, config_file=None, section_name=None, pipeline_role='prefill_and_decode', max_batch_size=None, precompiled_mefs=None, export_mefs=None, max_queue_size_tg=None, min_batch_size_tg=None, ep_size=1, ep_use_allreduce=False, eplb_profile=<factory>, ce_delay_ms=0.0, enable_prioritize_first_decode=False, enable_chunked_prefill=True, chunked_prefill_min_chunk_size=0, enable_in_flight_batching=False, enable_spec_decode_mixed_batches=False, eplb_replicas_per_gpu=0, max_batch_input_tokens=8192, use_experimental_kernels='false', use_vendor_blas='false', use_vendor_ccl='false', custom_architectures=<factory>, execute_empty_batches=False, max_batch_total_tokens=None, device_graph_capture=None, experimental_device_graph_synthesis=False, force=False, decode_stall_timeout_s=None, decode_request_ttl_s=None, enable_overlap_scheduler=False, dp_ce_balance_timeout_ms=-1.0, dp_ce_balance_threshold=0.8, dp_ce_balance_enable_dynamic_chunk_size=False, allow_unsupported_logprobs=False, allow_extra_request_fields=False, prefer_module_v3=False, reasoning_parser=None, tool_parser=None, emit_reasoning_content=False, temperature=None, top_k=None, thinking_temperature=None, vision_cache_utilization=0.05, max_vision_preprocess_cache_bytes=10737418240, max_video_preprocess_cache_bytes=10737418240, max_media_preprocess_cache_idle_seconds=300.0, denoising_cache=DenoisingCacheConfig(config_file=None, section_name=None, first_block_caching=False, taylorseer=False, taylorseer_cache_interval=5, taylorseer_warmup_steps=9, taylorseer_max_order=1))

source

Bases: ConfigFileModel

Model-agnostic runtime settings for pipeline execution.

Contains batching, scheduling, and execution configuration that is independent of any particular model architecture.

Parameters:

  • config_file (str | None)
  • section_name (str | None)
  • pipeline_role (Literal['prefill_and_decode', 'prefill_only', 'decode_only'])
  • max_batch_size (int | None)
  • precompiled_mefs (str | None)
  • export_mefs (str | None)
  • max_queue_size_tg (int | None)
  • min_batch_size_tg (int | None)
  • ep_size (int)
  • ep_use_allreduce (bool)
  • eplb_profile (bool)
  • ce_delay_ms (float)
  • enable_prioritize_first_decode (bool)
  • enable_chunked_prefill (bool)
  • chunked_prefill_min_chunk_size (Annotated[int, Ge(ge=0)])
  • enable_in_flight_batching (bool)
  • enable_spec_decode_mixed_batches (bool)
  • eplb_replicas_per_gpu (int)
  • max_batch_input_tokens (int)
  • use_experimental_kernels (str)
  • use_vendor_blas (str)
  • use_vendor_ccl (str)
  • custom_architectures (list[str])
  • execute_empty_batches (bool)
  • max_batch_total_tokens (int | None)
  • device_graph_capture (bool | None)
  • experimental_device_graph_synthesis (bool)
  • force (bool)
  • decode_stall_timeout_s (float | None)
  • decode_request_ttl_s (float | None)
  • enable_overlap_scheduler (bool)
  • dp_ce_balance_timeout_ms (float)
  • dp_ce_balance_threshold (float)
  • dp_ce_balance_enable_dynamic_chunk_size (bool)
  • allow_unsupported_logprobs (bool)
  • allow_extra_request_fields (bool)
  • prefer_module_v3 (bool)
  • reasoning_parser (str | None)
  • tool_parser (str | None)
  • emit_reasoning_content (bool)
  • temperature (float | None)
  • top_k (int | None)
  • thinking_temperature (float | None)
  • vision_cache_utilization (Annotated[float, Ge(ge=0), Le(le=1)])
  • max_vision_preprocess_cache_bytes (int)
  • max_video_preprocess_cache_bytes (int)
  • max_media_preprocess_cache_idle_seconds (float)
  • denoising_cache (DenoisingCacheConfig)

allow_extra_request_fields​

allow_extra_request_fields: bool

source

allow_unsupported_logprobs​

allow_unsupported_logprobs: bool

source

ce_delay_ms​

ce_delay_ms: float

source

chunked_prefill_min_chunk_size​

chunked_prefill_min_chunk_size: int

source

Minimum tokens in any chunk created by chunked prefill (0 = off).

config_file​

config_file: str | None

source

Path to the configuration file.

custom_architectures​

custom_architectures: list[str]

source

decode_request_ttl_s​

decode_request_ttl_s: float | None

source

decode_stall_timeout_s​

decode_stall_timeout_s: float | None

source

denoising_cache​

denoising_cache: DenoisingCacheConfig

source

device_graph_capture​

device_graph_capture: bool | None

source

dp_ce_balance_enable_dynamic_chunk_size​

dp_ce_balance_enable_dynamic_chunk_size: bool

source

Whether below-threshold CE steps run at a reduced chunk size.

dp_ce_balance_threshold​

dp_ce_balance_threshold: float

source

Occupancy threshold (0-1) that schedules CE work without deferral.

dp_ce_balance_timeout_ms​

dp_ce_balance_timeout_ms: float

source

Deferral deadline for DP-balanced CE scheduling (-1 = disabled).

emit_reasoning_content​

emit_reasoning_content: bool

source

enable_chunked_prefill​

enable_chunked_prefill: bool

source

enable_in_flight_batching​

enable_in_flight_batching: bool

source

enable_overlap_scheduler​

enable_overlap_scheduler: bool

source

enable_prioritize_first_decode​

enable_prioritize_first_decode: bool

source

enable_spec_decode_mixed_batches​

enable_spec_decode_mixed_batches: bool

source

ep_size​

ep_size: int

source

ep_use_allreduce​

ep_use_allreduce: bool

source

eplb_profile​

eplb_profile: bool

source

eplb_replicas_per_gpu​

eplb_replicas_per_gpu: int

source

execute_empty_batches​

execute_empty_batches: bool

source

experimental_device_graph_synthesis​

experimental_device_graph_synthesis: bool

source

export_mefs​

export_mefs: str | None

source

force​

force: bool

source

is_disaggregated​

property is_disaggregated: bool

source

Whether this worker is part of a disaggregated prefill/decode deployment.

max_batch_input_tokens​

max_batch_input_tokens: int

source

max_batch_size​

max_batch_size: int | None

source

max_batch_total_tokens​

max_batch_total_tokens: int | None

source

max_media_preprocess_cache_idle_seconds​

max_media_preprocess_cache_idle_seconds: float

source

max_queue_size_tg​

max_queue_size_tg: int | None

source

max_video_preprocess_cache_bytes​

max_video_preprocess_cache_bytes: int

source

max_vision_preprocess_cache_bytes​

max_vision_preprocess_cache_bytes: int

source

min_batch_size_tg​

min_batch_size_tg: int | None

source

model_config​

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True, 'strict': False}

source

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init()​

model_post_init(context, /)

source

This function is meant to behave like a BaseModel method to initialise private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

  • self (BaseModel) – The BaseModel instance.
  • context (Any) – The context.

Return type:

None

pipeline_role​

pipeline_role: PipelineRole

source

precompiled_mefs​

precompiled_mefs: str | None

source

prefer_module_v3​

prefer_module_v3: bool

source

reasoning_parser​

reasoning_parser: str | None

source

section_name​

section_name: str | None

source

Optional section name for comprehensive/multi-section config files.

If not provided, values are loaded from the YAML top-level (treating the file as an “individual config” file).

temperature​

temperature: float | None

source

thinking_temperature​

thinking_temperature: float | None

source

tool_parser​

tool_parser: str | None

source

top_k​

top_k: int | None

source

use_experimental_kernels​

use_experimental_kernels: str

source

use_vendor_blas​

use_vendor_blas: str

source

use_vendor_ccl​

use_vendor_ccl: str

source

vision_cache_utilization​

vision_cache_utilization: float

source

Models

Cookbook

Puzzles

Blog

Community