For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

MemoryPlan

MemoryPlan

class max.pipelines.lib.MemoryPlan(planned_max_batch_size, footprint, planned_max_length, available_cache_memory=None, device_specs=None, planned_max_batch_total_tokens=None, vision_cache_plan=None)

source

Bases: object

The memory plan computed when a pipeline is loaded.

Carries the effective batch size, sequence-length bound, and memory budgets that the pipeline and its schedulers consume. Produced by MemoryEstimator.plan().

Parameters:

• planned_max_batch_size (int)
• footprint (int)
• planned_max_length (int | None)
• available_cache_memory (int | None)
• device_specs (tuple[DeviceSpec, ...] | None)
• planned_max_batch_total_tokens (int | None)
• vision_cache_plan (VisionCachePlan | None)

available_cache_memory

available_cache_memory: int | None = None

source

The device memory committed to the KV cache, in bytes. None when the plan reserves no KV-cache budget, such as for models without a KV cache.

device_specs

device_specs: tuple[DeviceSpec, ...] | None = None

source

The device specs the plan was computed for, kept as specs rather than Device objects so the plan can cross process boundaries. None for plans that never load devices, such as diffusion pipelines.

footprint

footprint: int

source

The estimated total device memory the pipeline uses, in bytes.

planned_max_batch_size

planned_max_batch_size: int

source

the user’s runtime.max_batch_size, or the value planning inferred when the user left it unset.

Type:

The maximum number of requests scheduled together in one batch

planned_max_batch_total_tokens

planned_max_batch_total_tokens: int | None = None

source

the user’s runtime.max_batch_total_tokens, or planned_max_length for architectures that require a cap. None means no cap is configured.

Type:

Cap on the total context tokens resident across a batch

planned_max_length

planned_max_length: int | None

source

The resolved maximum sequence length after memory planning lowered it to fit device memory: the construction-resolved config.model.max_length, clamped to what the KV cache and any draft model can hold. None for pipelines with no main language model, such as diffusion pipelines.

require_device_specs()

require_device_specs()

source

Returns the device specs, which must be set on this plan.

Raises:

AssertionError – If the plan carries no device specs.

Return type:

tuple[DeviceSpec, …]

vision_cache_plan

vision_cache_plan: VisionCachePlan | None = None

source

Block-mode vision cache reservation; None means entry-count mode.