For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python module

max.pipelines.kv_cache

KV cache management for MAX pipelines.

Submodules

max.pipelines.kv_cache.paged_kv_cache.kv_group_coordinator	Cache group coordinators: one per set of leaves written in lockstep.
max.pipelines.kv_cache.paged_kv_cache.recurrent_coordinator	The cache group whose entry is a recurrent state rather than a span.

Memory planning

MemoryPlanner	Base class for pipeline model memory planning.
ModelConfig	Structural protocol for model configuration consumed by MemoryPlanner.
ModelConfigWithKVCache	Extension of ModelConfig for models with a KV cache.
PagedMemoryPlanner	Memory planner for models that use a paged KV cache.

Configuration

KVCacheConfig	Configuration for the paged KV cache.
KVConnectorConfig	KV cache connector configuration: the connector type and its settings.

Cache manager

ByteCount	A point-in-time snapshot of an external cache tier's occupancy, in bytes.
DummyKVCache	No-op KV cache implementation for testing or when cache is disabled.
InsufficientBlocksError	Exception raised when there are insufficient free blocks to satisfy an allocation.
PagedKVCacheManager	Paged KVCache manager with data and tensor parallelism support.

Transfer engine

KVTransferEngine	KVCache Transfer Engine with support for Data Parallelism (DP) and Tensor Parallelism (TP).
KVTransferEngineMetadata	Metadata associated with a KV cache transfer engine.
TransferReqData	Metadata associated with a transfer request.

Factory functions

available_port	Finds an available TCP port in the given range.
kv_cache_memory_size	Returns the KV bytes the manager for this cache needs for a full batch.
load_kv_manager	Loads a KV cache manager from the given params.
max_seq_len_fitting_in_cache	Returns the longest request the manager for this cache holds.

Utilities

cache_dtype_for_encoding	Returns the KV-cache dtype for a quantization encoding.
is_fp4_kv_cache_format	Returns whether kv_cache_format names the NVFP4 KV cache format.