For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

KVCacheConfig

KVCacheConfig

class max.pipelines.kv_cache.KVCacheConfig(*, config_file=None, section_name=None, kv_cache_page_size=128, enable_prefix_caching=True, enable_dp_cross_replica_prefix_copy=True, kv_connector_config=<factory>, device_memory_utilization=0.9, kv_cache_format=None, indexer_kv_cache_format=None, state_pool_dtype=None, kv_cache_hash_algo='ahash64', kv_cache_hash_seed=None)

source

Bases: ConfigFileModel

Configuration for the paged KV cache.

Parameters:

• config_file (str | None)
• section_name (str | None)
• kv_cache_page_size (int)
• enable_prefix_caching (bool)
• enable_dp_cross_replica_prefix_copy (bool)
• kv_connector_config (KVConnectorConfig)
• device_memory_utilization (float)
• kv_cache_format (str | None)
• indexer_kv_cache_format (str | None)
• state_pool_dtype (str | None)
• kv_cache_hash_algo (Literal['ahash64', 'sha256'])
• kv_cache_hash_seed (str | None)

config_file

config_file: str | None

source

Path to the configuration file.

device_memory_utilization

device_memory_utilization: float

source

The fraction of available device memory the process should consume.

enable_dp_cross_replica_prefix_copy

enable_dp_cross_replica_prefix_copy: bool

source

Whether DP cross-replica prefix-cache hits may be served by device-to-device copies.

enable_prefix_caching

enable_prefix_caching: bool

source

Whether to enable prefix caching for the paged KV cache.

indexer_kv_cache_format

indexer_kv_cache_format: str | None

source

Independent IndexK cache dtype for MiniMax sparse attention.

kv_cache_format

kv_cache_format: str | None

source

An override for the default data type of the KV cache.

kv_cache_hash_algo

kv_cache_hash_algo: KVHashAlgo

source

Hash algorithm used for KV-cache block identity.

kv_cache_hash_seed

kv_cache_hash_seed: str | None

source

Optional 32-byte hex seed for KV-cache hashing.

kv_cache_page_size

kv_cache_page_size: int

source

The number of tokens in a single page in the paged KV cache.

kv_connector_config

kv_connector_config: KVConnectorConfig

source

KV cache connector configuration, including the connector type.

model_config

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True, 'strict': False}

source

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_post_init()

model_post_init(context, /)

source

This function is meant to behave like a BaseModel method to initialise private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

• self (BaseModel) – The BaseModel instance.
• context (Any) – The context.

Return type:

None

section_name

section_name: str | None

source

Optional section name for comprehensive/multi-section config files.

If not provided, values are loaded from the YAML top-level (treating the file as an “individual config” file).

state_pool_dtype

state_pool_dtype: str | None

source

An override for the storage dtype of recurrent (SSM) state pools.

to_params()

to_params(dtype, n_kv_heads, head_dim, num_layers, devices, data_parallel_degree=1, is_mla=False, num_q_heads=None, kvcache_quant_config=None, speculative_method=None, num_draft_tokens=0, allow_kv_head_replication=False, page_size=None, slots_per_page=None, window_size=None, stride_alignment=None, scratch=False)

source

Returns KVCacheParams built from this config.

Selects the attention-type-specific subclass: a MLAKVCacheParams when is_mla is set, otherwise a MHAKVCacheParams.

Parameters:

• dtype (DType) – Data type for KV cache storage.
• n_kv_heads (int) – Total number of KV heads across all devices.
• head_dim (int) – Dimension of each attention head.
• num_layers (int) – Number of model layers.
• devices (Sequence[DeviceRef]) – Devices that host the KV cache.
• data_parallel_degree (int) – Degree of data parallelism.
• is_mla (bool) – Whether the model uses Multi-Latent Attention.
• num_q_heads (int | None) – Number of query attention heads. Required when is_mla is True.
• kvcache_quant_config (KVCacheQuantizationConfig | None) – KV cache quantization configuration.
• speculative_method (Literal['eagle', 'mtp', 'dflash', 'dflash2', 'dspark'] | None) – Speculative decoding method propagated from SpeculativeConfig. None when speculative decoding is disabled.
• num_draft_tokens (int) – Total draft tokens generated per speculative iteration. Zero when no speculative decoding.
• allow_kv_head_replication (bool) – Replicate KV heads for TP wider than the KV head count. An architecture fact: implementations of construct_kv_params pass it for the head layouts that need it.
• page_size (int | None) – Tokens per KV cache page. Defaults to None (falls back to the config’s kv_cache_page_size). Architectures with a kernel-imposed minimum page size pass their effective value here instead of mutating the shared config.
• slots_per_page (int | None) – Storage slots per KV cache page, which must divide the page size evenly. Defaults to None, one slot per token. An architecture whose attention compresses a run of tokens into one cached entry passes the entries a page compresses into.
• window_size (int | None) – Window size for the attention layer.
• stride_alignment (int | None) – The page stride alignment the cache’s kernels honor, letting the pool pad its pages, or None to keep them packed. See stride_alignment.
• scratch (bool) – Whether the cache is per-request scratch: one block a request, never published (see scratch).

Returns:

The constructed KV cache parameters.

Return type:

KVCacheParams