For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

KVConnectorConfig

KVConnectorConfig

class max.pipelines.kv_cache.KVConnectorConfig(*, config_file=None, section_name=None, type=KVConnectorType.null, host_offload_max_gb=None, disk_offload_dir=None, disk_offload_max_gb=None, num_disk_workers=32, block_store_endpoint=None, **extra_data)

source

Bases: ConfigFileModel

KV cache connector configuration: the connector type and its settings.

The type travels with its settings so the two are configured as one object:

--kv-connector-config '{"type": "rust_tiered"}'

Common fields are typed. Additional connector-specific fields pass through via extra="allow" and are accessible via model_extra.

Parameters:

• config_file (str | None)
• section_name (str | None)
• type (KVConnectorType)
• host_offload_max_gb (float | None)
• disk_offload_dir (str | None)
• disk_offload_max_gb (float | None)
• num_disk_workers (int)
• block_store_endpoint (str | None)
• extra_data (Any)

block_store_endpoint

block_store_endpoint: str | None

source

Endpoint for the co-located dKV service.

Remote dKV endpoints are discovered at runtime from the Orchestrator’s per-request dkv_cache_hint, not configured statically. The connector parses the hint in Rust and dials each named instance itself.

disk_offload_dir

disk_offload_dir: str | None

source

Directory for disk-based KV cache offloading.

disk_offload_max_gb

disk_offload_max_gb: float | None

source

Maximum disk space in GiB for KV cache offloading.

None sizes it to twice the device page pool. 0 builds the connector with no disk last level: nothing is written to disk and no offload directory is created. 0 cannot mean “unlimited” here, because the disk tier derives its block capacity from this budget – a 0 that still opened a disk tier would disable eviction and grow without bound.

host_offload_max_gb

host_offload_max_gb: float | None

source

Maximum host memory in GiB for KV cache offloading. None sizes it to 1.5 times the device page pool.

model_config

model_config: ClassVar[ConfigDict] = {'extra': 'allow', 'frozen': True, 'strict': False}

source

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

num_disk_workers

num_disk_workers: int

source

Number of disk I/O worker threads for the tiered connectors.

type

type: KVConnectorType

source

Type of KV cache connector to use.