For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

KVConnectorConfig

KVConnectorConfig

class max.pipelines.kv_cache.KVConnectorConfig(*, config_file=None, section_name=None, type=KVConnectorType.null, host_offload_max_gb=None, host_staging_alloc='registered', host_staging_mlock=False, disk_offload_dir=None, disk_offload_max_gb=None, num_disk_workers=32, num_h2d_onload_lanes=1, use_disk_direct_io=False, block_store_endpoint=None, **extra_data)

source

Bases: ConfigFileModel

KV cache connector configuration: the connector type and its settings.

The type travels with its settings so the two are configured as one object:

--kv-connector-config '{"type": "rust_tiered"}'

Common fields are typed. Additional connector-specific fields pass through via extra="allow" and are accessible via model_extra.

Parameters:

• config_file (str | None)
• section_name (str | None)
• type (KVConnectorType)
• host_offload_max_gb (float | None)
• host_staging_alloc (Literal['registered', 'driver'])
• host_staging_mlock (bool)
• disk_offload_dir (str | None)
• disk_offload_max_gb (float | None)
• num_disk_workers (int)
• num_h2d_onload_lanes (int)
• use_disk_direct_io (bool)
• block_store_endpoint (str | None)
• extra_data (Any)

block_store_endpoint

block_store_endpoint: str | None

source

Endpoint for the co-located dKV service.

Remote dKV endpoints are discovered at runtime from the Orchestrator’s per-request dkv_cache_hint, not configured statically. The connector parses the hint in Rust and dials each named instance itself.

disk_offload_dir

disk_offload_dir: str | None

source

Directory for disk-based KV cache offloading.

disk_offload_max_gb

disk_offload_max_gb: float | None

source

Maximum disk space in GiB for KV cache offloading.

None sizes it to twice the device page pool. 0 builds the connector with no disk last level: nothing is written to disk and no offload directory is created. 0 cannot mean “unlimited” here, because the disk tier derives its block capacity from this budget – a 0 that still opened a disk tier would disable eviction and grow without bound.

host_offload_max_gb

host_offload_max_gb: float | None

source

Maximum host memory in GiB for KV cache offloading. None sizes it to 1.5 times the device page pool.

host_staging_alloc

host_staging_alloc: HostStagingAlloc

source

How the tiered connector allocates its pinned host staging region.

registered (the default) maps ordinary anonymous host memory and registers it with the GPU. Those pages stay movable, so when the kernel compacts memory it can migrate them, and on AMD each migration evicts the process’s GPU queues until the registration is revalidated.

driver allocates the region as pinned host memory from the GPU driver, in chunks of whole huge blocks. On AMD the HIP runtime still backs such allocations with movable user pages unless HSA_USERPTR_FOR_PAGED_MEM=0 is set before it initializes; with it set they are driver-owned and count against the driver’s system memory limit, which every process on the node shares. Only the tiered connector (rust_tiered) reads this.

host_staging_mlock

host_staging_mlock: bool

source

Whether the registered host staging region is locked with mlock.

model_config

model_config: ClassVar[ConfigDict] = {'extra': 'allow', 'frozen': True, 'strict': False}

source

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

num_disk_workers

num_disk_workers: int

source

Number of disk I/O worker threads for the tiered connector.

num_h2d_onload_lanes

num_h2d_onload_lanes: int

source

Concurrent H2D onload lanes for the rust_tiered connector.

type

type: KVConnectorType

source

Type of KV cache connector to use.

use_disk_direct_io

use_disk_direct_io: bool

source

Whether the disk tier’s reads and writes bypass the page cache (O_DIRECT).