For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

KVConnectorType

KVConnectorType

class max.nn.kv_cache.KVConnectorType(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)

source

Bases: str, Enum

Identifies which off-device backing store the KV cache uses.

Set on KVCacheParams.kv_connector to control whether evicted cache pages stay on device only, spill to host memory, tier across host and disk, or route through a distributed block store.

dkv

dkv = 'dkv'

source

Routes pages through a distributed KV block store.

Requires a block_store_endpoint on the connector config.

local

local = 'local'

source

Spills evicted pages to host memory.

Requires enable_prefix_caching and host_kvcache_swap_space_gb to be set on KVCacheParams.

null

null = 'null'

source

No off-device backing store. Pages live on device only.

rust_tiered

rust_tiered = 'rust_tiered'

source

Tiers evicted pages across host memory and disk, backed by the Rust kv_tier_connector extension.

The performant, CUDA-only successor to tiered: it runs its copies and disk I/O on Rust threads (no GIL contention) and overlaps onloads with GPU compute via asynchronous transfer handles. Same config requirements as tiered. Falls back with an error on non-CUDA devices.

tiered

tiered = 'tiered'

source

Tiers evicted pages across host memory and disk.

Requires enable_prefix_caching, host_kvcache_swap_space_gb, and a disk_offload_dir on the connector config.