For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python function
create_connector
create_connector()
max.pipelines.kv_cache.connectors.create_connector(leaves, devices, replica_kv_memory, params, device_memory_bytes)
Create a KV cache connector instance from params.kv_connector_config.
A single connector serves every DP replica for all connector types:
replica_kv_memory holds each replica’s device buffers, and load/offload
select the replica via replica_idx (SERVOPT-1501). The host/disk tiers
back this with one shared pinned host buffer / disk cache, which the tiered
connector sizes from the config’s host_offload_max_gb /
disk_offload_max_gb (or from its own device page pool when either is
unset); the distributed dkv connector owns one Rust client per replica
internally.
tiered is a backward-compatible alias for the Rust rust_tiered
connector, which replaced its deleted Python implementation, and therefore
inherits rust_tiered’s CUDA/HIP requirement.
-
Parameters:
-
- devices (Sequence[Device]) – Devices for the KV cache tensors (all participating devices).
- replica_kv_memory (Sequence[Mapping[str, KVCacheMemory]]) – Per-replica offload-ready KV memory units, one
mapping per DP replica, keyed by the leaf ids
params.leaves()names. - params (KVCacheParamInterface) – KV-cache parameters. Carries the connector config (type and
settings); the
dkvconnector also uses them to derive its multi-tenant per-GPU handshake identity. - device_memory_bytes (int) – The device page pool this connector sizes its
tiers against. The caller states it rather than deriving it from
params, because the managers measure it differently: the paged manager counts its flat pages, Jenga its huge blocks. - leaves (Mapping[str, KVCacheGroupId])
-
Returns:
-
A connector instance implementing the KVConnector protocol.
-
Return type: