For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

NullConnector

NullConnector

class max.pipelines.kv_cache.connectors.NullConnector(*args, **kwargs)

source

Bases: KVConnector

No-op connector for when external caching is disabled.

leaves

property leaves: Mapping[str, KVCacheGroupId]

source

Returns a mapping from leaf_id to group_id serviced by the connector.

load()

load(block_ids, block_hashes, replica_idx=0, hint=None)

source

Loads exactly the blocks it is told to, leaf by leaf.

block_hashes[leaf_id][i] is read into block_ids[leaf_id][i], so the two are the same length on every leaf. Which hashes those are – the whole prefix for a full-attention leaf, a window’s worth for a windowed one – the caller settled from a lookup(). The slots below a leaf’s share are the null block, which the caller fills in; this never sees them.

Must follow a lookup() with the same replica_idx and hint, at most once, since a connector may be reading out of what that lookup found.

Parameters:

• block_ids (Mapping[str, Sequence[int]]) – Device block IDs to load into, per leaf. Leaf counts differ by design: a full-attention leaf takes one block per hash of the hit, a windowed leaf only its window’s worth.
• block_hashes (Mapping[str, Sequence[bytes]]) – The hashes to read, per leaf, in the order that leaf’s row holds them. Canonical bytes form (8 big-endian bytes for ahash64-family, 32 bytes for SHA-256).
• replica_idx (int) – DP replica whose device buffers receive the loaded blocks. The external tier itself is replica-agnostic (keyed by hash); this only selects the H2D destination.
• hint (bytes | None) – The request’s dkv_cache_hint as raw JSON bytes, or None when it carried none. Opaque to the manager: only the dKV connector reads it, to route blocks to the peer that holds them. It never affects correctness, since an unusable hint costs a cache miss and nothing else.

Returns:

A KVTransfer for the H2D copy, which the manager polls before reading the loaded KV. A load that moved nothing returns a CompletedTransfer.

Raises:

• KVLoadRefused – If it cannot move every block it was asked for. Loading less is not an option – see KVLoadRefused.
• KVLoadFailed – If posting the copy failed. A copy that fails after it was posted raises from the returned transfer instead.

Return type:

KVTransfer

lookup()

lookup(block_hashes, replica_idx=0, hint=None)

source

Holds nothing, so every block is absent.

Parameters:

• block_hashes (Sequence[bytes])
• replica_idx (int)
• hint (bytes | None)

Return type:

Mapping[str, Sequence[bool]]

metrics

property metrics: KVCacheMetrics

source

Transfer metrics for this connector. Returns empty metrics by default.

name

property name: str

source

Connector name for logging/debugging.

offload()

offload(block_ids, block_hashes, replica_idx=0)

source

Offloads each leaf’s blocks under that leaf’s hashes.

block_ids[leaf_id][i] is written under the hash block_hashes[leaf_id][i]. Leaves may differ in length, since a recurrent leaf publishes only at its checkpoint boundaries.

Parameters:

• block_ids (Mapping[str, Sequence[int]]) – Device block IDs to offload, per leaf.
• block_hashes (Mapping[str, Sequence[bytes]]) – Hashes per leaf, in prefix order. Canonical bytes form (8 big-endian bytes for ahash64-family, 32 for SHA-256).
• replica_idx (int) – DP replica whose device buffers source the offloaded blocks. The external tier itself is replica-agnostic.

Returns:

A KVConnectorTransfer for the D2H copy; g0_blocks_per_leaf are the device source blocks the manager keeps pinned until it lands. An offload that moved nothing returns a CompletedTransfer and is pinned nowhere.

Return type:

KVConnectorTransfer