For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python class
KVCacheMetrics
KVCacheMetrics
class max.nn.kv_cache.KVCacheMetrics(input_tokens=0, cache_tokens=0, device_blocks_served=0, h2d_bytes_copied=0, d2h_bytes_copied=0, cross_replica_blocks_copied=0, cross_replica_bytes_copied=0, disk_bytes_written=0, disk_bytes_read=0, inflight_disk_ops=0, connector_loads_refused=0, connector_load_failures=0, connector_offload_blocks_dropped=0, nixl_read_blocks=0, nixl_write_blocks=0, nixl_read_latency_total_ms=0.0, nixl_read_latency_count=0, nixl_read_latency_max_ms=0.0, nixl_write_latency_total_ms=0.0, nixl_write_latency_count=0, rpc_acquire_latency_total_ms=0.0, rpc_acquire_latency_count=0, rpc_read_latency_total_ms=0.0, rpc_read_latency_count=0, nixl_read_bytes=0, nixl_write_bytes=0, dkv_connected_clients=0, dkv_total_clients=0, dkv_reconnect_attempts=0, dkv_settle_queue_depth=0, dkv_settle_queue_oldest_age_ms=0.0, dkv_settle_jobs_finished=0, dkv_owner_releases_halted=0, dkv_peer_attaches=0, dkv_peer_attach_failures=0, dkv_peers_dropped=0, dkv_peer_loads=0, dkv_peer_load_failures=0, dkv_peer_transfer_failures=0, dkv_peer_metadata_refreshes=0, dkv_peer_metadata_refresh_failures=0, dkv_hints_rejected=0, dkv_hints_rejected_by_reason=<factory>, dkv_touches_deduplicated=0, dkv_prepare_load_degraded=0, dkv_load_with_plan_degraded=0, dkv_load_with_plan_expired=0, dkv_rpc_calls=<factory>, dkv_rpc_time_ms=<factory>, dkv_rpc_socket_wait_ms=<factory>, dkv_lock_waits=<factory>, dkv_lock_wait_ms=<factory>, dkv_blocked_calls=<factory>, dkv_blocked_ms=<factory>)
Bases: object
Metrics for the KV cache.
Tracks token usage and transfer statistics for KV cache operations.
Device (G0) figures are in blocks, the unit the manager allocates in. The connector’s external host and disk tiers are in bytes: those are byte budgets the operator sizes in bytes, their block width need not match the device’s, and bytes rate directly against PCIe and disk bandwidth.
-
Parameters:
-
- input_tokens (int)
- cache_tokens (int)
- device_blocks_served (int)
- h2d_bytes_copied (int)
- d2h_bytes_copied (int)
- cross_replica_blocks_copied (int)
- cross_replica_bytes_copied (int)
- disk_bytes_written (int)
- disk_bytes_read (int)
- inflight_disk_ops (int)
- connector_loads_refused (int)
- connector_load_failures (int)
- connector_offload_blocks_dropped (int)
- nixl_read_blocks (int)
- nixl_write_blocks (int)
- nixl_read_latency_total_ms (float)
- nixl_read_latency_count (int)
- nixl_read_latency_max_ms (float)
- nixl_write_latency_total_ms (float)
- nixl_write_latency_count (int)
- rpc_acquire_latency_total_ms (float)
- rpc_acquire_latency_count (int)
- rpc_read_latency_total_ms (float)
- rpc_read_latency_count (int)
- nixl_read_bytes (int)
- nixl_write_bytes (int)
- dkv_connected_clients (int)
- dkv_total_clients (int)
- dkv_reconnect_attempts (int)
- dkv_settle_queue_depth (int)
- dkv_settle_queue_oldest_age_ms (float)
- dkv_settle_jobs_finished (int)
- dkv_owner_releases_halted (int)
- dkv_peer_attaches (int)
- dkv_peer_attach_failures (int)
- dkv_peers_dropped (int)
- dkv_peer_loads (int)
- dkv_peer_load_failures (int)
- dkv_peer_transfer_failures (int)
- dkv_peer_metadata_refreshes (int)
- dkv_peer_metadata_refresh_failures (int)
- dkv_hints_rejected (int)
- dkv_hints_rejected_by_reason (dict[str, int])
- dkv_touches_deduplicated (int)
- dkv_prepare_load_degraded (int)
- dkv_load_with_plan_degraded (int)
- dkv_load_with_plan_expired (int)
- dkv_rpc_calls (dict[str, int])
- dkv_rpc_time_ms (dict[str, float])
- dkv_rpc_socket_wait_ms (dict[str, float])
- dkv_lock_waits (dict[str, int])
- dkv_lock_wait_ms (dict[str, float])
- dkv_blocked_calls (dict[str, int])
- dkv_blocked_ms (dict[str, float])
cache_hit_rate
property cache_hit_rate: float
Proportion of prompt tokens that were retrieved from cache.
-
Returns:
-
Ratio of cache_tokens to total prompt_tokens, or 0.0 if no tokens were processed.
cache_tokens
cache_tokens: int = 0
Number of tokens retrieved from cache (cache hits).
connector_load_failures
connector_load_failures: int = 0
Number of connector loads whose copy failed.
Counted by the cache manager, which serves each one by recomputing the request’s prefix. A transport fault rather than a miss, so any of these is worth chasing.
connector_loads_refused
connector_loads_refused: int = 0
Number of loads the connector refused: its host and disk tiers, or dKV.
A refused load is served by recomputing its blocks, so a steady rate of these is hit rate the tiers reported and then could not deliver.
connector_offload_blocks_dropped
connector_offload_blocks_dropped: int = 0
Number of offloaded blocks the connector’s host pool had no room for.
Counted per leaf block. A pool starved by blocks pinned for in-flight transfers drops offloads before its hit rate falls.
cross_replica_blocks_copied
cross_replica_blocks_copied: int = 0
Number of cache blocks copied device-to-device across DP replicas.
cross_replica_bytes_copied
cross_replica_bytes_copied: int = 0
Bytes moved by device-to-device copies across DP replicas.
d2h_bytes_copied
d2h_bytes_copied: int = 0
Bytes of KV copied from device to the connector’s host tier.
device_blocks_served
device_blocks_served: int = 0
Number of cache blocks served from the device tier, with no host or disk promotion. Jenga counts a block copied in from a peer replica; the legacy cache counts only blocks the replica already held.
disk_bytes_read
disk_bytes_read: int = 0
Bytes of KV read from disk.
disk_bytes_written
disk_bytes_written: int = 0
Bytes of KV written to disk.
dkv_blocked_calls
Calls into each connector entry point.
dkv_blocked_ms
Milliseconds the calling thread spent inside each entry point, lock wait and any RPC included. Near zero for an entry that returns at once, which is how the scheduler thread is shown to have stopped waiting.
dkv_connected_clients
dkv_connected_clients: int = 0
Number of dKV data-parallel replicas fully connected to the external tier.
Per REPLICA, not per client. A replica owns one connector client per KV
cache leaf, and counts as connected only when EVERY one of them is up,
because one dead leaf cannot serve a block. A four-leaf replica with one
leaf down reports 0 here against 1 in dkv_total_clients.
dkv_degraded
property dkv_degraded: bool
Whether a dKV tier is present but not every client is connected.
Delegates to the module-level dkv_tier_degraded so this predicate has a single definition shared with the scheduler’s per-batch log.
dkv_hints_rejected
dkv_hints_rejected: int = 0
Cache hints that were present but the connector could not use.
dkv_hints_rejected_by_reason
dkv_hints_rejected split by why each hint went unused.
dkv_load_with_plan_degraded
dkv_load_with_plan_degraded: int = 0
Loads whose connector degraded between the lookup and the post.
dkv_load_with_plan_expired
dkv_load_with_plan_expired: int = 0
Leases that aged past their deadline before the load posted them.
dkv_lock_wait_ms
Milliseconds each entry point waited for the connector lock.
dkv_lock_waits
Times each connector entry point took the connector lock.
dkv_owner_releases_halted
dkv_owner_releases_halted: int = 0
1 while any leaf client’s owner release halt is up, else 0.
While it is up, the owner’s failing cleanups wait on the orphan queue and its drain backs off.
dkv_peer_attach_failures
dkv_peer_attach_failures: int = 0
Peer attaches that failed at dial, probe, handshake, or timeout.
dkv_peer_attaches
dkv_peer_attaches: int = 0
Remote peers attached from a cache hint.
Cached per peer, so a stable peer set leaves this at zero while pulls keep
succeeding. dkv_peer_loads is the steady-state signal.
dkv_peer_load_failures
dkv_peer_load_failures: int = 0
Hinted peer loads that fell through to another source or to the co-located tier.
Memo-capped rather than per-request: a failed source suppresses further attempts against the same instance and epoch for a fixed window, so one dead peer charges roughly one failure per window however many requests it affects.
dkv_peer_loads
dkv_peer_loads: int = 0
Loads served from a hint-routed peer.
dkv_peer_metadata_refresh_failures
dkv_peer_metadata_refresh_failures: int = 0
In-place peer metadata reloads that NIXL refused.
Each strands its peer: the connector never routes to or dials it again for the life of the process.
dkv_peer_metadata_refreshes
dkv_peer_metadata_refreshes: int = 0
Stale peers whose NIXL metadata the connector reloaded in place.
dkv_peer_transfer_failures
dkv_peer_transfer_failures: int = 0
Hint-routed peer leases whose NIXL read failed to build or post.
Charged to the peer, which is skipped for the failure memo window and marked stale for a metadata refresh, rather than degrading the connector.
dkv_peers_dropped
dkv_peers_dropped: int = 0
Remote peers dropped, whether replaced after a restart, evicted over the
peer table’s cap, or torn down by the caller. Read against
dkv_peer_attaches to tell a churning peer table from a stable one.
dkv_prepare_load_degraded
dkv_prepare_load_degraded: int = 0
Lookups that found the connector degraded and planned nothing.
dkv_reconnect_attempts
dkv_reconnect_attempts: int = 0
Cumulative dKV reconnect attempts per replica, summed over replicas.
Per REPLICA, not per client: a replica owns one client per KV cache leaf and
they share a server and a network path, so they lose and recover it
together. The connector reports the max over a replica’s leaf clients, which
keeps this on the same denominator as dkv_total_clients – summing over
leaves would multiply a four-leaf deployment’s reading by four.
dkv_rpc_calls
dKV RPCs issued, by operation.
dkv_rpc_socket_wait_ms
Milliseconds those RPCs queued for their client’s one REQ socket.
dkv_rpc_time_ms
Wall time of those RPCs in milliseconds, socket wait included.
dkv_settle_jobs_finished
dkv_settle_jobs_finished: int = 0
Settle jobs the workers finished or dropped this window, summed.
dkv_settle_queue_depth
dkv_settle_queue_depth: int = 0
Settle jobs queued and not yet finished, summed over every leaf client.
dkv_settle_queue_oldest_age_ms
dkv_settle_queue_oldest_age_ms: float = 0.0
How long the oldest unfinished settle job has waited, in milliseconds.
The max over every leaf client, and zero when none is queued. Read it
against dkv_settle_jobs_finished. A head stuck on an RPC or a reconnect
ages at wall-clock rate while that count stays flat, and a worker that is
only behind under load ages more slowly while the count keeps rising.
dkv_total_clients
dkv_total_clients: int = 0
Number of dKV data-parallel replicas, regardless of connection state.
The denominator for dkv_connected_clients. Named “clients” from when a
replica owned exactly one; it counts replicas, not the per-leaf clients a
replica now holds.
dkv_touches_deduplicated
dkv_touches_deduplicated: int = 0
Recency touches the client dropped because a touch it sent within the last second already covered their keys, each an RPC the store was spared.
h2d_bytes_copied
h2d_bytes_copied: int = 0
Bytes of KV copied from the connector’s host tier to device.
inflight_disk_ops
inflight_disk_ops: int = 0
Number of in-flight disk operations.
input_tokens
input_tokens: int = 0
Number of tokens processed as new input (cache misses).
nixl_read_blocks
nixl_read_blocks: int = 0
Number of cache blocks read via NIXL (dKV GET).
nixl_read_bytes
nixl_read_bytes: int = 0
Total bytes transferred via NIXL READ.
nixl_read_gib_per_s
property nixl_read_gib_per_s: float
NIXL READ throughput in GiB/s.
nixl_read_latency_avg_ms
property nixl_read_latency_avg_ms: float
Average NIXL READ transfer latency in milliseconds.
nixl_read_latency_count
nixl_read_latency_count: int = 0
Number of NIXL READ transfer completions.
nixl_read_latency_max_ms
nixl_read_latency_max_ms: float = 0.0
Longest single dKV READ transfer in milliseconds.
The tail the total/count pair above cannot show, since that pair only ever
yields a mean. Combined across replicas by taking the max, not by adding.
The connector’s ConnectorMetrics::read_transfer_latency_max_ms is the
canonical account of what the figure does and does not mean.
nixl_read_latency_total_ms
nixl_read_latency_total_ms: float = 0.0
Cumulative NIXL READ transfer latency in milliseconds.
nixl_write_blocks
nixl_write_blocks: int = 0
Number of cache blocks written via NIXL (dKV PUT).
nixl_write_bytes
nixl_write_bytes: int = 0
Total bytes transferred via NIXL WRITE.
nixl_write_gib_per_s
property nixl_write_gib_per_s: float
NIXL WRITE throughput in GiB/s.
nixl_write_latency_avg_ms
property nixl_write_latency_avg_ms: float
Average NIXL WRITE transfer latency in milliseconds.
nixl_write_latency_count
nixl_write_latency_count: int = 0
Number of NIXL WRITE transfer completions.
nixl_write_latency_total_ms
nixl_write_latency_total_ms: float = 0.0
Cumulative NIXL WRITE transfer latency in milliseconds.
prompt_tokens
property prompt_tokens: int
Total number of prompt tokens (input + cached).
-
Returns:
-
Sum of input_tokens and cache_tokens.
rpc_acquire_latency_avg_ms
property rpc_acquire_latency_avg_ms: float
Average dKV acquire_blocks RPC latency in milliseconds.
rpc_acquire_latency_count
rpc_acquire_latency_count: int = 0
Number of acquire_blocks RPC calls.
rpc_acquire_latency_total_ms
rpc_acquire_latency_total_ms: float = 0.0
Cumulative dKV acquire_blocks RPC latency in milliseconds.
rpc_read_latency_avg_ms
property rpc_read_latency_avg_ms: float
Average dKV read_blocks RPC latency in milliseconds.
rpc_read_latency_count
rpc_read_latency_count: int = 0
Number of read_blocks RPC calls.
rpc_read_latency_total_ms
rpc_read_latency_total_ms: float = 0.0
Cumulative dKV read_blocks RPC latency in milliseconds.