For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python module

max.nn.kv_cache

Key-value cache types for MAX layers.

Defines cache parameters, host-side buffers, and the input values that attention layers consume, for paged and recurrent caches alike.

Cache configuration

`CacheLeafParamInterface`	What every child of a cache tree contributes: leaves, inputs, cost.
`HostStagingAlloc`	How a tiered KV connector allocates its pinned host staging region.
`KVCacheAssignments`	Assignments of request blocks to KV cache pages for a replica.
`KVCacheBuffer`	A collection of KVCache buffers for one data-parallel replica.
`KVCacheBufferInterface`	Interface for a KV cache buffer (single leaf or a tree of leaves).
`KVCacheParamInterface`	A cache leaf a model reads through an attention op.
`KVCacheParams`	Configuration parameters for key-value cache management in transformer models.
`KVCacheGroupId`	Identifies the caches a model reuses and evicts together.
`MHAKVCacheParams`	KV cache parameters for multi-head attention (MHA).
`MLAKVCacheParams`	KV cache parameters for multi-latent attention (MLA).
`MSAKVCacheParams`	KV cache parameters for multi-step attention (MSA).
`KVCacheQuantizationConfig`	Configuration for KVCache quantization.
`KVConnectorType`	Identifies which off-device backing store the KV cache uses.
`KVConnectorConfigInterface`	The KV connector configuration contract: a type plus per-tier settings.
`NullKVConnectorConfig`	Connector config for no off-device backing store.
`KVCacheMemory`	One logical `(child, kind)` KV tensor as per-TP-shard `uint8` views.
`MultiKVCacheBuffer`	A tree of KVCache buffers for one data-parallel replica.
`MultiKVCacheParams`	Aggregates multiple cache parameter sets into a recursive tree.

`KVCacheInputs`	The KV-cache inputs one model step consumes: a tree with one entry per device, holding either that device's paged-cache inputs or its recurrent-state leaves.
`KVCacheInputsPerDevice`	Symbolic graph input types for a single device's paged KV cache.
`BatchCharacteristics`	Upper-bound batch shape used to prepare decode attention metadata.
`PagedCacheValues`	Paged KV cache inputs for one device, with cache blocks and lengths as graph values.

`CacheLeafKind`	What a child of a cache tree holds.
`KVLeafRegion`	One addressable region of the cache pool.
`PagedKVLeafRegion`	A leaf the graph reaches through a per-forward page table.
`RecurrentKVLeafRegion`	A leaf addressed by row: one block holds one request's whole state.
`RecurrentLeafInputs`	One state leaf's graph inputs on one device.
`RecurrentStateBuffer`	One replica's recurrent state pool, viewed as the pages it holds.
`RecurrentStateInputsPerDevice`	One device's recurrent-state leaves.
`RecurrentStateParams`	A cache leaf whose entry is a state rather than a span of tokens.
`RecurrentStateRegion`	Shape and dtype of one kind of recurrent state, for one pool leaf.
`ScratchRecurrentKVLeafRegion`	A row-addressed leaf drawn once per request and never published.

`AttnKey`	A resolved decode-attention dispatch shape.
`AttnKeyInterface`	Common base for resolved attention keys.
`MHAAttnKey`	Decode dispatch metadata for multi-head attention (MHA).
`MLAAttnKey`	Decode dispatch metadata for multi-latent attention (MLA).
`MSAAttnKey`	Decode dispatch metadata for multi-step attention (MSA).

`KVCacheMetrics`	Metrics for the KV cache.

`KVHashAlgo`	Supported hash algorithms for KV-cache block identity.
`PACKED_PAGE_STRIDE`	`page_stride` sentinel meaning the pages are packed.

`build_max_lengths_tensors`	Builds two `[1]` uint32 scalar buffers of maximum lengths.
`compute_max_seq_len_fitting_in_cache`	Computes the maximum sequence length that can fit in the available memory.
`compute_num_device_blocks`	Computes the number of blocks that can be allocated based on the available cache memory.
`estimated_memory_size`	Computes the estimated memory size of the KV cache used by all replicas.
`max_pages_per_request`	Returns the most KV pages one request can address.
`packed_page_stride`	Returns the packed `page_stride` sentinel, typed to match `like`.
`padded_lut_cols`	Rounds a page lookup-table inner dim up to a kernel-safe width.
`recurrent_leaf`	Returns the first state a cache keeps, or `None` if it keeps none.
`recurrent_leaves`	Returns every state a cache keeps, in tree order.
`spec_decode_cache_slack`	Computes the extra KV positions a request may occupy past `max_seq_len`.