For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

MultiKVCacheBuffer

MultiKVCacheBuffer

class max.nn.kv_cache.MultiKVCacheBuffer(children)

source

Bases: KVCacheBufferInterface

A tree of KVCache buffers for one data-parallel replica.

children maps a cache name (such as "target"/"draft" for speculative decoding, or "sliding"/"global" for hybrid models) to that cache’s buffer for this replica.

Parameters:

children (dict[str, KVCacheBufferInterface])

all_buffers

property all_buffers: list[Buffer]

source

Returns all buffers across every child cache.

children

children: dict[str, KVCacheBufferInterface]

source

to_memory()

to_memory()

source

Returns the offload-ready memory units for all children.

Raises:

ValueError – If two children claim the same leaf.

Return type:

Mapping[str, KVCacheMemory]