IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.

Skip to main content

For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

KVCacheInputs

`KVCacheInputs`

max.nn.kv_cache.KVCacheInputs

source

The KV-cache inputs one model step consumes: a tree with one entry per device, holding either that device’s paged-cache inputs or its recurrent-state leaves.

alias of KVCacheInputsPerDevice[_Tensor, _Buffer] | RecurrentStateInputsPerDevice[_Tensor, _Buffer] | list[Tree[_T_co]] | tuple[Tree[_T_co], …] | dict[Any, Tree[_T_co]] | OrderedDict[Any, Tree[_T_co]] | defaultdict[Any, Tree[_T_co]] | _NamedTuple[Tree[_T_co]] | _Flattenable[KVCacheInputsPerDevice[_Tensor, _Buffer] | RecurrentStateInputsPerDevice[_Tensor, _Buffer]]

KVCacheInputs