For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python function

kv_cache_memory_size

kv_cache_memory_size()

max.pipelines.kv_cache.kv_cache_memory_size(params, available_cache_memory, max_batch_size, max_seq_len, is_di_enabled, model_name)

source

Returns the KV bytes the manager for this cache needs for a full batch.

Jenga carves every leaf, states included, out of whole huge blocks, so it sizes its own slab. The legacy pool charges every leaf a page per slot.

Returns:

The bytes across all devices, at most available_cache_memory.

Parameters:

• params (KVCacheParamInterface)
• available_cache_memory (int)
• max_batch_size (int)
• max_seq_len (int)
• is_di_enabled (bool)
• model_name (str)

Return type:

int