For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python function
compute_max_seq_len_fitting_in_cache
compute_max_seq_len_fitting_in_cache()
max.nn.kv_cache.compute_max_seq_len_fitting_in_cache(params, available_cache_memory, include_null_block=False)
Computes the maximum sequence length that can fit in the available memory.
-
Parameters:
-
- available_cache_memory (int) – The amount of cache memory available across all devices.
- include_null_block (bool) – Whether to include room for the null block. Reserves one page of the reported capacity for it.
- params (KVCacheParamInterface)
-
Returns:
-
The maximum sequence length that fits, or None where no length exhausts the cache. A recurrent state costs the same however long a request runs, which is what the manager reports once it exists.
-
Return type:
-
int | None