For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python function
compute_max_seq_len_fitting_in_cache
compute_max_seq_len_fitting_in_cache()
max.nn.kv_cache.compute_max_seq_len_fitting_in_cache(params, available_cache_memory, include_null_block=False)
Computes the maximum sequence length that can fit in the available memory.
-
Parameters:
-
- available_cache_memory (int) – The amount of cache memory available across
- devices. (all)
- include_null_block (bool) – Whether to include room for the null block.
- params (KVCacheParamInterface)
-
Returns:
-
The maximum sequence length that can fit in the available cache memory.
-
Return type: