For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

BatchCharacteristics

BatchCharacteristics

class max.nn.kv_cache.BatchCharacteristics(batch_size, max_prompt_length, max_cache_valid_length)

source

Bases: object

Upper-bound batch shape used to prepare decode attention metadata.

Captures the (batch_size, max_prompt_length, max_cache_valid_length) a decode forward should prepare its attention dispatch metadata for, which may exceed the batch’s real per-request values.

runtime_inputs() uses it to resolve the dispatch key once: e.g. for graph-capture replay, max_cache_valid_length is aligned up to a cache length recorded during capture and every data-parallel replica must run the identical captured graph. The batch’s real values must not exceed these.

Parameters:

• batch_size (int)
• max_prompt_length (int)
• max_cache_valid_length (int)

batch_size

batch_size: int

source

Upper bound on requests in the decode batch.

max_cache_valid_length

max_cache_valid_length: int

source

Upper bound on the batch’s valid cache length.

max_prompt_length

max_prompt_length: int

source

Upper bound on the batch’s prompt length.