For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

KVCacheQuantizationConfig

`KVCacheQuantizationConfig`

class max.nn.kv_cache.KVCacheQuantizationConfig(scale_dtype=float32, quantization_granularity=128, fp8_tail_dim=0)

source

Configuration for KVCache quantization.

Supports FP8 and int8 values with per-block scales, and NVFP4: uint8 storage packing two E2M1 values per byte with E4M3 scales.

Parameters:

fp8_tail_dim: int = 0

source

Trailing elements of each head row that a packed NVFP4 cache stores as raw FP8 bytes instead of E2M1 pairs.

They carry no scales. An MLA latent cache keeps its rope dims here, which dominate the attention scores and lose accuracy in E2M1.

quantization_granularity: int = 128

source

Block-size used for KVCache quantization along head-dimension, such as 128.

Counted in logical elements, so a packed NVFP4 cache uses 16.

scale_dtype: DType = 81

source

Data type of quantization scales, if quantization is enabled