For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python class
KVCacheQuantizationConfig
KVCacheQuantizationConfigโ
class max.nn.kv_cache.KVCacheQuantizationConfig(scale_dtype=float32, quantization_granularity=128)
Bases: object
Configuration for KVCache quantization.
Supports FP8 and int8 values with per-block scales, and NVFP4: uint8
storage packing two E2M1 values per byte with E4M3 scales.
quantization_granularityโ
quantization_granularity: int = 128
Block-size used for KVCache quantization along head-dimension, such as 128.
Counted in logical elements, so a packed NVFP4 cache uses 16.
scale_dtypeโ
scale_dtype: DType = 81
Data type of quantization scales, if quantization is enabled