For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

ArchConfigWithAttentionKVCache

ArchConfigWithAttentionKVCache

class max.pipelines.lib.interfaces.ArchConfigWithAttentionKVCache(dtype, max_seq_len, devices=<factory>, cache_dtype=None, quantization_encoding=None, kv_cache=<factory>, data_parallel_degree=1, huggingface_config=None, _kv_params=None)

source

Bases: ArchConfigWithKVCache, ABC

Predefined configuration for architectures that use attention KV cache blocks.

Subclasses must define the following attributes:

• num_key_value_heads: int
• head_dim: int
• num_layers: int
• DEFAULT_ENCODING: SupportedEncoding

Parameters:

• dtype (DType)
• max_seq_len (int)
• devices (list[DeviceRef])
• cache_dtype (DType | None)
• quantization_encoding (SupportedEncoding | None)
• kv_cache (KVCacheConfig)
• data_parallel_degree (int)
• huggingface_config (AutoConfig | None)
• _kv_params (KVCacheParams | None)

DEFAULT_ENCODING

DEFAULT_ENCODING: ClassVar[SupportedEncoding]

source

SUPPORTED_ENCODINGS

SUPPORTED_ENCODINGS: ClassVar[set[SupportedEncoding]]

source

cache_dtype

cache_dtype: DType | None = None

source

The data type to use for the KV cache.

data_parallel_degree

data_parallel_degree: int = 1

source

The data parallel degree to use when running the model.

devices

devices: list[DeviceRef]

source

The physical devices to use when running the model.

dtype

dtype: DType

source

The data type to use for the model.

get_kv_params()

get_kv_params()

source

Returns the KV cache parameters for this architecture.

Return type:

KVCacheParams

get_max_seq_len()

get_max_seq_len()

source

Returns the maximum sequence length received at initialization.

Return type:

int

head_dim

abstract property head_dim: int

source

Dimensionality of each attention head.

huggingface_config

huggingface_config: AutoConfig | None = None

source

initialize()

classmethod initialize(pipeline_config, model_config=None, *, max_seq_len)

source

Initialize the config from a PipelineConfig.

Parameters:

• pipeline_config (PipelineConfig) – The pipeline configuration.
• model_config (MAXModelConfig | None) – The model configuration to read from. When None (the default), pipeline_config.model is used. Pass an explicit config (e.g. pipeline_config.draft_model) to initialize the arch config for a different model.
• max_seq_len (int) – The effective maximum sequence length to store on the config. The value is received, never derived here: the pipeline model passes the memory plan’s VRAM-clamped length, while memory planning (which runs before a plan exists) passes the construction-resolved model_config.max_length. Configs whose sequence length is pure model metadata (e.g. diffusion components) ignore it.

Return type:

Self

kv_cache

kv_cache: KVCacheConfig

source

The KV cache configuration to use when running the model.

max_seq_len

max_seq_len: int

source

The effective maximum sequence length, received at initialization.

num_key_value_heads

abstract property num_key_value_heads: int

source

Number of key-value heads to use for the KV cache.

num_layers

abstract property num_layers: int

source

Number of hidden layers in the model.

quantization_encoding

quantization_encoding: SupportedEncoding | None = None

source

The resolved weight encoding the model runs with.