For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python module

max.pipelines.architectures.idefics3

Idefics3 vision-language architecture for multimodal text generation.

Idefics3Config

class max.pipelines.architectures.idefics3.Idefics3Config(*, devices, scale_factor, image_token_id, vision_config, text_config, quantization_encoding=None)

source

Bases: ArchVLConfigWithTextSubconfig, ArchConfigWithKVCache

Configuration for Idefics3 models.

Parameters:

• devices (list[DeviceRef])
• scale_factor (int)
• image_token_id (int)
• vision_config (Idefics3VisionConfig)
• text_config (Llama3Config)
• quantization_encoding (SupportedEncoding | None)

DEFAULT_ENCODING

DEFAULT_ENCODING: ClassVar[SupportedEncoding] = 'bfloat16'

source

SUPPORTED_ENCODINGS

SUPPORTED_ENCODINGS: ClassVar[set[SupportedEncoding]] = {'bfloat16'}

source

devices

devices: list[DeviceRef]

source

Devices that the Idefics3 model is parallelized over.

finalize()

finalize(huggingface_config, llm_state_dict, return_logits, norm_method='rms_norm')

source

Finalize the Idefics3Config instance with state_dict dependent fields.

Parameters:

• huggingface_config (AutoConfig) – HuggingFace model configuration.
• llm_state_dict (dict[str, WeightData]) – Language model weights dictionary.
• dtype – Data type for model parameters.
• return_logits (ReturnLogits) – Return logits configuration.
• norm_method (Literal['rms_norm', 'layer_norm']) – Normalization method.

Return type:

None

get_kv_params()

get_kv_params()

source

Returns the KV cache parameters from the embedded text config.

Return type:

KVCacheParams

get_num_layers()

static get_num_layers(huggingface_config)

source

Get number of layers in the language model.

Parameters:

huggingface_config (AutoConfig)

Return type:

int

image_seq_len

property image_seq_len: int

source

Calculate the number of image tokens after connector processing.

image_token_id

image_token_id: int

source

Token ID used to represent image tokens in the text sequence.

initialize()

classmethod initialize(pipeline_config, model_config=None)

source

Initializes an Idefics3Config instance from pipeline configuration.

Parameters:

• pipeline_config (PipelineConfig) – The MAX Engine pipeline configuration.
• model_config (MAXModelConfig | None)

Returns:

An Idefics3Config instance with fields initialized from config.

Return type:

Self

quantization_encoding

quantization_encoding: SupportedEncoding | None = None

source

scale_factor

scale_factor: int

source

Scale factor for pixel shuffle operation in the connector.

text_config

text_config: Llama3Config

source

Text model configuration (Llama3-based).

vision_config

vision_config: Idefics3VisionConfig

source

Vision encoder configuration (SigLIP-based).

Idefics3Inputs

class max.pipelines.architectures.idefics3.Idefics3Inputs(tokens, input_row_offsets, return_n_logits, pixel_values=None, image_token_indices=None, *, kv_cache_inputs=None, lora=None, lora_buffers=(), vision_embeddings=<factory>, vision_scatter_indices=<factory>, hidden_states=None)

source

Bases: ModelInputs

A class representing inputs for the Idefics3 model.

Parameters:

• tokens (Buffer)
• input_row_offsets (Buffer)
• return_n_logits (Buffer)
• pixel_values (Buffer | None)
• image_token_indices (Buffer | None)
• kv_cache_inputs (KVCacheInputsInterface[Buffer, Buffer] | None)
• lora (LoRAInputs | None)
• lora_buffers (tuple[Buffer, ...])
• vision_embeddings (list[Buffer])
• vision_scatter_indices (list[Buffer])
• hidden_states (Buffer | list[Buffer] | None)

has_vision_inputs

property has_vision_inputs: bool

source

Check if this input contains vision data.

image_token_indices

image_token_indices: Buffer | None = None

source

Pre-computed indices of image tokens in the input sequence.

input_row_offsets

input_row_offsets: Buffer

source

Tensor containing the offsets for each row in the ragged input sequence.

pixel_values

pixel_values: Buffer | None = None

source

Pixel values for vision inputs.

return_n_logits

return_n_logits: Buffer

source

Number of logits to return, used by speculative decoding for example.

tokens

tokens: Buffer

source

Tensor containing the input token IDs.

Idefics3Model

class max.pipelines.architectures.idefics3.Idefics3Model(pipeline_config, session, devices, kv_cache_config, weights, adapter=None, return_logits=ReturnLogits.LAST_TOKEN, max_batch_size=1)

source

Bases: MultiGraphPipelineModelWithKVCache[TextAndVisionContext]

An Idefics3 pipeline model for multimodal text generation.

Parameters:

• pipeline_config (PipelineConfig)
• session (InferenceSession)
• devices (list[Device])
• kv_cache_config (KVCacheConfig)
• weights (Weights)
• adapter (WeightsAdapter | None)
• return_logits (ReturnLogits)
• max_batch_size (int)

batch_processor_cls

batch_processor_cls

source

alias of Idefics3BatchProcessor

calculate_max_seq_len()

classmethod calculate_max_seq_len(pipeline_config, huggingface_config)

source

Uses max_length when set, else text_config.max_position_embeddings (config bounds).

Parameters:

• pipeline_config (PipelineConfig)
• huggingface_config (AutoConfig)

Return type:

int

execute()

execute(model_inputs)

source

Executes the Idefics3 model with the prepared inputs.

Parameters:

model_inputs (ModelInputs)

Return type:

ModelOutputs

language_model

language_model: Model

source

The compiled language model for text generation.

model_config_cls

model_config_cls

source

alias of Idefics3Config

vision_model

vision_model: Model | None

source

The compiled vision model for processing images.

Idefics3VisionConfig

class max.pipelines.architectures.idefics3.Idefics3VisionConfig(dtype, hidden_size, intermediate_size, image_size, patch_size, num_channels, num_attention_heads, head_dim, layer_norm_eps, hidden_act, num_hidden_layers, initializer_range, scale_factor, text_config_hidden_size)

source

Bases: object

Configuration for Idefics3 Vision Model (SigLIP-based).

Parameters:

• dtype (DType)
• hidden_size (int)
• intermediate_size (int)
• image_size (int)
• patch_size (int)
• num_channels (int)
• num_attention_heads (int)
• head_dim (int)
• layer_norm_eps (float)
• hidden_act (str)
• num_hidden_layers (int)
• initializer_range (float)
• scale_factor (int)
• text_config_hidden_size (int)

dtype

dtype: DType

source

DType of the Idefics3 vision model weights.

head_dim

head_dim: int

source

Dimension of each attention head.

hidden_act

hidden_act: str

source

Activation function used in the vision encoder.

hidden_size

hidden_size: int

source

Hidden size of the vision encoder.

image_size

image_size: int

source

Input image size.

initialize_from_config()

classmethod initialize_from_config(pipeline_config, huggingface_config, text_config_hidden_size)

source

Initialize Idefics3VisionConfig from HuggingFace config.

Parameters:

• pipeline_config (PipelineConfig)
• huggingface_config (AutoConfig)
• text_config_hidden_size (int)

Return type:

Idefics3VisionConfig

initializer_range

initializer_range: float

source

Standard deviation for weight initialization.

intermediate_size

intermediate_size: int

source

Intermediate size in the vision encoder’s feed-forward layers.

layer_norm_eps

layer_norm_eps: float

source

Epsilon for layer normalization.

num_attention_heads

num_attention_heads: int

source

Number of attention heads in the vision encoder.

num_channels

num_channels: int

source

Number of input channels (typically 3 for RGB).

num_hidden_layers

num_hidden_layers: int

source

Number of hidden layers in the vision encoder.

patch_size

patch_size: int

source

Vision transformer patch size.

scale_factor

scale_factor: int

source

Scale factor for pixel shuffle operation in the connector.

text_config_hidden_size

text_config_hidden_size: int

source

Hidden size from the text config for modality projection.