For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python module

max.pipelines.architectures.flint

FlintForCausalLM architecture.

FlintConfig

class max.pipelines.architectures.flint.FlintConfig(*, hidden_size, num_attention_heads, num_key_value_heads, num_hidden_layers, rope_theta, max_seq_len, intermediate_size, vocab_size, dtype, kv_params, head_dim, devices, layer_types=<factory>, sliding_window, num_experts, num_experts_per_tok, moe_intermediate_size, norm_topk_prob, hidden_act, attention_bias, mlp_only_layers=<factory>, decoder_sparse_step=1, shared_expert_intermediate_size=1024, return_logits=ReturnLogits.LAST_TOKEN, rms_norm_eps=None, tie_word_embeddings=False, return_hidden_states=ReturnHiddenStates.NONE, quantization_encoding=None)

source

Bases: ArchConfigWithStoredKVParams, ArchConfigWithKVCache

Configuration for Flint.

Parameters:

• hidden_size (int)
• num_attention_heads (int)
• num_key_value_heads (int)
• num_hidden_layers (int)
• rope_theta (float)
• max_seq_len (int)
• intermediate_size (int)
• vocab_size (int)
• dtype (DType)
• kv_params (MultiKVCacheParams)
• head_dim (int)
• devices (list[DeviceRef])
• layer_types (list[str])
• sliding_window (int)
• num_experts (int)
• num_experts_per_tok (int)
• moe_intermediate_size (int)
• norm_topk_prob (bool)
• hidden_act (str)
• attention_bias (bool)
• mlp_only_layers (list[int])
• decoder_sparse_step (int)
• shared_expert_intermediate_size (int)
• return_logits (ReturnLogits)
• rms_norm_eps (float | None)
• tie_word_embeddings (bool)
• return_hidden_states (ReturnHiddenStates)
• quantization_encoding (SupportedEncoding | None)

DEFAULT_ENCODING

DEFAULT_ENCODING: ClassVar[SupportedEncoding] = 'bfloat16'

source

SUPPORTED_ENCODINGS

SUPPORTED_ENCODINGS: ClassVar[set[SupportedEncoding]] = {'bfloat16', 'float32'}

source

attention_bias

attention_bias: bool

source

construct_kv_params()

classmethod construct_kv_params(huggingface_config, pipeline_config, devices, kv_cache_config, cache_dtype, *, allow_kv_head_replication=False)

source

Builds one paged cache per attention type.

The sliding child is bounded by the sliding window; the full child is unbounded.

Parameters:

• huggingface_config (AutoConfig)
• pipeline_config (PipelineConfig)
• devices (list[DeviceRef])
• kv_cache_config (KVCacheConfig)
• cache_dtype (DType)
• allow_kv_head_replication (bool)

Return type:

MultiKVCacheParams

decoder_sparse_step

decoder_sparse_step: int = 1

source

devices

devices: list[DeviceRef]

source

dtype

dtype: DType

source

finalize()

finalize(huggingface_config, state_dict, return_logits, return_hidden_states=ReturnHiddenStates.NONE)

source

Resolves the fields that depend on the loaded weights.

Parameters:

• huggingface_config (AutoConfig)
• state_dict (dict[str, WeightData])
• return_logits (ReturnLogits)
• return_hidden_states (ReturnHiddenStates)

Return type:

None

get_head_dim()

static get_head_dim(huggingface_config)

source

Attention head size from head_dim or hidden_size // num_attention_heads.

Parameters:

huggingface_config (AutoConfig)

Return type:

int

get_num_layers()

static get_num_layers(huggingface_config)

source

Layer count for the decoder stack (override when HF uses a different field).

Parameters:

huggingface_config (AutoConfig)

Return type:

int

get_rope_theta()

static get_rope_theta(huggingface_config)

source

Returns the rope theta stored in the HF config.

transformers 5.x may nest the value under rope_parameters, so read that first and fall back to the top-level key.

Parameters:

huggingface_config (AutoConfig)

Return type:

float

head_dim

head_dim: int

source

hidden_act

hidden_act: str

source

hidden_size

hidden_size: int

source

initialize()

classmethod initialize(pipeline_config, model_config=None, *, max_seq_len)

source

Initialize the config from a PipelineConfig.

Parameters:

• pipeline_config (PipelineConfig) – The pipeline configuration.
• model_config (MAXModelConfig | None) – The model configuration to read from. When None (the default), pipeline_config.model is used. Pass an explicit config (such as pipeline_config.draft_model) to initialize the arch config for a different model.
• max_seq_len (int) – The effective maximum sequence length to store on the config. The value is received, never derived here: the pipeline model passes the memory plan’s VRAM-clamped length, while memory planning (which runs before a plan exists) passes the construction-resolved model_config.max_length. Configs whose sequence length is pure model metadata (such as diffusion components) ignore it.

Return type:

Self

intermediate_size

intermediate_size: int

source

kv_params

kv_params: MultiKVCacheParams

source

layer_types

layer_types: list[str]

source

max_seq_len

max_seq_len: int

source

mlp_only_layers

mlp_only_layers: list[int]

source

moe_intermediate_size

moe_intermediate_size: int

source

norm_topk_prob

norm_topk_prob: bool

source

num_attention_heads

num_attention_heads: int

source

num_experts

num_experts: int

source

num_experts_per_tok

num_experts_per_tok: int

source

num_hidden_layers

num_hidden_layers: int

source

num_key_value_heads

num_key_value_heads: int

source

quantization_encoding

quantization_encoding: SupportedEncoding | None = None

source

return_hidden_states

return_hidden_states: ReturnHiddenStates = 'none'

source

return_logits

return_logits: ReturnLogits = 'last_token'

source

rms_norm_eps

rms_norm_eps: float | None = None

source

rope_theta

rope_theta: float

source

shared_expert_intermediate_size

shared_expert_intermediate_size: int = 1024

source

sliding_window

sliding_window: int

source

tie_word_embeddings

tie_word_embeddings: bool = False

source

vocab_size

vocab_size: int

source

FlintInputs

class max.pipelines.architectures.flint.FlintInputs(tokens: 'Buffer', input_row_offsets: 'Buffer', return_n_logits: 'Buffer', *, kv_cache_inputs: 'KVCacheInputs[Buffer, Buffer] | None' = None, lora_buffers: 'tuple[Buffer, ...]' = (), vision_embeddings: 'list[Buffer]' = <factory>, vision_scatter_indices: 'list[Buffer]' = <factory>, hidden_states: 'Buffer | list[Buffer] | None' = None)

source

Bases: ModelInputs

Parameters:

• tokens (Buffer)
• input_row_offsets (Buffer)
• return_n_logits (Buffer)
• kv_cache_inputs (KVCacheInputs[Buffer, Buffer] | None)
• lora_buffers (tuple[Buffer, ...])
• vision_embeddings (list[Buffer])
• vision_scatter_indices (list[Buffer])
• hidden_states (Buffer | list[Buffer] | None)

buffers

property buffers: tuple[Buffer, ...]

source

Returns positional Buffer inputs for model ABI calls.

input_row_offsets

input_row_offsets: Buffer

source

return_n_logits

return_n_logits: Buffer

source

tokens

tokens: Buffer

source

FlintModel

class max.pipelines.architectures.flint.FlintModel(pipeline_config, session, *args, **kwargs)

source

Bases: LogProbabilitiesMixin, ModuleV3PipelineModelWithKVCache[TextContext, Any]

Builds and compiles the ModuleV3 network for Flint.

Parameters:

• pipeline_config (Any)
• session (InferenceSession)
• args (Any)
• kwargs (Any)

batch_processor_cls

batch_processor_cls

source

alias of FlintBatchProcessor

config_class

config_class

source

alias of FlintConfig

model_config_cls

model_config_cls

source

alias of FlintConfig