IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python module

max.pipelines.architectures.qwen3_5

Qwen3_5Config​

class max.pipelines.architectures.qwen3_5.Qwen3_5Config(*, hidden_size, num_attention_heads, num_key_value_heads, num_hidden_layers, rope_theta, rope_scaling_params, max_seq_len, intermediate_size, interleaved_rope_weights, vocab_size, dtype, model_quantization_encoding, quantization_config, kv_params, return_logits=ReturnLogits.LAST_TOKEN, norm_method='rms_norm', norm_dtype=None, attention_bias=False, rms_norm_eps=None, tie_word_embeddings=False, stacked_mlp=False, stacked_qkv=False, attention_multiplier, embedding_multiplier, residual_multiplier, devices, clip_qkv, quant_config=None, longrope_scaling_params=None, logits_scaling=1.0, return_hidden_states=ReturnHiddenStates.NONE, target_layer_ids=None, use_subgraphs=True, data_parallel_degree=1, sliding_window=None, quantization_encoding=None, layer_types=<factory>, full_attention_interval=4, linear_key_head_dim=128, linear_value_head_dim=128, linear_num_key_heads=16, linear_num_value_heads=48, linear_conv_kernel_dim=4, partial_rotary_factor=0.25, attn_output_gate=True, mamba_ssm_dtype=float32, state_pool_dtype=None, vision_config=None, image_token_id=None, video_token_id=None, vision_start_token_id=None, mrope_section=None, hf_quantization_config=None, quant_scheme=None, declared_dtype=None)

source

Bases: Llama3Config, ArchConfigWithVisionCache

Configuration for Qwen3.5 hybrid attention models.

Qwen3.5 uses a hybrid architecture with both full (standard) attention and linear attention (Gated DeltaNet) layers. Every full_attention_interval-th layer uses full attention, and the rest use linear attention.

Parameters:

DEFAULT_ENCODING​

DEFAULT_ENCODING: ClassVar[SupportedEncoding] = 'bfloat16'

source

SUPPORTED_ENCODINGS​

SUPPORTED_ENCODINGS: ClassVar[set[SupportedEncoding]] = {'bfloat16', 'float32', 'float4_e2m1fnx2', 'float8_e4m3fn'}

source

attn_kv_params​

property attn_kv_params: KVCacheParams

source

The attention child of kv_params.

attn_output_gate​

attn_output_gate: bool = True

source

Whether full attention layers use a sigmoid output gate.

calculate_attention_multiplier()​

static calculate_attention_multiplier(huggingface_config)

source

Compute attention scaling factor using explicit head_dim.

Parameters:

huggingface_config (AutoConfig)

Return type:

float

calculate_max_seq_len()​

classmethod calculate_max_seq_len(huggingface_config, model_config)

source

Bounds against the text config’s max_position_embeddings.

Parameters:

Return type:

int

compute_dtype​

property compute_dtype: DType

source

Dtype of activations and every unquantized weight.

dtype is the storage dtype the resolved encoding implies (uint8 for packed NVFP4), which is not what the norms, embeddings, conv1d or linear-attention state pools use.

quant_scheme is authoritative but only exists after finalize; declared_dtype covers the earlier window so a pre-finalize caller does not silently read the storage dtype.

construct_kv_params()​

classmethod construct_kv_params(huggingface_config, pipeline_config, devices, kv_cache_config, cache_dtype, *, allow_kv_head_replication=False)

source

Returns the attention leaf beside the recurrent state.

The graph’s input types are built from these params, so a state derived any later cannot appear among them. A model with no linear-attention layers gets the attention leaf alone.

Parameters:

Return type:

KVCacheParamInterface

declared_dtype​

declared_dtype: DType | None = None

source

The dtype the checkpoint declares for its unquantized tensors.

Captured at initialize_from_config so compute_dtype is right before finalize resolves quant_scheme. Memory planning runs in that window.

declared_state_dtype()​

static declared_state_dtype(text_config, kv_cache_config)

source

Returns state_dtype as far as it is knowable pre-finalize.

The state leaves are declared before a quantization scheme resolves. cache_dtype is not a stand-in: --kv-cache-format may put the KV in fp8 while the state stays bf16.

Parameters:

Return type:

DType

estimate_vision_cache_entry_bytes()​

classmethod estimate_vision_cache_entry_bytes(huggingface_config)

source

Estimates per-entry bytes for the Qwen3.5 vision encoder cache.

Qwen3.5’s tower is NaViT-style: an image keeps its aspect ratio and yields one token per merged patch, so — unlike Gemma4’s fixed pooled patch count — no per-image token count exists in the config to read. The bound that does exist is the image processor’s post-resize pixel ceiling, which every served image is smart-resized under, so the largest entry the cache can be asked to hold is that ceiling divided by the patch area and the spatial merge.

Returns:

Estimated bytes per vision cache entry, or 0 for a checkpoint with no vision tower.

Parameters:

huggingface_config (AutoConfig)

Return type:

int

full_attention_interval​

full_attention_interval: int = 4

source

Every N-th layer uses full attention.

get_num_layers()​

static get_num_layers(huggingface_config)

source

Layer count for the decoder stack (override when HF uses a different field).

Parameters:

huggingface_config (AutoConfig)

Return type:

int

get_vision_cache_row_spec()​

classmethod get_vision_cache_row_spec(huggingface_config)

source

One embedding row per merged vision token, at the LM hidden size.

The dtype is the checkpoint’s declared one rather than a fixed bfloat16, because the block pool has to match the buffers the encoder hands it: those are cast to compute_dtype, which for every encoding this architecture supports resolves to the declared dtype.

Parameters:

huggingface_config (AutoConfig)

Return type:

tuple[int, DType] | None

hf_quantization_config​

hf_quantization_config: dict[str, Any] | None = None

source

The checkpoint’s resolved Hugging Face quantization config.

Captured at initialize_from_config because it lives on the top-level multimodal config, while finalize is handed text_config.

image_token_id​

image_token_id: int | None = None

source

Token ID used for image placeholders in the input sequence.

infer_optimal_batch_size()​

infer_optimal_batch_size(devices, *, weights_size, device_memory_utilization, extra_per_request_bytes=0)

source

Return a memory-safe default max_batch_size for this architecture.

The states get up to half the post-weights utilization budget and the KV absorbs the rest, under the same device_memory_utilization headroom factor as the rest of the pipeline. The halving bounds concurrency; it allocates nothing.

Falls back to 32—safe for the 27B model on H100/A100 (80 GB)—when the device query fails.

Parameters:

  • devices (list[Device]) – Loaded devices the model will run on.
  • weights_size (int) – Estimated model weights size in bytes.
  • device_memory_utilization (float) – Headroom factor.
  • extra_per_request_bytes (int) – Per-request state the architecture holds beyond the pool set this config declares – a speculative arch’s shadow set, for one. Added to the divisor so the inferred batch fits what will really be allocated.

Return type:

int

initialize()​

classmethod initialize(pipeline_config, model_config=None, *, max_seq_len)

source

Initialize the config from a PipelineConfig.

Parameters:

  • pipeline_config (PipelineConfig) – The pipeline configuration.
  • model_config (MAXModelConfig | None) – The model configuration to read from. When None (the default), pipeline_config.model is used. Pass an explicit config (e.g. pipeline_config.draft_model) to initialize the arch config for a different model.
  • max_seq_len (int) – The effective maximum sequence length to store on the config. The value is received, never derived here: the pipeline model passes the memory plan’s VRAM-clamped length, while memory planning (which runs before a plan exists) passes the construction-resolved model_config.max_length. Configs whose sequence length is pure model metadata (e.g. diffusion components) ignore it.

Return type:

Self

initialize_from_config()​

classmethod initialize_from_config(pipeline_config, huggingface_config, model_config=None, *, max_seq_len)

source

Initialize config from pipeline and HuggingFace configurations.

Handles both multimodal (Qwen3_5ForConditionalGeneration) and text-only (Qwen3_5ForCausalLM) configs by extracting the text config.

Parameters:

Return type:

Self

kv_params​

kv_params: KVCacheParamInterface

source

an attention child beside a recurrent-state one.

attn_kv_params is the leaf the attention layers take.

Type:

The cache tree

layer_types​

layer_types: list[str]

source

‘full_attention’ or ‘linear_attention’.

Type:

Per-layer attention type

linear_conv_kernel_dim​

linear_conv_kernel_dim: int = 4

source

Causal conv1d kernel size for linear attention layers.

linear_key_head_dim​

linear_key_head_dim: int = 128

source

Key head dimension for linear attention layers.

linear_num_key_heads​

linear_num_key_heads: int = 16

source

Number of key heads for linear attention layers.

linear_num_value_heads​

linear_num_value_heads: int = 48

source

Number of value heads for linear attention layers.

linear_value_head_dim​

linear_value_head_dim: int = 128

source

Value head dimension for linear attention layers.

mamba_ssm_dtype​

mamba_ssm_dtype: DType = 81

source

Dtype for SSM (state space model) computations in linear attention layers.

mrope_section​

mrope_section: list[int] | None = None

source

MRoPE section lengths for multimodal rotary position encoding.

partial_rotary_factor​

partial_rotary_factor: float = 0.25

source

Fraction of head_dim that gets rotary position embedding.

quant_scheme​

quant_scheme: Qwen3_5QuantScheme | None = None

source

Which modules are quantized and how; set by _parse_quant_config().

state_dtype​

property state_dtype: DType

source

Storage dtype of the linear-attention state pools.

Every declarer of a pool buffer reads this one property, since the base and fused speculative graphs share one allocation at serve time. Never the KV cache’s dtype.

state_pool_dtype​

state_pool_dtype: DType | None = None

source

Storage dtype override for the linear-attention state pools.

None (the default) stores both pools at compute_dtype (bfloat16), the configuration every exported artifact and the Mach registry declare. float32 makes a speculated generation follow the exact state trajectory of an unspeculated one — the recurrence rounds to the pool dtype only at a call boundary, so a lossy pool makes the trajectory depend on speculation’s chunking — at roughly double the per-request state memory (74.8 to 149.6 MiB for Qwen3.8-27B). Set via the state_pool_dtype KV-cache config knob; read through state_dtype.

video_token_id​

video_token_id: int | None = None

source

Token ID used for video placeholders in the input sequence.

vision_config​

vision_config: VisionConfig | None = None

source

Vision encoder configuration; None for text-only models.

vision_start_token_id​

vision_start_token_id: int | None = None

source

Token ID that marks the start of vision content.

Qwen3_5Inputs​

class max.pipelines.architectures.qwen3_5.Qwen3_5Inputs(tokens, input_row_offsets, signal_buffers, return_n_logits, data_parallel_splits=None, request_ids=None, decoder_position_ids=None, *, kv_cache_inputs=None, lora_buffers=(), vision_embeddings=<factory>, vision_scatter_indices=<factory>, hidden_states=None)

source

Bases: Llama3Inputs

Inputs for Qwen3.5, including the linear-attention state pools.

Image embeddings come from the pipeline-driven encoder cache on the base vision_embeddings / vision_scatter_indices fields.

Parameters:

buffers​

property buffers: tuple[Buffer, ...]

source

Returns positional Buffer inputs for model ABI calls.

decoder_position_ids​

decoder_position_ids: Buffer | None = None

source

[3, total_seq_len] M-RoPE positions, one column per active token.

Present only when the graph was built with M-RoPE wired in; see Qwen3_5.mrope_enabled.

request_ids​

request_ids: list[RequestID] | None = None

source

Request IDs for this batch, used to manage per-request state cache slots.

Qwen3_5Model​

class max.pipelines.architectures.qwen3_5.Qwen3_5Model(pipeline_config, session, devices, kv_cache_config, weights, *, memory_plan, adapter=None, return_logits=ReturnLogits.LAST_TOKEN, return_hidden_states=ReturnHiddenStates.NONE, max_batch_size=1)

source

Bases: AlwaysSignalBuffersMixin, LlamaModelBase

Qwen3.5 pipeline model implementation.

Supports the hybrid linear/full attention architecture with KV cache for full attention layers and conv/recurrent states for linear layers.

Parameters:

attention_bias​

attention_bias: bool = False

source

Whether to use attention bias.

batch_processor_cls​

batch_processor_cls

source

alias of Qwen3_5BatchProcessor

check_state_budget()​

check_state_budget()

source

Checks the state the cache declares against what planning budgeted.

The two resolve state_dtype either side of finalize, and a disagreement surfaces only as an OOM at load.

Return type:

None

empty_vision_embeddings()​

empty_vision_embeddings(devices)

source

Per-device zero-row image embeddings for cached / text-only batches.

Cached: hit on every text-only / decode step, so it must not allocate per call, and graph-capture replay only skips an input refresh for an identical buffer object.

Parameters:

devices (list[Device])

Return type:

list[Buffer]

execute()​

execute(model_inputs)

source

Executes the graph with the given inputs.

Parameters:

model_inputs (ModelInputs) – The model inputs to execute, containing tensors and any other required data for model execution.

Returns:

ModelOutputs containing the pipeline’s output tensors.

Return type:

ModelOutputs

This is an abstract method that must be implemented by concrete PipelineModels to define their specific execution logic.

load_model()​

load_model(session)

source

Build, compile, and load the model graph into session.

Parameters:

session (InferenceSession)

Return type:

Model

model​

model: Model

source

Compiled and initialized model ready for inference.

model_config_cls​

model_config_cls

source

alias of Qwen3_5Config

norm_method​

norm_method: Literal['rms_norm', 'layer_norm'] = 'rms_norm'

source

Normalization layer.

pack_vision_inputs()​

pack_vision_inputs(selection, devices)

source

Pack the pipeline-selected cache-miss images to device.

Runs in the pipeline’s prep-ahead window so the host-to-device copy overlaps the prior batch.

Parameters:

Return type:

Qwen3_5VisionInputs | None

state_dict​

state_dict: dict[str, Any]

source

Weights to load into the model.

vision_execute()​

vision_execute(selection, devices, packed)

source

Run the vision encoder on the images pack_vision_inputs packed.

Returns embeddings only; the pipeline derives per-image token counts from its selection, which match because the tokenizer emits exactly one placeholder per merged patch.

Parameters:

Return type:

VisionEncodeResult

vision_model​

vision_model: Model | None = None

source

Qwen3_5ReasoningParser​

class max.pipelines.architectures.qwen3_5.Qwen3_5ReasoningParser(think_start_token_id, think_end_token_id, tool_call_start_token_id=None)

source

Bases: ReasoningParser

Qwen 3.5 / 3.6 reasoning parser for <think>...</think> sections.

Qwen 3.5/3.6’s chat template prepends <think>\n to every assistant turn when enable_thinking is true (the default), so reasoning begins implicitly without an explicit <think> token in the model output stream. Reasoning ends explicitly at </think>, or implicitly when a tool call begins (<tool_call>) — the tool-call marker is left in the content region for the tool parser to consume.

Parameters:

  • think_start_token_id (int)
  • think_end_token_id (int)
  • tool_call_start_token_id (int | None)

REASONING_END​

REASONING_END: ClassVar[str] = '</think>'

source

Text delimiter that closes a reasoning span (e.g. "</think>").

See REASONING_START for the declaration contract.

REASONING_START​

REASONING_START: ClassVar[str] = '<think>'

source

Text delimiter that opens a reasoning span (e.g. "<think>").

Subclasses declare their delimiters here and resolve token ids from them, so each model’s delimiters are written down exactly once. Consumers that work in the text domain rather than the token domain read them from here instead of restating them.

Declare both delimiters or neither. None means this parser has no text form at all, so a text-domain consumer cannot bound a reasoning span and will leave reasoning in the assistant’s content; declaring only one is a bug, since a span needs both ends. A parser whose chat template prefills the opening delimiter still declares it – whether a given turn emits it is a property of the request, not of the parser.

from_tokenizer()​

async classmethod from_tokenizer(tokenizer)

source

Construct a reasoning parser from a tokenizer.

Parameters:

tokenizer (PipelineTokenizer[Any, Any, Any])

Return type:

Qwen3_5ReasoningParser

reasoning_end_token_id()​

async classmethod reasoning_end_token_id(tokenizer)

source

Returns the </think> token id that closes a reasoning span.

Parameters:

tokenizer (PipelineTokenizer[Any, Any, Any])

Return type:

int | None

stream()​

stream(delta_token_ids, is_currently_reasoning=True)

source

Identify a reasoning span within a streaming delta chunk.

When is_currently_reasoning=False and the chunk contains no <think> opener, returns an empty span so post-reasoning content chunks aren’t misclassified as reasoning.

Parameters:

Return type:

ParsedReasoningDelta

will_reason_after_prompt()​

will_reason_after_prompt(prompt_token_ids)

source

Decide whether the next generated token continues a reasoning span.

Overrides the ABC default (which delegates to stream scanning left-to-right). That default is wrong for Qwen: the chat template embeds a literal <tool_call> example in the tool instructions, and <tool_call> is a reasoning-end delimiter — so a left-to-right scan hits the example and falsely concludes reasoning already ended, leaking the model’s <think> block into content.

Multi-turn prompts can also contain <think>/</think> tokens from prior assistant turns; only the most-recently-emitted delimiter describes the current state. Scan right-to-left: the last delimiter before generation is the chat template’s prefilled <think>.

Parameters:

prompt_token_ids (Sequence[int])

Return type:

bool

Qwen3_5ToolParser​

class max.pipelines.architectures.qwen3_5.Qwen3_5ToolParser

source

Bases: object

Parser for Qwen 3.5 / 3.6 tool calls.

CALL_BEGIN​

CALL_BEGIN: ClassVar[str] = '<tool_call>'

source

parse_complete()​

parse_complete(response)

source

Parse a complete model response into tool calls.

Parameters:

response (str)

Return type:

ParsedToolResponse

parse_delta()​

parse_delta(delta)

source

Incrementally process one decoded-token delta.

Returns content text to forward to the client and any tool-call increments to emit, in the order they were produced. Content deltas have content set; tool-call deltas have one or more of id / name / arguments set.

Parameters:

delta (str)

Return type:

list[ParsedToolCallDelta] | None

reset()​

reset()

source

Reset internal state for a new streaming session.

Return type:

None

set_streaming_tool_schemas()​

set_streaming_tool_schemas(schemas)

source

No-op: this format does not need schema-driven streaming.

See ToolParser.set_streaming_tool_schemas.

Parameters:

schemas (Mapping[str, dict[str, Any]])

Return type:

None

Models

Cookbook

Puzzles

Blog

Community

Contact