IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python module

max.pipelines.architectures.qwen3_5

Qwen3.5 architecture: a hybrid model interleaving Gated DeltaNet linear attention with full attention, plus a vision encoder.

Qwen3_5Config​

class max.pipelines.architectures.qwen3_5.Qwen3_5Config(*, hidden_size, num_attention_heads, num_key_value_heads, num_hidden_layers, rope_theta, rope_scaling_params, max_seq_len, intermediate_size, interleaved_rope_weights, vocab_size, dtype, model_quantization_encoding, quantization_config, kv_params, return_logits=ReturnLogits.LAST_TOKEN, norm_method='rms_norm', norm_dtype=None, attention_bias=False, rms_norm_eps=None, tie_word_embeddings=False, stacked_mlp=False, stacked_qkv=False, attention_multiplier, embedding_multiplier, residual_multiplier, devices, clip_qkv, quant_config=None, longrope_scaling_params=None, logits_scaling=1.0, return_hidden_states=ReturnHiddenStates.NONE, target_layer_ids=None, use_subgraphs=True, data_parallel_degree=1, sliding_window=None, quantization_encoding=None, layer_types=<factory>, full_attention_interval=4, linear_key_head_dim=128, linear_value_head_dim=128, linear_num_key_heads=16, linear_num_value_heads=48, linear_conv_kernel_dim=4, partial_rotary_factor=0.25, attn_output_gate=True, mamba_ssm_dtype=float32, state_pool_dtype=None, vision_config=None, mrope_without_encoder=False, image_token_id=None, video_token_id=None, vision_start_token_id=None, mrope_section=None, hf_quantization_config=None, quant_scheme=None, declared_dtype=None)

source

Bases: Llama3Config, ArchConfigWithVisionCache

Configuration for Qwen3.5 hybrid attention models.

Qwen3.5 uses a hybrid architecture with both full (standard) attention and linear attention (Gated DeltaNet) layers. Every full_attention_interval-th layer uses full attention, and the rest use linear attention.

Parameters:

DEFAULT_ENCODING​

DEFAULT_ENCODING: ClassVar[SupportedEncoding] = 'bfloat16'

source

SUPPORTED_ENCODINGS​

SUPPORTED_ENCODINGS: ClassVar[set[SupportedEncoding]] = {'bfloat16', 'float32', 'float4_e2m1fnx2', 'float8_e4m3fn'}

source

attn_kv_params​

property attn_kv_params: KVCacheParams

source

The attention child of kv_params.

attn_output_gate​

attn_output_gate: bool = True

source

Whether full attention layers use a sigmoid output gate.

calculate_attention_multiplier()​

static calculate_attention_multiplier(huggingface_config)

source

Compute attention scaling factor using explicit head_dim.

Parameters:

huggingface_config (AutoConfig)

Return type:

float

calculate_max_seq_len()​

classmethod calculate_max_seq_len(huggingface_config, model_config)

source

Bounds against the text config’s max_position_embeddings.

Parameters:

Return type:

int

compute_dtype​

property compute_dtype: DType

source

Dtype of activations and every unquantized weight.

dtype is the storage dtype the resolved encoding implies (uint8 for packed NVFP4), which is not what the norms, embeddings, conv1d or linear-attention state pools use.

quant_scheme is authoritative but only exists after finalize; declared_dtype covers the earlier window so a pre-finalize caller does not silently read the storage dtype.

construct_kv_params()​

classmethod construct_kv_params(huggingface_config, pipeline_config, devices, kv_cache_config, cache_dtype, *, allow_kv_head_replication=False)

source

Returns the attention leaf beside the recurrent state.

The graph’s input types are built from these params, so a state derived any later cannot appear among them. A model with no linear-attention layers gets the attention leaf alone.

Parameters:

Return type:

KVCacheParamInterface

declared_dtype​

declared_dtype: DType | None = None

source

The dtype the checkpoint declares for its unquantized tensors.

Captured at initialize_from_config so compute_dtype is right before finalize resolves quant_scheme. Memory planning runs in that window.

declared_state_dtype()​

static declared_state_dtype(text_config, kv_cache_config)

source

Returns state_dtype as far as it is knowable pre-finalize.

The state leaves are declared before a quantization scheme resolves. cache_dtype is not a stand-in: --kv-cache-format may put the KV in fp8 while the state stays bf16.

Parameters:

Return type:

DType

estimate_vision_cache_entry_bytes()​

classmethod estimate_vision_cache_entry_bytes(huggingface_config)

source

Estimates per-entry bytes for the Qwen3.5 vision encoder cache.

Qwen3.5’s tower is NaViT-style: an image keeps its aspect ratio and yields one token per merged patch, so — unlike Gemma4’s fixed pooled patch count — no per-image token count exists in the config to read. The bound that does exist is the image processor’s post-resize pixel ceiling, which every served image is smart-resized under, so the largest entry the cache can be asked to hold is that ceiling divided by the patch area and the spatial merge.

Returns:

Estimated bytes per vision cache entry, or 0 for a checkpoint with no vision tower.

Parameters:

huggingface_config (AutoConfig)

Return type:

int

full_attention_interval​

full_attention_interval: int = 4

source

Every N-th layer uses full attention.

get_num_layers()​

static get_num_layers(huggingface_config)

source

Layer count for the decoder stack (override when HF uses a different field).

Parameters:

huggingface_config (AutoConfig)

Return type:

int

get_vision_cache_row_spec()​

classmethod get_vision_cache_row_spec(huggingface_config)

source

One embedding row per merged vision token, at the LM hidden size.

The dtype is the checkpoint’s declared one rather than a fixed bfloat16, because the block pool has to match the buffers the encoder hands it: those are cast to compute_dtype, which for every encoding this architecture supports resolves to the declared dtype.

Parameters:

huggingface_config (AutoConfig)

Return type:

tuple[int, DType] | None

hf_quantization_config​

hf_quantization_config: dict[str, Any] | None = None

source

The checkpoint’s resolved Hugging Face quantization config.

Captured at initialize_from_config because it lives on the top-level multimodal config, while finalize is handed text_config.

image_token_id​

image_token_id: int | None = None

source

Token ID used for image placeholders in the input sequence.

infer_optimal_batch_size()​

infer_optimal_batch_size(devices, *, weights_size, device_memory_utilization, extra_per_request_bytes=0)

source

Return a memory-safe default max_batch_size for this architecture.

The states get up to half the post-weights utilization budget and the KV absorbs the rest, under the same device_memory_utilization headroom factor as the rest of the pipeline. The halving bounds concurrency; it allocates nothing.

Falls back to 32—safe for the 27B model on H100/A100 (80 GB)—when the device query fails.

Parameters:

  • devices (list[Device]) – Loaded devices the model will run on.
  • weights_size (int) – Estimated model weights size in bytes.
  • device_memory_utilization (float) – Headroom factor.
  • extra_per_request_bytes (int) – Per-request state the architecture holds beyond the pool set this config declares – a speculative arch’s verify ring, for one. Added to the divisor so the inferred batch fits what will really be allocated.

Return type:

int

initialize()​

classmethod initialize(pipeline_config, model_config=None, *, max_seq_len)

source

Initialize the config from a PipelineConfig.

Parameters:

  • pipeline_config (PipelineConfig) – The pipeline configuration.
  • model_config (MAXModelConfig | None) – The model configuration to read from. When None (the default), pipeline_config.model is used. Pass an explicit config (such as pipeline_config.draft_model) to initialize the arch config for a different model.
  • max_seq_len (int) – The effective maximum sequence length to store on the config. The value is received, never derived here: the pipeline model passes the memory plan’s VRAM-clamped length, while memory planning (which runs before a plan exists) passes the construction-resolved model_config.max_length. Configs whose sequence length is pure model metadata (such as diffusion components) ignore it.

Return type:

Self

initialize_from_config()​

classmethod initialize_from_config(pipeline_config, huggingface_config, model_config=None, *, max_seq_len)

source

Initialize config from pipeline and HuggingFace configurations.

Handles both multimodal (Qwen3_5ForConditionalGeneration) and text-only (Qwen3_5ForCausalLM) configs by extracting the text config.

Parameters:

Return type:

Self

kv_params​

kv_params: KVCacheParamInterface

source

The cache tree: an attention child beside a recurrent-state one.

attn_kv_params is the leaf the attention layers take.

layer_types​

layer_types: list[str]

source

Per-layer attention type: ‘full_attention’ or ‘linear_attention’.

linear_conv_kernel_dim​

linear_conv_kernel_dim: int = 4

source

Causal conv1d kernel size for linear attention layers.

linear_key_head_dim​

linear_key_head_dim: int = 128

source

Key head dimension for linear attention layers.

linear_num_key_heads​

linear_num_key_heads: int = 16

source

Number of key heads for linear attention layers.

linear_num_value_heads​

linear_num_value_heads: int = 48

source

Number of value heads for linear attention layers.

linear_value_head_dim​

linear_value_head_dim: int = 128

source

Value head dimension for linear attention layers.

mamba_ssm_dtype​

mamba_ssm_dtype: DType = 81

source

Dtype for SSM (state space model) computations in linear attention layers.

mrope_enabled​

property mrope_enabled: bool

source

Whether the model takes 3-axis M-RoPE positions.

vision_config stands in for “an image can appear in context”, and mrope_without_encoder keeps them without the tower.

mrope_section​

mrope_section: list[int] | None = None

source

MRoPE section lengths for multimodal rotary position encoding.

mrope_without_encoder​

mrope_without_encoder: bool = False

source

Keep M-RoPE on a graph that compiles no vision encoder.

vision_config normally carries two facts at once: that an encoder must be built, and that images can appear in a request’s context. The fused speculative graph separates them — it compiles no encoder and takes the encoder’s output as inputs instead, but a request whose context holds an image still needs 3-axis positions for every token after it. Setting this keeps mrope_enabled true with vision_config cleared.

partial_rotary_factor​

partial_rotary_factor: float = 0.25

source

Fraction of head_dim that gets rotary position embedding.

quant_scheme​

quant_scheme: Qwen3_5QuantScheme | None = None

source

Which modules are quantized and how; set by _parse_quant_config().

state_dtype​

property state_dtype: DType

source

Storage dtype of the linear-attention state pools.

Every declarer of a pool buffer reads this one property, since the base and fused speculative graphs share one allocation at serve time. Never the KV cache’s dtype.

state_pool_dtype​

state_pool_dtype: DType | None = None

source

Storage dtype override for the linear-attention state pools.

None (the default) stores both pools at compute_dtype (bfloat16), the configuration every exported artifact and the Mach registry declare. float32 makes a speculated generation follow the exact state trajectory of an unspeculated one — the recurrence rounds to the pool dtype only at a call boundary, so a lossy pool makes the trajectory depend on speculation’s chunking — at roughly double the per-request state memory (74.8 to 149.6 MiB for Qwen3.8-27B). Set via the state_pool_dtype KV-cache config knob; read through state_dtype.

video_token_id​

video_token_id: int | None = None

source

Token ID used for video placeholders in the input sequence.

vision_config​

vision_config: VisionConfig | None = None

source

Vision encoder configuration; None for text-only models.

vision_start_token_id​

vision_start_token_id: int | None = None

source

Token ID that marks the start of vision content.

Qwen3_5Inputs​

class max.pipelines.architectures.qwen3_5.Qwen3_5Inputs(tokens, input_row_offsets, signal_buffers, return_n_logits, data_parallel_splits=None, request_ids=None, decoder_position_ids=None, vision_inputs_in_graph=True, *, kv_cache_inputs=None, lora_buffers=(), vision_embeddings=<factory>, vision_scatter_indices=<factory>, hidden_states=None)

source

Bases: Llama3Inputs

Inputs for Qwen3.5, including the linear-attention state pools.

Image embeddings come from the pipeline-driven encoder cache on the base vision_embeddings / vision_scatter_indices fields.

Parameters:

buffers​

property buffers: tuple[Buffer, ...]

source

Returns positional Buffer inputs for model ABI calls.

decoder_position_ids​

decoder_position_ids: Buffer | None = None

source

[3, total_seq_len] M-RoPE positions, one column per active token.

Present only when the graph was built with M-RoPE wired in; see Qwen3_5.mrope_enabled.

request_ids​

request_ids: list[RequestID] | None = None

source

Request IDs for this batch, used to manage per-request state cache slots.

vision_inputs_in_graph​

vision_inputs_in_graph: bool = True

source

Whether the graph declares the vision-merge inputs. False for text-only checkpoints, which compile no vision encoder and so take none.

Qwen3_5Model​

class max.pipelines.architectures.qwen3_5.Qwen3_5Model(pipeline_config, session, devices, kv_cache_config, weights, *, memory_plan, adapter=None, return_logits=ReturnLogits.LAST_TOKEN, return_hidden_states=ReturnHiddenStates.NONE, max_batch_size=1)

source

Bases: AlwaysSignalBuffersMixin, LlamaModelBase

Qwen3.5 pipeline model implementation.

Supports the hybrid linear/full attention architecture with KV cache for full attention layers and conv/recurrent states for linear layers.

Parameters:

attention_bias​

attention_bias: bool = False

source

Whether to use attention bias.

batch_processor_cls​

batch_processor_cls

source

alias of Qwen3_5BatchProcessor

build_module()​

classmethod build_module(context)

source

Builds the hybrid model from the checkpoint context names.

A caller that sized the cache passes its KV params. A module that encodes no images builds no vision tower, but keeps the 3-axis positions of a checkpoint that has one.

Parameters:

context (ModelLoadContext)

Return type:

tuple[Qwen3_5, Qwen3_5Config]

check_state_budget()​

check_state_budget()

source

Checks the state the cache declares against what planning budgeted.

The two resolve state_dtype either side of finalize, and a disagreement surfaces only as an OOM at load.

Return type:

None

empty_vision_embeddings()​

empty_vision_embeddings(devices)

source

Per-device zero-row image embeddings for cached / text-only batches.

Cached: hit on every text-only / decode step, so it must not allocate per call, and graph-capture replay only skips an input refresh for an identical buffer object.

Parameters:

devices (list[Device])

Return type:

list[Buffer]

execute()​

execute(model_inputs)

source

Executes the graph with the given inputs.

Parameters:

model_inputs (ModelInputs) – The model inputs to execute, containing tensors and any other required data for model execution.

Returns:

ModelOutputs containing the pipeline’s output tensors.

Return type:

ModelOutputs

This is an abstract method that must be implemented by concrete PipelineModels to define their specific execution logic.

graph_specs()​

graph_specs(model_config)

source

Returns the decoder, and the vision encoder unless the checkpoint is text-only, in one compile group.

Parameters:

model_config (Qwen3_5Config)

Return type:

dict[str, GraphSpec]

load_model()​

load_model(session)

source

Loads the decoder graph graph_specs() returns.

A subclass that still overrides _build_graph_for_compile loads through that hook instead.

Parameters:

session (InferenceSession) – The session that compiles and initializes the graph.

Returns:

The initialized decoder.

Return type:

Model

model​

model: Model

source

Compiled and initialized model ready for inference.

model_config_cls​

model_config_cls

source

alias of Qwen3_5Config

norm_method​

norm_method: Literal['rms_norm', 'layer_norm'] = 'rms_norm'

source

Normalization layer.

pack_vision_inputs()​

pack_vision_inputs(selection, devices)

source

Pack the pipeline-selected cache-miss images to device.

Runs in the pipeline’s prep-ahead window so the host-to-device copy overlaps the prior batch.

Parameters:

Return type:

Qwen3_5VisionInputs | None

state_dict​

state_dict: dict[str, Any]

source

Weights to load into the model.

vision_execute()​

vision_execute(selection, devices, packed)

source

Run the vision encoder on the media pack_vision_inputs packed.

Returns explicit per-item token counts rather than letting the driver derive them from placeholder spans. A clip’s span covers grid_t placeholder runs AND the timestamp text between them, so its width exceeds the rows the encoder emits; an image’s span happens to match, which is why the derived counts were correct until now.

Parameters:

Return type:

VisionEncodeResult

vision_model​

vision_model: Model | None = None

source

Qwen3_5ReasoningParser​

class max.pipelines.architectures.qwen3_5.Qwen3_5ReasoningParser(think_start_token_id, think_end_token_id, tool_call_start_token_id=None)

source

Bases: ReasoningParser

Qwen 3.5 / 3.6 reasoning parser for <think>...</think> sections.

Qwen 3.5/3.6’s chat template prepends <think>\n to every assistant turn when enable_thinking is true (the default), so reasoning begins implicitly without an explicit <think> token in the model output stream. Reasoning ends explicitly at </think>, or implicitly when a tool call begins (<tool_call>) — the tool-call marker is left in the content region for the tool parser to consume.

Parameters:

  • think_start_token_id (int)
  • think_end_token_id (int)
  • tool_call_start_token_id (int | None)

REASONING_END​

REASONING_END: ClassVar[str] = '</think>'

source

Text delimiter that closes a reasoning span, such as "</think>".

See REASONING_START for the declaration contract.

REASONING_START​

REASONING_START: ClassVar[str] = '<think>'

source

Text delimiter that opens a reasoning span, such as "<think>".

Subclasses declare their delimiters here and resolve token ids from them, so each model’s delimiters are written down exactly once. Consumers that work in the text domain rather than the token domain read them from here instead of restating them.

Declare both delimiters or neither. None means this parser has no text form at all, so a text-domain consumer cannot bound a reasoning span and will leave reasoning in the assistant’s content; declaring only one is a bug, since a span needs both ends. A parser whose chat template prefills the opening delimiter still declares it – whether a given turn emits it is a property of the request, not of the parser.

from_tokenizer()​

async classmethod from_tokenizer(tokenizer)

source

Construct a reasoning parser from a tokenizer.

Parameters:

tokenizer (PipelineTokenizer[Any, Any, Any])

Return type:

Qwen3_5ReasoningParser

reasoning_end_token_id()​

async classmethod reasoning_end_token_id(tokenizer)

source

Returns the </think> token id that closes a reasoning span.

Parameters:

tokenizer (PipelineTokenizer[Any, Any, Any])

Return type:

int | None

stream()​

stream(delta_token_ids, is_currently_reasoning=True)

source

Identify a reasoning span within a streaming delta chunk.

When is_currently_reasoning=False and the chunk contains no <think> opener, returns an empty span so post-reasoning content chunks aren’t misclassified as reasoning.

Parameters:

Return type:

ParsedReasoningDelta

will_reason_after_prompt()​

will_reason_after_prompt(prompt_token_ids)

source

Decide whether the next generated token continues a reasoning span.

Overrides the ABC default (which delegates to stream scanning left-to-right). That default is wrong for Qwen: the chat template embeds a literal <tool_call> example in the tool instructions, and <tool_call> is a reasoning-end delimiter — so a left-to-right scan hits the example and falsely concludes reasoning already ended, leaking the model’s <think> block into content.

Multi-turn prompts can also contain <think>/</think> tokens from prior assistant turns; only the most-recently-emitted delimiter describes the current state. Scan right-to-left: the last delimiter before generation is the chat template’s prefilled <think>.

Parameters:

prompt_token_ids (Sequence[int])

Return type:

bool

Qwen3_5ToolParser​

class max.pipelines.architectures.qwen3_5.Qwen3_5ToolParser

source

Bases: object

Parser for Qwen 3.5 / 3.6 tool calls.

CALL_BEGIN​

CALL_BEGIN: ClassVar[str] = '<tool_call>'

source

XGRAMMAR_FORMAT​

XGRAMMAR_FORMAT = 'qwen_3_5'

source

generate_tool_call_grammar()​

static generate_tool_call_grammar(response_format_schema=None, tools=None, tokenizer=None, backend='xgrammar', tool_choice=None, reject_unsupported=False, **kwargs)

source

Build a constrained-decoding grammar for Qwen 3.5 tool calls.

Returns a serialized xgrammar StructuralTag that frames the <tool_call> envelope and constrains each call’s arguments to that tool’s JSON schema. When response_format_schema is provided, the grammar also accepts a JSON response matching the schema (the model’s first tokens select the branch).

Parameters:

  • response_format_schema (dict[str, Any] | None) – Optional JSON schema dict. When provided, the grammar also accepts a JSON response matching the schema.
  • tools (list[dict[str, Any]] | None) – Optional list of OpenAI-style tool dicts.
  • tokenizer (Any | None) – Unused (the xgrammar tag references literal markers).
  • backend (str) – Structured-output backend; must be "xgrammar".
  • tool_choice (str | dict[str, Any] | None) – "auto", "required", or a named choice.
  • reject_unsupported (bool) – Whether to raise an error during grammar compilation when a tool’s schema cannot be enforced. Otherwise, the schema is enforced best-effort.
  • **kwargs (Any) – Ignored; accepts future kwargs.

Returns:

The StructuralTag serialized as a JSON string.

Return type:

str

parse_complete()​

parse_complete(response)

source

Parse a complete model response into tool calls.

Parameters:

response (str)

Return type:

ParsedToolResponse

parse_delta()​

parse_delta(delta)

source

Incrementally process one decoded-token delta.

Returns content text to forward to the client and any tool-call increments to emit, in the order they were produced. Content deltas have content set; tool-call deltas have one or more of id / name / arguments set.

Parameters:

delta (str)

Return type:

list[ParsedToolCallDelta] | None

reset()​

reset()

source

Reset internal state for a new streaming session.

Return type:

None

set_streaming_tool_schemas()​

set_streaming_tool_schemas(schemas)

source

Records the schemas for typed argument decoding while streaming.

See ToolParser.set_streaming_tool_schemas and set_tool_schemas().

Parameters:

schemas (Mapping[str, dict[str, Any]])

Return type:

None

set_tool_schemas()​

set_tool_schemas(schemas)

source

Supplies each tool’s parameters schema, keyed by function name.

Optional hook, called by the router before parsing a request’s output (streaming or not) so values are decoded by their declared type.

Parameters:

schemas (Mapping[str, dict[str, Any]])

Return type:

None

Models

Cookbook

Puzzles

Blog

Community