IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python module

max.pipelines.architectures.deepseekV4

DeepSeek-V4-Flash mixture-of-experts architecture for text generation.

DeepseekV4Config​

class max.pipelines.architectures.deepseekV4.DeepseekV4Config(*, dtype, kv_params, devices, use_subgraphs=True, data_parallel_degree=1, quantization_encoding=None, max_seq_len, vocab_size=129280, hidden_size=4096, num_hidden_layers=43, num_attention_heads=64, num_key_value_heads=1, rms_norm_eps=1e-06, hidden_act='silu', tie_word_embeddings=False, attention_bias=False, attention_dropout=0.0, head_dim=512, qk_rope_head_dim=64, q_lora_rank=1024, o_lora_rank=1024, o_groups=8, sliding_window=128, compress_ratios=<factory>, compress_rope_theta=160000.0, index_head_dim=128, index_n_heads=64, index_topk=512, moe_intermediate_size=2048, n_routed_experts=256, n_shared_experts=1, num_experts_per_tok=6, routed_scaling_factor=1.5, scoring_func='sqrtsoftplus', norm_topk_prob=True, swiglu_limit=10.0, expert_dtype='fp4', num_hash_layers=3, hc_mult=4, hc_sinkhorn_iters=20, hc_eps=1e-06, dspark_block_size=5, dspark_noise_token_id=128799, dspark_target_layer_ids=<factory>, dspark_markov_rank=256, max_position_embeddings=1048576, rope_theta=10000.0, rope_scaling=None, rope_interleave=True, norm_dtype=bfloat16, gate_dtype=None, correction_bias_dtype=None, max_batch_context_length=131072, quant_config=None, native_routed_experts=None, graph_mode='auto', return_logits=ReturnLogits.LAST_TOKEN, return_hidden_states=ReturnHiddenStates.NONE, dspark_stages=True)

source

Bases: ArchConfigWithKVCache

Configuration for DeepSeek-V4-Flash models.

V4 is not a V3.2 variant despite the shared lineage. It replaces V3.2’s MLA (kv_lora_rank + qk_nope_head_dim + v_head_dim + kv_b_proj) with a single head_dim-wide shared latent produced by wkv: query and key live in the same 512-dim space, the last rope_head_dim dims carry RoPE, and there is no per-head KV up-projection at all. It also adds sliding-window + compressed-token sparse attention (CSA), a grouped low-rank output projection, hyper-connection residuals (mHC), and hash routing on the first num_hash_layers MoE layers. Hence a standalone config rather than a subclass of DeepseekV3_2Config.

Parameters:

DEFAULT_ENCODING​

DEFAULT_ENCODING: ClassVar[SupportedEncoding] = 'float8_e4m3fn'

source

SUPPORTED_ENCODINGS​

SUPPORTED_ENCODINGS: ClassVar[set[SupportedEncoding]] = {'float32', 'float8_e4m3fn'}

source

attention_bias​

attention_bias: bool = False

source

attention_dropout​

attention_dropout: float = 0.0

source

calculate_max_seq_len()​

classmethod calculate_max_seq_len(huggingface_config, model_config)

source

Returns the resolved maximum sequence length.

Bounds or defaults the user’s model_config.max_length with the model’s own limits. Construction runs this once and stores the result on model_config.max_length; memory planning may lower it further, but only on the memory plan.

Parameters:

  • huggingface_config (AutoConfig) – The HuggingFace config to read model bounds from.
  • model_config (MAXModelConfig) – The model config whose max_length carries the user’s setting.

Return type:

int

compress_ratios​

compress_ratios: list[int]

source

compress_rope_theta​

compress_rope_theta: float = 160000.0

source

compressed_layer_index()​

compressed_layer_index(layer_idx)

source

Rank of a compressed layer among trunk layers sharing its ratio.

That rank is its layer index within the ratio’s zone and state leaves.

Parameters:

layer_idx (int)

Return type:

int

construct_kv_params()​

static construct_kv_params(huggingface_config, pipeline_config, devices, kv_cache_config, cache_dtype)

source

Build the seven-leaf KV tree (see kv_leaf_specs).

The reference sizes each layer’s cache as window_size + max_seq_len // compress_ratio. Here the window is one sliding-window leaf over every layer and each ratio’s compressed entries are a zone leaf with page_size // ratio slots per page, so no layer over-allocates.

Parameters:

Return type:

KVCacheParamInterface

correction_bias_dtype​

correction_bias_dtype: DType | None = None

source

data_parallel_degree​

data_parallel_degree: int = 1

source

devices​

devices: list[DeviceRef]

source

dspark_block_size​

dspark_block_size: int = 5

source

dspark_markov_rank​

dspark_markov_rank: int = 256

source

dspark_noise_token_id​

dspark_noise_token_id: int = 128799

source

dspark_stages​

dspark_stages: bool = True

source

dspark_target_layer_ids​

dspark_target_layer_ids: list[int]

source

dtype​

dtype: DType

source

expert_dtype​

expert_dtype: str | None = 'fp4'

source

gate_dtype​

gate_dtype: DType | None = None

source

get_kv_params()​

get_kv_params()

source

KV cache parameters to use when running the model.

Return type:

KVCacheParamInterface

get_max_seq_len()​

get_max_seq_len()

source

Returns the effective maximum sequence length for the model.

For configs that store a deployment length, this is the value initialize received; for metadata-only configs it derives from the checkpoint.

Return type:

int

get_num_layers()​

static get_num_layers(huggingface_config)

source

Parameters:

huggingface_config (AutoConfig)

Return type:

int

graph_mode​

graph_mode: str = 'auto'

source

hc_eps​

hc_eps: float = 1e-06

source

hc_mult​

hc_mult: int = 4

source

hc_sinkhorn_iters​

hc_sinkhorn_iters: int = 20

source

head_dim​

head_dim: int = 512

source

hidden_act​

hidden_act: str = 'silu'

source

hidden_size​

hidden_size: int = 4096

source

index_head_dim​

index_head_dim: int = 128

source

index_n_heads​

index_n_heads: int = 64

source

index_topk​

index_topk: int = 512

source

initialize()​

classmethod initialize(pipeline_config, model_config=None, *, max_seq_len)

source

Initialize the config from a PipelineConfig.

Parameters:

  • pipeline_config (PipelineConfig) – The pipeline configuration.
  • model_config (MAXModelConfig | None) – The model configuration to read from. When None (the default), pipeline_config.model is used. Pass an explicit config (such as pipeline_config.draft_model) to initialize the arch config for a different model.
  • max_seq_len (int) – The effective maximum sequence length to store on the config. The value is received, never derived here: the pipeline model passes the memory plan’s VRAM-clamped length, while memory planning (which runs before a plan exists) passes the construction-resolved model_config.max_length. Configs whose sequence length is pure model metadata (such as diffusion components) ignore it.

Return type:

Self

kv_leaf_specs()​

kv_leaf_specs()

source

Return type:

list[KVLeafSpec]

kv_params​

kv_params: KVCacheParamInterface

source

layer_compress_ratio()​

layer_compress_ratio(layer_idx)

source

Parameters:

layer_idx (int)

Return type:

int

layer_has_indexer()​

layer_has_indexer(layer_idx)

source

Parameters:

layer_idx (int)

Return type:

bool

layer_is_hash_routed()​

layer_is_hash_routed(layer_idx)

source

Parameters:

layer_idx (int)

Return type:

bool

max_batch_context_length​

max_batch_context_length: int = 131072

source

max_position_embeddings​

max_position_embeddings: int = 1048576

source

max_seq_len​

max_seq_len: int

source

moe_intermediate_size​

moe_intermediate_size: int = 2048

source

n_routed_experts​

n_routed_experts: int = 256

source

n_shared_experts​

n_shared_experts: int = 1

source

native_routed_experts​

native_routed_experts: bool | None = None

source

norm_dtype​

norm_dtype: DType = 80

source

norm_topk_prob​

norm_topk_prob: bool = True

source

num_attention_heads​

num_attention_heads: int = 64

source

num_experts_per_tok​

num_experts_per_tok: int = 6

source

num_hash_layers​

num_hash_layers: int = 3

source

num_hidden_layers​

num_hidden_layers: int = 43

source

num_key_value_heads​

num_key_value_heads: int = 1

source

o_groups​

o_groups: int = 8

source

o_lora_rank​

o_lora_rank: int = 1024

source

q_lora_rank​

q_lora_rank: int = 1024

source

qk_nope_head_dim​

property qk_nope_head_dim: int

source

Width of the non-RoPE portion of the shared latent.

qk_rope_head_dim​

qk_rope_head_dim: int = 64

source

quant_config​

quant_config: QuantConfig | None = None

source

quantization_encoding​

quantization_encoding: SupportedEncoding | None = None

source

return_hidden_states​

return_hidden_states: ReturnHiddenStates = 'none'

source

return_logits​

return_logits: ReturnLogits = 'last_token'

source

rms_norm_eps​

rms_norm_eps: float = 1e-06

source

rope_interleave​

rope_interleave: bool = True

source

rope_scaling​

rope_scaling: dict[str, Any] | None = None

source

rope_theta​

rope_theta: float = 10000.0

source

routed_experts_native​

property routed_experts_native: bool

source

Whether the routed experts run on the W4A8 grouped kernel.

routed_scaling_factor​

routed_scaling_factor: float = 1.5

source

scoring_func​

scoring_func: str = 'sqrtsoftplus'

source

sliding_window​

sliding_window: int = 128

source

swiglu_limit​

swiglu_limit: float = 10.0

source

tie_word_embeddings​

tie_word_embeddings: bool = False

source

use_subgraphs​

use_subgraphs: bool = True

source

vocab_size​

vocab_size: int = 129280

source

window_ratios​

property window_ratios: list[int]

source

The compression ratios the trunk’s attention layers use.

DeepseekV4Model​

class max.pipelines.architectures.deepseekV4.DeepseekV4Model(pipeline_config, session, devices, kv_cache_config, weights, *, memory_plan, adapter=None, return_logits=ReturnLogits.LAST_TOKEN, max_batch_size=1)

source

Bases: GraphPipelineModelWithKVCache[TextContext, Any]

A DeepSeek-V4-Flash pipeline model for text generation.

Parameters:

batch_processor_cls​

batch_processor_cls

source

alias of DeepseekV4BatchProcessor

execute()​

execute(model_inputs)

source

Executes the graph with the given inputs.

Parameters:

model_inputs (ModelInputs) – The model inputs to execute, containing tensors and any other required data for model execution.

Returns:

ModelOutputs containing the pipeline’s output tensors.

Return type:

ModelOutputs

This is an abstract method that must be implemented by concrete PipelineModels to define their specific execution logic.

graph_specs()​

graph_specs(model_config)

source

Returns the decoder graph, tensor parallel on several devices.

Parameters:

model_config (DeepseekV4Config)

Return type:

dict[str, GraphSpec]

model​

property model: Model

source

The compiled and initialized MAX Engine model.

model_config_cls​

model_config_cls

source

alias of DeepseekV4Config

DeepseekV4ReasoningParser​

class max.pipelines.architectures.deepseekV4.DeepseekV4ReasoningParser(think_start_token_id, think_end_token_id)

source

Bases: ReasoningParser

DeepSeek-V4 reasoning parser for spans framed by <think> and </think>.

In thinking mode the prompt ends with a prefilled <think>, so reasoning begins implicitly; in chat mode it ends with </think>. Only </think> ends reasoning: the encoder’s tool instructions require the complete reasoning block before any <｜DSML｜tool_calls> block.

Parameters:

  • think_start_token_id (int)
  • think_end_token_id (int)

REASONING_END​

REASONING_END: ClassVar[str] = '</think>'

source

Text delimiter that closes a reasoning span, such as "</think>".

See REASONING_START for the declaration contract.

REASONING_START​

REASONING_START: ClassVar[str] = '<think>'

source

Text delimiter that opens a reasoning span, such as "<think>".

Subclasses declare their delimiters here and resolve token ids from them, so each model’s delimiters are written down exactly once. Consumers that work in the text domain rather than the token domain read them from here instead of restating them.

Declare both delimiters or neither. None means this parser has no text form at all, so a text-domain consumer cannot bound a reasoning span and will leave reasoning in the assistant’s content; declaring only one is a bug, since a span needs both ends. A parser whose chat template prefills the opening delimiter still declares it – whether a given turn emits it is a property of the request, not of the parser.

from_tokenizer()​

async classmethod from_tokenizer(tokenizer)

source

Constructs a reasoning parser from a tokenizer.

Parameters:

tokenizer (PipelineTokenizer[Any, Any, Any])

Return type:

DeepseekV4ReasoningParser

reasoning_end_token_id()​

async classmethod reasoning_end_token_id(tokenizer)

source

Returns the </think> token id.

Parameters:

tokenizer (PipelineTokenizer[Any, Any, Any])

Return type:

int | None

stream()​

stream(delta_token_ids, is_currently_reasoning=True)

source

Identifies a reasoning span within a streaming delta chunk.

When is_currently_reasoning=False and the chunk contains no <think> opener, returns an empty span so chat-mode output and chunks after reasoning ended aren’t misclassified as reasoning.

Parameters:

Return type:

ParsedReasoningDelta

will_reason_after_prompt()​

will_reason_after_prompt(prompt_token_ids)

source

Predicts whether the model will emit reasoning after this prompt.

Scans right-to-left: earlier turns and the tool instructions also carry <think>/</think>, but the last delimiter is the one the encoder prefilled after <｜Assistant｜>.

Parameters:

prompt_token_ids (Sequence[int])

Return type:

bool

DeepseekV4Tokenizer​

class max.pipelines.architectures.deepseekV4.DeepseekV4Tokenizer(model_path, pipeline_config, **kwargs)

source

Bases: ReasoningTextTokenizer

TextTokenizer for DeepSeek-V4, whose checkpoint ships no chat template.

Chat prompts are rendered by the checkpoint’s reference encoder (encoding_dsv4.py) unless --chat-template is given. Implements ReasoningPipelineTokenizer.

Parameters:

apply_chat_template()​

apply_chat_template(messages, tools, **chat_template_options)

source

Renders messages with the DeepSeek-V4 encoder (see render_prompt()).

Parameters:

Return type:

str

DeepseekV4ToolParser​

class max.pipelines.architectures.deepseekV4.DeepseekV4ToolParser

source

Bases: StructuralTagToolParser

Parses DeepSeek-V4 <｜DSML｜tool_calls> blocks into tool calls.

The router hands this parser content only: the deepseekv4 reasoning parser has already moved everything up to </think> into reasoning. A DSML block inside the reasoning is therefore never a tool call. The reference decoder rejects such output outright; here it stays verbatim in the reasoning.

The section marker includes the \n\n the reference decoder requires before <｜DSML｜tool_calls>, so the content ahead of a call streams without it, as it parses. Arguments stream one complete parameter at a time; string="true|false" carries each value’s type, so no schema is needed.

CALL_BEGIN​

CALL_BEGIN: ClassVar[str] = '<｜DSML｜invoke'

source

CALL_END​

CALL_END: ClassVar[str] = '</｜DSML｜invoke>'

source

SECTION_BEGIN​

SECTION_BEGIN: ClassVar[str] = '\n\n<｜DSML｜tool_calls>'

source

SECTION_END​

SECTION_END: ClassVar[str] = '</｜DSML｜tool_calls>'

source

parse_complete()​

parse_complete(response)

source

Parses a complete response with the reference decoder.

Output without any ｜DSML｜ token is returned as content, so plain replies never meet the decoder’s special-token checks. Otherwise the reference decoder decides, in chat mode because reasoning was split off upstream; the response has its EOS stripped by detokenization, so it is restored first.

Parameters:

response (str) – The content the model generated after its reasoning.

Returns:

The content before the tool-call block (None when empty) and the parsed tool calls.

Raises:

ValueError – If the DSML is malformed or holds no tool call.

Return type:

ParsedToolResponse

Models

Cookbook

Puzzles

Blog

Community