IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python module

max.pipelines.architectures.inkling

InklingConfig​

class max.pipelines.architectures.inkling.InklingConfig(*, devices, dtype, kv_params, max_seq_len, text_config, vision_config, quant_config=None, use_subgraphs=True, mtp=None)

source

Bases: ArchConfigWithKVCache

Top-level Inkling config wrapping the text and vision backbones.

Parameters:

DEFAULT_ENCODING​

DEFAULT_ENCODING: ClassVar[max.pipelines.lib.config.SupportedEncoding] = 'bfloat16'

source

SUPPORTED_ENCODINGS​

SUPPORTED_ENCODINGS: ClassVar[set[max.pipelines.lib.config.SupportedEncoding]] = {'bfloat16', 'float4_e2m1fnx2'}

source

calculate_max_seq_len()​

static calculate_max_seq_len(huggingface_config, model_config)

source

Returns the resolved maximum sequence length.

Bounds or defaults the user’s model_config.max_length with the model’s own limits. Construction runs this once and stores the result on model_config.max_length; memory planning may lower it further, but only on the memory plan.

Parameters:

  • huggingface_config (AutoConfig) – The HuggingFace config to read model bounds from.
  • model_config (MAXModelConfig) – The model config whose max_length carries the user’s setting.

Return type:

int

construct_kv_params()​

static construct_kv_params(huggingface_config, pipeline_config, devices, kv_cache_config, cache_dtype)

source

One cache per attention flavor; they differ in KV head count.

Parameters:

Return type:

MultiKVCacheParams

devices​

devices: list[DeviceRef]

source

dtype​

dtype: DType

source

finalize()​

finalize(huggingface_config, state_dict)

source

Sets quant_config from which routed tensors ship block scales; the released NVFP4 checkpoint leaves one MoE layer bfloat16.

Parameters:

Return type:

None

get_kv_params()​

get_kv_params()

source

KV cache parameters to use when running the model.

Return type:

MultiKVCacheParams

get_max_seq_len()​

get_max_seq_len()

source

Returns the effective maximum sequence length for the model.

For configs that store a deployment length, this is the value initialize received; for metadata-only configs it derives from the checkpoint.

Return type:

int

initialize()​

classmethod initialize(pipeline_config, model_config=None, *, max_seq_len)

source

Initialize the config from a PipelineConfig.

Parameters:

  • pipeline_config (PipelineConfig) – The pipeline configuration.
  • model_config (MAXModelConfig | None) – The model configuration to read from. When None (the default), pipeline_config.model is used. Pass an explicit config (e.g. pipeline_config.draft_model) to initialize the arch config for a different model.
  • max_seq_len (int) – The effective maximum sequence length to store on the config. The value is received, never derived here: the pipeline model passes the memory plan’s VRAM-clamped length, while memory planning (which runs before a plan exists) passes the construction-resolved model_config.max_length. Configs whose sequence length is pure model metadata (e.g. diffusion components) ignore it.

Return type:

Self

kv_params​

kv_params: MultiKVCacheParams

source

max_seq_len​

max_seq_len: int

source

mtp​

mtp: InklingMTPConfig | None = None

source

Set on the unified MTP path; the text-only architecture leaves it None.

quant_config​

quant_config: QuantConfig | None = None

source

Set by finalize() when the routed experts are packed FP4.

text_config​

text_config: InklingTextConfig

source

use_subgraphs​

use_subgraphs: bool = True

source

vision_config​

vision_config: InklingVisionConfig

source

InklingInputs​

class max.pipelines.architectures.inkling.InklingInputs(tokens, input_row_offsets, positions, return_n_logits, image_embeddings, image_indices, signal_buffers, slot_idx, has_initial_state, conv_pools, *, kv_cache_inputs=None, lora_buffers=(), vision_embeddings=<factory>, vision_scatter_indices=<factory>, hidden_states=None)

source

Bases: ModelInputs

Ragged token inputs plus the convolution-state pool addressing.

Parameters:

buffers​

property buffers: tuple[Buffer, ...]

source

Returns positional Buffer inputs for model ABI calls.

conv_pools​

conv_pools: list[Buffer]

source

Per device, one pool per convolution site per layer, mutated in place.

has_initial_state​

has_initial_state: list[Buffer]

source

Per device, whether each request has convolution history to read.

image_embeddings​

image_embeddings: Buffer

source

image_indices​

image_indices: Buffer

source

Token-stream row each vision row replaces; negative entries are skipped.

input_row_offsets​

input_row_offsets: Buffer

source

positions​

positions: Buffer

source

return_n_logits​

return_n_logits: Buffer

source

signal_buffers​

signal_buffers: list[Buffer]

source

slot_idx​

slot_idx: list[Buffer]

source

tokens​

tokens: Buffer

source

InklingModel​

class max.pipelines.architectures.inkling.InklingModel(pipeline_config, session, devices, kv_cache_config, weights, *, memory_plan, adapter=None, return_logits=ReturnLogits.LAST_TOKEN, return_hidden_states=ReturnHiddenStates.NONE, max_batch_size=1)

source

Bases: LogProbabilitiesMixin, MultiGraphPipelineModelWithKVCache[TextAndVisionContext], SupportsSSMStateWarmup

Pipeline model for Inkling’s decoder and its vision tower.

Parameters:

batch_processor_cls​

batch_processor_cls

source

alias of InklingBatchProcessor

execute()​

execute(model_inputs)

source

Executes the graph with the given inputs.

Parameters:

model_inputs (ModelInputs) – The model inputs to execute, containing tensors and any other required data for model execution.

Returns:

ModelOutputs containing the pipeline’s output tensors.

Return type:

ModelOutputs

This is an abstract method that must be implemented by concrete PipelineModels to define their specific execution logic.

model​

model: Model

source

model_config_cls​

model_config_cls

source

alias of InklingConfig

release()​

release(request_id)

source

Drops the request’s convolution state, freeing its slot.

Parameters:

request_id (RequestID)

Return type:

None

release_warmup_state()​

release_warmup_state(request_ids)

source

Frees the slots a graph-capture warmup probe claimed.

Without this the second probe finds no free slot and serving never starts.

Parameters:

request_ids (list[RequestID])

Return type:

None

InklingReasoningParser​

class max.pipelines.architectures.inkling.InklingReasoningParser(thinking_start_token_id, end_message_token_id, tool_call_start_token_id=None)

source

Bases: ReasoningParser

Inkling reasoning parser for <|content_thinking|>...<|end_message|>.

Inkling frames every message as <role_token>[name]<content_type_token>...<|end_message|>. The generation prompt ends at <|message_model|> with no content-type marker, so the model picks the content type as its first generated token and reasoning opens only on an explicit <|content_thinking|>. There is no implicit start.

<|end_message|> closes reasoning and is consumed. It terminates every Inkling message type, so it is honored only while a span is open. A <|content_invoke_tool_json|> opener also ends reasoning, left unconsumed so the tool parser receives it. <|content_model_end_sampling|> terminates the whole assistant turn and acts as a stop token rather than a delimiter.

Parameters:

  • thinking_start_token_id (int)
  • end_message_token_id (int)
  • tool_call_start_token_id (int | None)

REASONING_END​

REASONING_END: ClassVar[str] = '<|end_message|>'

source

Text delimiter that closes a reasoning span (e.g. "</think>").

See REASONING_START for the declaration contract.

REASONING_START​

REASONING_START: ClassVar[str] = '<|content_thinking|>'

source

Text delimiter that opens a reasoning span (e.g. "<think>").

Subclasses declare their delimiters here and resolve token ids from them, so each model’s delimiters are written down exactly once. Consumers that work in the text domain rather than the token domain read them from here instead of restating them.

Declare both delimiters or neither. None means this parser has no text form at all, so a text-domain consumer cannot bound a reasoning span and will leave reasoning in the assistant’s content; declaring only one is a bug, since a span needs both ends. A parser whose chat template prefills the opening delimiter still declares it – whether a given turn emits it is a property of the request, not of the parser.

from_tokenizer()​

async classmethod from_tokenizer(tokenizer)

source

Constructs a reasoning parser from a tokenizer.

Parameters:

tokenizer (PipelineTokenizer[Any, Any, Any])

Return type:

InklingReasoningParser

reasoning_end_token_id()​

async classmethod reasoning_end_token_id(tokenizer)

source

Returns the <|end_message|> token id that closes reasoning.

Parameters:

tokenizer (PipelineTokenizer[Any, Any, Any])

Return type:

int | None

stream()​

stream(delta_token_ids, is_currently_reasoning=True)

source

Identifies a reasoning span within a streaming delta chunk.

When is_currently_reasoning=False and the chunk contains no <|content_thinking|> opener, returns an empty span so the whole chunk routes to content: Inkling prefills no thinking marker, so a turn that answers directly is not reasoning.

Parameters:

Return type:

ParsedReasoningDelta

will_reason_after_prompt()​

will_reason_after_prompt(prompt_token_ids)

source

Predicts whether generation starts inside a reasoning span.

Inkling’s generation prompt ends at <|message_model|> with no content-type marker, so a delimiter-free prompt is False: the model, not the template, decides whether to think. A True suspends grammar enforcement for constrained decoding until <|end_message|> fires, which never comes if the model answers directly.

The scan runs right-to-left because the most recent delimiter decides, and it covers the one reachable True: a caller prefilling a partial assistant turn whose tail is an open <|content_thinking|> block.

Parameters:

prompt_token_ids (Sequence[int])

Return type:

bool

InklingTokenizer​

class max.pipelines.architectures.inkling.InklingTokenizer(model_path, pipeline_config, *, revision=None, max_length=None, trust_remote_code=False, **unused_kwargs)

source

Bases: TextAndVisionTokenizer

Tokenizer for Inkling, whose own AutoProcessor class is not shipped with the checkpoint, so InklingProcessor takes its place.

Also exposes Inkling’s reasoning-delimiter ids, satisfying the ReasoningPipelineTokenizer protocol that OverlapTextGenerationPipeline requires of any architecture that names a reasoning parser.

skipped_special_token_ids opts the streaming detokenizer into a per-id filter so <|content_invoke_tool_json|> survives decode for the tool parser. Only that marker: a request without tools runs no tool parser, so anything else kept here leaks verbatim into message.content.

Parameters:

decode()​

async decode(encoded, **kwargs)

source

Decodes tokens, dropping every special id except the tool-call marker.

skip_special_tokens=True would drop the marker too, so filter by id here and decode with the flag off.

Parameters:

encoded (ndarray[tuple[Any, ...], dtype[integer[Any]]] | int)

Return type:

str

reasoning_end_token_id​

property reasoning_end_token_id: int

source

Token id of <|end_message|>.

reasoning_start_token_id​

property reasoning_start_token_id: int

source

Token id of <|content_thinking|>.

InklingToolParser​

class max.pipelines.architectures.inkling.InklingToolParser

source

Bases: StructuralTagToolParser

Parses Inkling tool calls, which reach this parser detokenized as NAME<|content_invoke_tool_json|>{"name":...,"args":{...}}.

Two hooks depart from the base class: bodies are bounded by brace balancing because there is no visible closing marker, and content deltas must drop the trailing function name, which precedes CALL_BEGIN.

CALL_BEGIN​

CALL_BEGIN: ClassVar[str] = '<|content_invoke_tool_json|>'

source

XGRAMMAR_FORMAT​

XGRAMMAR_FORMAT: ClassVar[str] = 'inkling'

source

generate_tool_call_grammar()​

static generate_tool_call_grammar(response_format_schema=None, tools=None, tokenizer=None, backend='xgrammar', tool_choice=None, **kwargs)

source

Builds the decode-time grammar that constrains tool calls.

Parameters:

Return type:

str

parse_complete()​

parse_complete(response)

source

Parses a complete response into tool calls.

In section-wrapped mode, walks every SECTION_BEGIN … SECTION_END pair. In flat mode (no section markers), passes everything from the first CALL_BEGIN onward to _parse_complete_section(). Content before the first marker is preserved.

Parameters:

response (str)

Return type:

ParsedToolResponse

reset()​

reset()

source

Resets internal state for a new streaming session.

Return type:

None

set_streaming_tool_schemas()​

set_streaming_tool_schemas(schemas)

source

Records declared names for the streaming holdback, longest first.

The router omits tools without parameters, so their names surface once in streamed content. Cosmetic; non-streaming is unaffected.

Parameters:

schemas (Mapping[str, dict[str, Any]])

Return type:

None

Models

Cookbook

Puzzles

Blog

Community

Contact