For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

GraphApiDecoderPipelineModel

GraphApiDecoderPipelineModel

class max.pipelines.lib.registration_graph_api.GraphApiDecoderPipelineModel(pipeline_config, session, devices, kv_cache_config, weights, adapter=None, return_logits=ReturnLogits.LAST_TOKEN, *, memory_plan, return_hidden_states=ReturnHiddenStates.NONE, max_batch_size=1)

source

Bases: LogProbabilitiesMixin, PipelineModelWithKVCache[TextContext, Any, Model]

Serves a registered graph-API DECODER module.

graph_api_pipeline_model() subclasses it for each registration. The decoder is constructed as decoder_cls(config) from the finalized config, completed with the deployment and expert-parallel runtime before construction.

Builds, compiles and initializes the decoder graph.

Raises:

ValueError – If data parallelism is on and the decoder doesn’t support it, or doesn’t give each device its own replica.

Parameters:

• pipeline_config (PipelineConfig)
• session (InferenceSession)
• devices (list[Device])
• kv_cache_config (KVCacheConfig)
• weights (Weights)
• adapter (WeightsAdapter | None)
• return_logits (ReturnLogits)
• memory_plan (MemoryPlan)
• return_hidden_states (ReturnHiddenStates)
• max_batch_size (int)

batch_processor_cls

batch_processor_cls

source

alias of GraphApiDecoderBatchProcessor

decoder_cls

decoder_cls: ClassVar[type[GraphApiModule]]

source

The registered DECODER class.

ep_comm_initializer

ep_comm_initializer: ExpertParallelExchange | None = None

source

execute()

execute(model_inputs)

source

Runs the decoder on the prepared batch.

Parameters:

model_inputs (ModelInputs)

Return type:

ModelOutputs

graph_specs()

graph_specs(model_config)

source

Returns the decoder graph, with the checkpoint loaded into it.

Parameters:

model_config (Any)

Return type:

dict[str, GraphSpec]

load_graphs()

load_graphs(session)

source

Loads the graphs with the session used to initialize EP first.

Parameters:

session (InferenceSession)

Return type:

dict[str, Model]

model

property model: Model

source

The compiled and initialized decoder.