IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

ModuleV3PipelineModelWithKVCache

ModuleV3PipelineModelWithKVCache​

class max.pipelines.lib.ModuleV3PipelineModelWithKVCache(pipeline_config, session, devices, kv_cache_config, weights, *, memory_plan, adapter=None, return_logits=ReturnLogits.LAST_TOKEN, return_hidden_states=ReturnHiddenStates.NONE, max_batch_size=1)

source

Bases: PipelineModelWithKVCache[BaseContextType, _ModelConfigT, Module[…, Any]]

The base class for a ModuleV3 model architecture that uses a KV cache.

A subclass implements _create_model_config(), which builds the architecture’s config, and _instantiate_module(), which constructs the root module and places it on a device or device mesh. graph_specs() builds the module under F.lazy() and registers it, with the adapted checkpoint weights, as the decoder. A subclass can also override _prepare_state_dict(), _init_distributed_runtime(), _module_default_dtype(), or _get_compile_input_types().

A vision-language model that compiles several modules subclasses ModuleV3MultiGraphPipelineModelWithKVCache.

The constructor compiles the decoder through load_graphs() on the pipeline’s session into model, and the default execute() passes it model_inputs.buffers.

Parameters:

  • pipeline_config (PipelineConfig) – The pipeline configuration, including the model path and its Hugging Face config.
  • session (InferenceSession) – The inference session the pipeline runs in.
  • devices (list[Device]) – The devices to run the model on.
  • kv_cache_config (KVCacheConfig) – The KV cache configuration.
  • weights (Weights) – The checkpoint weights to load.
  • memory_plan (MemoryPlan) – The memory plan that sizes the KV cache.
  • adapter (WeightsAdapter | None) – The weight adapter that renames checkpoint keys to the root module’s parameter names. Defaults to None, which loads the checkpoint keys unchanged.
  • return_logits (ReturnLogits) – Which logits the model returns. Defaults to ReturnLogits.LAST_TOKEN.
  • return_hidden_states (ReturnHiddenStates) – Which hidden states the model returns. Defaults to ReturnHiddenStates.NONE.
  • max_batch_size (int) – The maximum number of requests in one batch. Defaults to 1.

Raises:

ValueError – If the pipeline config enables LoRA while the KV cache config enables prefix caching, or enables LoRA for a subclass that doesn’t set lora_modulev3.

checkpoint_weights()​

classmethod checkpoint_weights(context, config)

source

Returns the weights build_module()’s module compiles with.

They come from the checkpoint context names, laid out as Module.compile loads them.

Raises:

NotImplementedError – If the architecture builds its module only inside its own graph.

Parameters:

  • context (ModelLoadContext)
  • config (Any)

Return type:

dict[str, Any]

execute()​

execute(model_inputs)

source

Runs model on model_inputs.buffers.

Parameters:

model_inputs (ModelInputs) – The prepared inputs, whose buffers match the compiled input order.

Returns:

The outputs mapped by _to_model_outputs().

Return type:

ModelOutputs

graph_specs()​

graph_specs(model_config)

source

Returns the decoder, built from the checkpoint weights.

Prepares the weights, initializes the distributed runtime, and constructs the root module under F.lazy(), wrapping it for LoRA when serving adapters.

Parameters:

model_config (_ModelConfigT) – The config _create_model_config() returned.

Return type:

dict[str, GraphSpec]

model​

model: Callable[..., ModuleV3Outputs]

source