For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python class
ModuleV3PipelineModelWithKVCache
ModuleV3PipelineModelWithKVCache
class max.pipelines.lib.ModuleV3PipelineModelWithKVCache(pipeline_config, session, devices, kv_cache_config, weights, *, memory_plan, adapter=None, return_logits=ReturnLogits.LAST_TOKEN, return_hidden_states=ReturnHiddenStates.NONE, max_batch_size=1)
Bases: PipelineModelWithKVCache[BaseContextType, _ModelConfigT, Module[…, Any]]
The base class for a ModuleV3 model architecture that uses a KV cache.
A subclass implements _create_model_config(), which builds the
architecture’s config, and _instantiate_module(), which constructs the
root module and places it on a device or device mesh. graph_specs()
builds the module under F.lazy() and registers it, with the adapted
checkpoint weights, as the decoder. A subclass can also override
_prepare_state_dict(), _init_distributed_runtime(),
_module_default_dtype(), or _get_compile_input_types().
A vision-language model that compiles several modules subclasses
ModuleV3MultiGraphPipelineModelWithKVCache.
The constructor compiles the decoder through load_graphs() on the
pipeline’s session into model, and the default execute()
passes it model_inputs.buffers.
-
Parameters:
-
- pipeline_config (PipelineConfig) – The pipeline configuration, including the model path and its Hugging Face config.
- session (InferenceSession) – The inference session the pipeline runs in.
- devices (list[Device]) – The devices to run the model on.
- kv_cache_config (KVCacheConfig) – The KV cache configuration.
- weights (Weights) – The checkpoint weights to load.
- memory_plan (MemoryPlan) – The memory plan that sizes the KV cache.
- adapter (WeightsAdapter | None) – The weight adapter that renames checkpoint keys to the root
module’s parameter names. Defaults to
None, which loads the checkpoint keys unchanged. - return_logits (ReturnLogits) – Which logits the model returns. Defaults to
ReturnLogits.LAST_TOKEN. - return_hidden_states (ReturnHiddenStates) – Which hidden states the model returns. Defaults
to
ReturnHiddenStates.NONE. - max_batch_size (int) – The maximum number of requests in one batch. Defaults
to
1.
-
Raises:
-
ValueError – If the pipeline config enables LoRA while the KV cache config enables prefix caching, or enables LoRA for a subclass that doesn’t set
lora_modulev3.
checkpoint_weights()
classmethod checkpoint_weights(context, config)
Returns the weights build_module()’s module compiles with.
They come from the checkpoint context names, laid out as
Module.compile loads them.
-
Raises:
-
NotImplementedError – If the architecture builds its module only inside its own graph.
-
Parameters:
-
- context (ModelLoadContext)
- config (Any)
-
Return type:
execute()
execute(model_inputs)
Runs model on model_inputs.buffers.
-
Parameters:
-
model_inputs (ModelInputs) – The prepared inputs, whose
buffersmatch the compiled input order. -
Returns:
-
The outputs mapped by
_to_model_outputs(). -
Return type:
graph_specs()
graph_specs(model_config)
Returns the decoder, built from the checkpoint weights.
Prepares the weights, initializes the distributed runtime, and
constructs the root module under F.lazy(), wrapping it for LoRA
when serving adapters.
model
model: Callable[..., ModuleV3Outputs]