IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

AutoModel

AutoModel​

class max.pipelines.AutoModel(model_path, *, max_batch_size=1, devices=None, device_graph_capture=False, **config_overrides)

source

Bases: object

Loads a text generation model in-process.

Unlike Hugging Face’s base-model AutoModel, this one generates. The Hugging Face equivalent is AutoModelForCausalLM. Serving features are tested through max serve.

The config turns off the overlap scheduler, device graph capture, prefix caching and chunked prefill. On AMD it also turns off the vendor BLAS fallback, as serve does with capture on, so matmuls compile to serve’s kernels. As in serve, the architecture’s required_arguments win over the overrides.

References to the pipeline, its KV cache manager, an open GenerateIterator, or a live traceback from inside the pipeline keep device memory alive past close().

Parameters:

  • model_path (str) – A Hugging Face repo ID or a local model directory.
  • max_batch_size (int) – The most prompts per data-parallel replica, fixed at construction. With device_graph_capture=True, warmup captures decode graphs for every batch size up to this value, so a larger value costs more warmup time and device memory.
  • devices (str | None) – The devices to load on, written as the CLI’s --devices takes them, such as "cpu", "gpu:0" or "gpu:0,1". Defaults to the first visible device.
  • device_graph_capture (bool) – Builds the overlap pipeline and runs its warmup_graph_capture(), so decode steps replay captured graphs. Steps stay synchronous unless enable_overlap_scheduler=True is also passed.
  • **config_overrides (object) – Flat PipelineArgs field names, such as device_specs or enable_prefix_caching. They win over the debug defaults.

Raises:

  • TypeError – If both devices and device_specs are passed, if an override passes a runtime, kv_cache or model section, or a config_file, instead of flat fields, or a model_override entry sets one of them whole.
  • ValueError – If the config needs a feature that only max serve runs, such as LoRA, if the model takes image inputs or does not generate text, or if device_graph_capture=True builds a pipeline that cannot capture graphs.

close()​

close()

source

Shuts down the KV cache manager and releases the pipeline.

Calling it again does nothing. The pipeline is released even if the shutdown raises.

Return type:

None

generate()​

generate(input_ids, attention_mask=None, *, max_new_tokens, min_new_tokens=None, temperature=None, top_k=None, top_p=None, min_p=None, frequency_penalty=None, presence_penalty=None, repetition_penalty=None, ignore_eos=False, stop_token_ids=None, seed=0, output_logits=False, output_logprobs=0)

source

Runs generate() on pipeline.

Takes the same arguments, so model.generate(**tokenizer(prompts), max_new_tokens=16) works with an AutoTokenizer.

Raises:

Parameters:

Return type:

GenerateResult

generate_iter()​

generate_iter(input_ids, attention_mask=None, *, max_new_tokens, min_new_tokens=None, temperature=None, top_k=None, top_p=None, min_p=None, frequency_penalty=None, presence_penalty=None, repetition_penalty=None, ignore_eos=False, stop_token_ids=None, seed=0, output_logits=False, output_logprobs=0)

source

Opens a GenerateIterator on pipeline.

Takes the same keyword arguments. Use it as a context manager so its rows are released.

Raises:

Parameters:

Return type:

GenerateIterator

pipeline​

property pipeline: TextGenerationPipelineInterface[TextContext]

source

The loaded pipeline.

Raises:

RuntimeError – If the model has been closed.

pipeline_config​

property pipeline_config: PipelineConfig

source

The resolved config that built the pipeline.

Models

Cookbook

Puzzles

Blog

Community