For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python class
BatchPipeline
BatchPipeline
class max.pipelines.lib.generate.BatchPipeline(*args, **kwargs)
Bases: Protocol
The pipeline surface GenerateIterator drives.
TextGenerationPipeline and
OverlapTextGenerationPipeline over
TextContext satisfy it. A pipeline whose
execute returns the previous call’s results also defines a boolean
overlap_active property, read the way serve’s schedulers read it. A
pipeline that can replay captured device graphs defines a boolean
graph_capture_ready property, true once its graphs are captured.
GenerateIterator also reads the memory plan’s max length, batch token
limits and LoRA adapters from the private _pipeline_model, the only
place they live, so another implementation must define it as well.
execute()
execute(inputs)
Runs one model step over inputs.
-
Parameters:
-
inputs (TextGenerationInputs[TextContext])
-
Return type:
kv_manager
property kv_manager: _KVCacheClaims
The KV cache manager that owns each row’s cache blocks.
max_batch_size
property max_batch_size: int
The planned maximum batch size per data-parallel replica.
pipeline_config
property pipeline_config: PipelineConfig
The config supplying the DP degree and sampling defaults.
release()
release(request_id)
Releases the pipeline’s per-request state for request_id.
-
Parameters:
-
request_id (RequestID)
-
Return type:
-
None
tokenizer
property tokenizer: _EOSTokenSource
The tokenizer supplying the model’s EOS token ids.