For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).
Python class
TextGenerationInputs
TextGenerationInputs
class max.pipelines.modeling.types.TextGenerationInputs(batches, input_tokens=-1, batch_type=BatchType.TG, per_replica_input_tokens=<factory>, per_replica_context_tokens=<factory>)
Bases: PipelineInputs, Generic[TextGenerationContextType]
Input parameters for text generation pipeline operations.
-
Parameters:
batch_echo
List indicating whether echo is enabled for each context in the batch.
batch_top_log_probs
List of requested top log probabilities per context in the batch.
batch_type
batch_type: BatchType = 'TG'
Type of batch.
batches
Variable list of batches, with each batch being a list of contexts.
There can be multiple batches when using data parallelism, in which each batch is mapped to a different device replica.
enable_echo
property enable_echo: bool
True if any context in the batch has echo enabled.
enable_log_probs
property enable_log_probs: bool
True if any context in the batch requests log probabilities.
flat_batch
property flat_batch: list[TextGenerationContextType]
Flattened list of contexts across all replicas.
input_tokens
input_tokens: int = -1
Number of input tokens.
per_replica_context_tokens
Per-replica processed-token (context) sums, excluding DP padding
dummies. Frozen at construction like per_replica_input_tokens.
per_replica_input_tokens
Per-replica active-token sums, excluding DP padding dummies. Frozen at
construction: token windows mutate during scheduling, so later reads of
active_length no longer describe this batch.
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!