For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

TextGenerationInputs

TextGenerationInputs

class max.pipelines.modeling.types.pipeline_variants.TextGenerationInputs(batches, input_tokens=-1, batch_type=BatchType.TG, per_replica_input_tokens=<factory>, per_replica_context_tokens=<factory>)

source

Bases: PipelineInputs, Generic[TextGenerationContextType]

Input parameters for text generation pipeline operations.

Parameters:

• batches (list[list[TextGenerationContextType]])
• input_tokens (int)
• batch_type (BatchType)
• per_replica_input_tokens (list[int])
• per_replica_context_tokens (list[int])

batch_echo

property batch_echo: list[bool]

source

List indicating whether echo is enabled for each context in the batch.

batch_size

property batch_size: int

source

Number of requests in the batch.

batch_top_log_probs

property batch_top_log_probs: list[int]

source

List of requested top log probabilities per context in the batch.

batch_type

batch_type: BatchType = 'TG'

source

Type of batch.

batches

batches: list[list[TextGenerationContextType]]

source

Variable list of batches, with each batch being a list of contexts.

There can be multiple batches when using data parallelism, in which each batch is mapped to a different device replica.

enable_echo

property enable_echo: bool

source

True if any context in the batch has echo enabled.

enable_log_probs

property enable_log_probs: bool

source

True if any context in the batch requests log probabilities.

flat_batch

property flat_batch: list[TextGenerationContextType]

source

Flattened list of contexts across all replicas.

input_tokens

input_tokens: int = -1

source

Number of input tokens.

per_replica_context_tokens

per_replica_context_tokens: list[int]

source

Per-replica processed-token (context) sums, excluding DP padding dummies. Frozen at construction like per_replica_input_tokens.

per_replica_input_tokens

per_replica_input_tokens: list[int]

source

Per-replica active-token sums, excluding DP padding dummies. Frozen at construction: token windows mutate during scheduling, so later reads of active_length no longer describe this batch.