For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python function

generate

generate()

max.pipelines.lib.generate.generate(pipeline, input_ids, attention_mask=None, *, max_new_tokens, min_new_tokens=None, temperature=None, top_k=None, top_p=None, min_p=None, frequency_penalty=None, presence_penalty=None, repetition_penalty=None, ignore_eos=False, stop_token_ids=None, seed=0, output_logits=False, output_logprobs=0, caller='generate()')

source

Generates tokens for a fixed batch of prompts, run as one batch.

Steps a GenerateIterator until every row finishes. The arguments and their defaults are the same as for GenerateIterator, and the sampling arguments match the max generate flags of the same name.

Parameters:

• pipeline (BatchPipeline) – The pipeline to run.
• input_ids (Iterable[Sequence[int]]) – One token-id prompt per row, as lists or a 2D array.
• attention_mask (Iterable[Sequence[int]] | None) – Zeros mark padding in input_ids to drop.
• max_new_tokens (int) – The most tokens to generate for each row.
• min_new_tokens (int | None) – The fewest tokens a row generates before EOS can end it.
• temperature (float | None) – The sampling temperature.
• top_k (int | None) – The top-k limit.
• top_p (float | None) – The nucleus sampling threshold.
• min_p (float | None) – The minimum token probability relative to the most likely token.
• frequency_penalty (float | None) – The penalty proportional to a token’s count.
• presence_penalty (float | None) – The flat penalty for a token already present.
• repetition_penalty (float | None) – The divisor for logits of repeated tokens.
• ignore_eos (bool) – Keeps generating past EOS, so the batch size never changes.
• stop_token_ids (Sequence[int] | None) – Token ids that end a row, in addition to EOS.
• seed (int) – The sampling seed for row 0. Row i uses seed + i * 2**32.
• output_logits (bool) – Returns each row’s next-token logits, as GenerateIterator.logits.
• output_logprobs (int) – Returns the log probabilities of each token and of this many top candidates, as GenerateIterator.log_probs.
• caller (str) – What the user called, which errors name.

Returns:

Each row’s prompt followed by its new tokens, in input order, with the requested logits and log probabilities.

Raises:

• TypeError – If input_ids or attention_mask is not a list of integer lists.
• ValueError – If the request needs a feature GenerateIterator leaves to max serve, such as device graph capture with data_parallel_degree > 1 on a warmed-up pipeline, the batch does not fit the pipeline’s plan, or the pipeline cannot return the requested outputs, as described for GenerateIterator.
• RuntimeError – If the pipeline is not synchronous and another batch on it is still open, or if the pipeline returns log probabilities or logits rows that do not line up one per generated token, as when the model does not compute log probabilities.

Return type:

GenerateResult