IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

TextGenerationRequest

TextGenerationRequest​

class max.pipelines.modeling.types.TextGenerationRequest(request_id, model_name, prompt=None, messages=<factory>, images=<factory>, videos=<factory>, decoded_images=<factory>, max_decoded_bytes=None, tools=None, response_format=None, timestamp_ns=0, request_path='/', logprobs=0, echo=False, chat_template_options=None, sampling_params=<factory>, target_endpoint=None, dkv_cache_hint=None, cache_salt=None)

source

Bases: object

An immutable request for text token generation from a pipeline.

Parameters:

cache_salt​

cache_salt: str | None = None

source

Optional per-request salt that isolates this prompt’s prefix-cache entries from other requests sharing the same tokens.

Combined with kv_cache_hash_seed via XOR to seed the block hash. Works under any kv_cache_hash_algo: a cryptographic guarantee under sha256 (64-bit key separation on the dKV external tier), best-effort under ahash64. Capped at 512 chars at the OpenAI schema layer.

chat_template_options​

chat_template_options: dict[str, Any] | None = None

source

Optional dictionary of options to pass when applying the chat template.

decoded_images​

decoded_images: list[PILImage | None]

source

Decoded PIL.Image objects corresponding 1: 1 to images, decoded once at request admission (the API server validates images by fully decoding them, so the decoded result is carried here to avoid a second decode in the tokenizer). API-process-only: this is never serialized across the worker boundary, so it must stay populated only for the in-process tokenization step. Empty when images were not pre-decoded (offline/test callers); tokenizers fall back to decoding images in that case. An individual entry is None when the admission decode was skipped because the tokenizer already holds that image’s preprocessed tensor – the same per-index fallback to images applies. generate() also fills it, with each PIL image its caller passed and None for each image passed as bytes.

dkv_cache_hint​

dkv_cache_hint: dict[str, Any] | None = None

source

Cache hint from the Orchestrator for distributed KV cache.

The serving layer never reads it: it re-serializes the object onto TextContext.dkv_cache_hint and hands those bytes to the dKV connector, which parses them in Rust to route each block to the instance that holds it. See dkv/docs/cache-hint.md.

echo​

echo: bool = False

source

If set to True, the response will include the original prompt along with the generated output. This can be useful for debugging or when you want to see how the input relates to the output.

images​

images: list[bytes]

source

A list of image byte arrays that can be included as part of the request. This field is optional and may be used for multimodal inputs where images are relevant to the prompt or task.

images_for_processing()​

images_for_processing()

source

Return the images for tokenizer preprocessing, decoded once.

Prefers the pre-decoded decoded_images (decoded and validated once at the API server) and falls back to the raw images bytes for offline and test callers. Tokenizers consume images through this so the decode-once policy lives in one place rather than being repeated at every per-model decode site.

The fallback is per index, not per request: the API server skips the admission decode for an image whose preprocessed tensor the tokenizer already holds, leaving a None in that slot. Such an image is only decoded if the tokenizer actually needs its pixels, which a cache hit means it does not. A None also marks an image that a generate() caller passed as bytes.

Return type:

list[bytes | PILImage]

logprobs​

logprobs: int = 0

source

The number of top log probabilities to return for each generated token. A value of 0 means that log probabilities will not be returned. Useful for analyzing model confidence in its predictions.

max_decoded_bytes​

max_decoded_bytes: int | None = None

source

Decoded (in-memory) bytes each of this request’s videos may retain: an equal share of the Settings.max_media_bytes budget left after every encoded image and video and every decoded image is accounted for. A video is decoded later, in the tokenizer, so the allowance travels with the request to bound how many frames that decode may keep. Because the shares sum to what remains, the videos cannot jointly exceed the budget. API-process-only, like decoded_images. None means unbounded (offline/test callers, or the limit disabled).

messages​

messages: list[TextGenerationRequestMessage]

source

A list of messages for chat-based interactions. This is used in chat completion APIs, where each message represents a turn in the conversation. If provided, the model will generate responses based on these messages.

model_name​

model_name: str

source

The name of the model to be used for generating tokens. This should match the available models on the server and determines the behavior and capabilities of the response generation.

number_of_images​

property number_of_images: int

source

Returns the total number of image-type contents across all provided messages.

Returns:

Total count of image-type contents found in messages.

number_of_videos​

property number_of_videos: int

source

Returns the total number of video-type contents across all provided messages.

Returns:

Total count of video-type contents found in messages.

prompt​

prompt: str | Sequence[int] | None = None

source

The prompt to be processed by the model. This field supports legacy completion APIs and can accept either a string or a sequence of integers representing token IDs. If not provided, the model may generate output based on the messages field.

request_id​

request_id: RequestID

source

A unique identifier for the request.

request_path​

request_path: str = '/'

source

The endpoint path for the request. This is typically used for routing and logging requests within the server infrastructure.

response_format​

response_format: TextGenerationResponseFormat | None = None

source

Specifies the desired format for the model’s output. When set, it enables structured generation, which adheres to the json_schema provided.

sampling_params​

sampling_params: SamplingParams

source

Token sampling configuration parameters for the request.

target_endpoint​

target_endpoint: str | None = None

source

Optional target endpoint identifier for routing the request to a specific service or model instance. This should be used in disaggregate serving scenarios, when you want to dynamically route to a specific instance. If not specified, the request will be routed to the default endpoint.

timestamp_ns​

timestamp_ns: int = 0

source

The time (in nanoseconds) when the request was received by the server. This can be useful for performance monitoring and logging purposes.

tools​

tools: list[TextGenerationRequestTool] | None = None

source

A list of tools that can be invoked during the generation process. This allows the model to utilize external functionalities or APIs to enhance its responses.

videos​

videos: list[bytes]

source

A list of video byte arrays that can be included as part of the request. Each video is decoded into frames during preprocessing.

Models

Cookbook

Puzzles

Blog

Community