For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).
Python class
PipelineTokenizer
PipelineTokenizer
class max.pipelines.modeling.types.PipelineTokenizer(*args, **kwargs)
Bases: Protocol[UnboundContextType, TokenizerEncoded, RequestType]
Interface for LLM tokenizers.
decode()
async decode(encoded, **kwargs)
Decodes response tokens to text.
-
Parameters:
-
- encoded (TokenizerEncoded) – Encoded response tokens.
- **kwargs – Additional decoder options (for example,
skip_special_tokens).
-
Returns:
-
Un-encoded response text.
-
Return type:
encode()
async encode(prompt, add_special_tokens)
Encodes text prompts as tokens.
-
Parameters:
-
Raises:
-
ValueError – If the prompt exceeds the configured maximum length.
-
Return type:
-
TokenizerEncoded
eos_token_ids
The full set of token ids that end generation for this model.
The tokenizer’s declared EOS plus any additional terminators the model ends its turn with (for example, chat turn-end tokens from the model’s generation config).
expects_content_wrapping
property expects_content_wrapping: bool
If True, this tokenizer expects messages to be wrapped as a dict.
Text messages are formatted as:
{
"role": "user",
"content": [{ "type": "text", "text": "text content" }]
}instead of:
{ "role": "user", "content": "text_content" }NOTE: Multimodal messages omit the content property.
Both image_urls and image content parts are converted to:
{ "type": "image" }Their content is provided as byte arrays through the top-level property
on the request object, that is, RequestType.images.
new_context()
async new_context(request)
Creates a new context from a request object.
This is sent to the worker process once and then cached locally.
-
Parameters:
-
request (RequestType) – Incoming request.
-
Returns:
-
Initialized context.
-
Return type:
-
UnboundContextType
Was this page helpful?
Thank you! We'll create more content like this.
Thank you for helping us improve!