For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

AutoTokenizer

AutoTokenizer

class max.pipelines.AutoTokenizer(model_path, *, revision=None, trust_remote_code=False)

source

Bases: object

Loads the Hugging Face tokenizer serve uses to encode a model’s text.

Calling it returns the input_ids and attention_mask that AutoModel.generate() takes, with special tokens added as serve adds them to a completion prompt.

tokenizer = AutoTokenizer("HuggingFaceTB/SmolLM2-135M")
model = AutoModel("HuggingFaceTB/SmolLM2-135M", max_batch_size=2)
inputs = tokenizer(["The capital of France is", "2 + 2 ="])
outputs = model.generate(**inputs, max_new_tokens=16)
print(tokenizer.batch_decode(outputs.sequences))

Parameters:

• model_path (str) – A Hugging Face repo ID or a local model directory.
• revision (str | None) – The repo revision to load.
• trust_remote_code (bool) – Whether to run tokenizer code from the repo.

batch_decode()

batch_decode(sequences, *, skip_special_tokens=False)

source

Returns the text for each token-id list in sequences.

Parameters:

• sequences (Iterable[Sequence[int]])
• skip_special_tokens (bool)

Return type:

list[str]

decode()

decode(token_ids, *, skip_special_tokens=False)

source

Returns the text for token_ids.

Parameters:

• token_ids (Sequence[int])
• skip_special_tokens (bool)

Return type:

str