For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Generate tokens offline
With MAX's AutoModel, you
run inference on a model from a Python script, with no server to start and no
HTTP requests to send. We call this offline generation. AutoModel builds the
pipeline that max serve runs, with serving features such as
the overlap scheduler and prefix caching turned off, so your script calls the
same model and sampling code as a served endpoint, with no request scheduler
between your code and the model.
Because the model runs in your script, you work with it directly:
- You pass token IDs in and get token IDs back. Pass
output_logits=Truetogenerate()to also get the model's logits for every token it generates. - You decide which prompts run together, down to a single forward pass. Call
generate_iter()to get aGenerateIterator: eachstep()runs one pass, andremove()drops a row from later passes. - With
eager=True, a breakpoint in the model'sforward()stops on every step with real tensor values.
The API follows Hugging Face Transformers:
AutoTokenizer encodes
prompts, AutoModel generates from them as Transformers'
AutoModelForCausalLM does, and the tokenizer decodes the result.
AutoPipeline wraps
AutoTokenizer and AutoModel, so it takes prompt strings and returns text in
one call.
On this page, you'll install MAX and generate text with LFM3, a 1.7-billion-parameter model from Liquid AI. The first run downloads the checkpoint to your Hugging Face cache, and later runs load it from there.
Set up your environment
LFM3 runs only on a GPU, so check that yours is on the GPU compatibility list. Then create a Python project and install MAX:
- pixi
- uv
- If you don't have it, install
pixi:curl -fsSL https://pixi.sh/install.sh | shThen restart your terminal for the changes to take effect.
- Create a project:
pixi init offline-quickstart \ -c https://conda.modular.com/max-nightly/ -c conda-forge \ && cd offline-quickstart - Install
maxwith all dependencies (nightly):pixi add max-all - Start the virtual environment:
pixi shell
- If you don't have it, install
uv:curl -LsSf https://astral.sh/uv/install.sh | shThen restart your terminal to make
uvaccessible. - Create a project:
uv init offline-quickstart && cd offline-quickstart - Create and start a virtual environment:
uv venv && source .venv/bin/activate - Install
maxwith all dependencies (nightly):uv add "max[all]" \ --index https://whl.modular.com/nightly/simple/ \ --prerelease allow
Generate text
With the environment active, generate text from the
LiquidAI/LFM3-1.7B-Base
model:
-
Save this script as
generate.py:generate.pyfrom max.pipelines import AutoModel, AutoTokenizer model_id = "LiquidAI/LFM3-1.7B-Base" tokenizer = AutoTokenizer(model_id) inputs = tokenizer(["The capital of France is"]) with AutoModel(model_id, max_length=4096) as model: result = model.generate(**inputs, max_new_tokens=8, top_k=1) print(tokenizer.decode(result.sequences[0], skip_special_tokens=True))Calling
tokenizer()returns theinput_idsandattention_maskthatgenerate()takes. Settingtop_k=1makes generation greedy, so each step picks the most likely token. When thewithblock exits,AutoModelcallsclose(), which frees the memory the model uses.max_lengthcaps the number of tokens in a sequence. If you leave it out, MAX plans for the model's full context length, 262,144 tokens for LFM3, which reserves much more GPU memory for the KV cache. -
Run the script:
python generate.pyAutoModeldownloads the 3.46 GB checkpoint, compiles the model, and generates eight tokens on the first GPU it finds.The script prints the prompt followed by the new tokens:
The capital of France is Paris. The population of Paris is
To run another model, replace model_id with a Hugging Face repository ID
from Supported models.
Generate for several prompts
The script passed one prompt to tokenizer(). Pass a list of prompts to run
them as one batch, and set max_batch_size to at least the number of prompts,
because AutoModel loads with room for one prompt by default:
from max.pipelines import AutoModel, AutoTokenizer
model_id = "LiquidAI/LFM3-1.7B-Base"
prompts = ["The capital of France is", "The capital of Japan is"]
tokenizer = AutoTokenizer(model_id)
inputs = tokenizer(prompts)
with AutoModel(model_id, max_length=4096, max_batch_size=len(prompts)) as model:
result = model.generate(**inputs, max_new_tokens=8, top_k=1)
for text in tokenizer.batch_decode(result.sequences, skip_special_tokens=True):
print(text)result.sequences holds one row per prompt, in the order you passed them:
The capital of France is Paris. The population of Paris is
The capital of Japan is Tokyo, and the capital of the UnitedChat with an instruct model
The base model continues raw text. An instruct model, such as
LiquidAI/LFM3-1.7B, expects a
conversation in its chat template. Pass the conversation to generate() as
messages, and MAX applies the template for you:
from max.pipelines import AutoModel, AutoTokenizer
model_id = "LiquidAI/LFM3-1.7B"
tokenizer = AutoTokenizer(model_id)
messages = [[{"role": "user", "content": "What is the capital of France?"}]]
with AutoModel(model_id, max_length=4096) as model:
result = model.generate(
messages=messages,
chat_template_kwargs={"enable_thinking": False},
max_new_tokens=32,
top_k=1,
)
reply = result.sequences[0][result.prompt_lengths[0] :]
print(tokenizer.decode(reply, skip_special_tokens=True))Each row in result.sequences starts with the prompt after MAX applies the
template, and result.prompt_lengths gives that prompt's length, so the slice
keeps only the reply. LFM3 writes its reasoning before it answers, and
{"enable_thinking": False} turns that off in the chat template. The script
prints up to 32 tokens of the reply:
The capital of France is **Paris**. It is known for its art, fashion, gastronomy, and culture. Paris is located in the north-centralNext steps
- Step through a model with pdb:
pass
eager=TruetoAutoModelto stop on every step with real tensor values. - Model bring-up workflow: implement an
architecture that
AutoModelandmax servecan load. - Quickstart for serving: start a local endpoint and send it requests.