For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Prefix caching

Prefix caching is a technique that caches the key-value (KV) cache of existing inference requests so that new queries can reuse the context encoded in the KV cache if they share the same prefix. This eliminates redundant computations and improves performance for workloads with repeated prefixes.

Prefix caching is enabled by default when serving a model with the max serve CLI command. You can disable it with the --no-enable-prefix-caching flag.

When to use prefix caching

Prefix caching speeds up the pre-fill stage of inference, which reduces time to first token (TTFT). It can also reduce memory usage within the KV cache for all requests, which makes room for scheduling larger batches and yielding higher throughput.

Prefix caching can provide significant performance improvements in the following scenarios:

• Similar queries: When a user repeatedly makes similar queries that use the same system prompt instructions, the KV cache of the prefix can be stored in advance to reduce redundant computation.
• Multi-round conversations: In chat applications, users often ask follow-up queries related to previous inputs. Since the server releases KV cache memory after each request, prefix caching preserves computation from past conversation turns without requiring an explicit session.

Prefix caching won't result in performance degradation. However, it also doesn't provide additional benefit in the following cases:

• Unique queries: If new queries don't share prefixes with previous queries, there's no opportunity to reuse cached KV values, making prefix caching ineffective.
• Long response generation: Prefix caching only speeds up the pre-fill phase of a request. If a request spends most of its time generating new tokens (decoding), caching has little impact.

How prefix caching works

Prefix caching works by storing the key-value (KV) cache for a prefix and applying it to future prompts that include the same prefix, reducing redundant computation. MAX manages the KV cache with PagedAttention, which stores the cache in fixed-size pages that MAX can reuse across requests that share a prefix.

The following flags control prefix caching behavior:

• --enable-prefix-caching: enables prefix caching. This is on by default, so you only need it to re-enable caching after disabling it. Use --no-enable-prefix-caching to disable it.
• --kv-cache-page-size: sets the number of tokens per page in the paged KV cache. The page size must be a multiple of 128.
• --enable-dp-cross-replica-prefix-copy: when serving with data parallelism (--data-parallel-degree greater than 1), lets MAX serve a prefix-cache hit that lives on another replica's GPU by copying it device-to-device onto the requesting replica. This is on by default. When disabled, MAX serves cross-replica reuse only from the shared host or disk tier, or recomputes it.

Prefix caching works on both CPU and GPU. To deploy a model with prefix caching using the max CLI, you can use the flag --devices cpu for CPU or --devices gpu for GPU workloads. If you omit --devices, MAX uses the model or config default.

MAX doesn't support prefix caching for multimodal models or LoRA adapters. Because of this, MAX automatically disables prefix caching for multimodal models. When serving with LoRA adapters, disable prefix caching explicitly with the --no-enable-prefix-caching flag.

Quickstart

Prefix caching is enabled by default when serving a model with MAX. To install the max CLI, see the installation guide.

The following command serves Gemma 3 with prefix caching enabled by default:

max serve --model google/gemma-3-27b-it

Explicitly enable prefix caching when serving your model with the --enable-prefix-caching flag:

max serve --model google/gemma-3-27b-it --enable-prefix-caching

Disable prefix caching

To disable prefix caching when serving a model, use the following command:

max serve --model google/gemma-3-27b-it --no-enable-prefix-caching

Next steps

Now that you know the basics of prefix caching, keep customizing how you serve models on MAX:

• Speculative decoding: Use speculative decoding to accelerate LLM inference.
• Benchmark MAX on NVIDIA or AMD GPUs: Learn how to use our benchmarking script to measure the performance of MAX.
• Deploy MAX on GPU with self-hosted endpoints: Learn how to deploy MAX pipelines to cloud.
• Read the LLM Inference Handbook: Learn more about prefix caching.