IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content

MAX v26.6

Highlights​

  • Added support for audio generation with MiniMax-Music3 (MiniMaxAI/MiniMax-Music3). You can use this model to generate music in 44.1 kHz stereo audio from a text prompt, including lyrics. Give it a try by following the new audio generation guide.

  • Added support for Inkling (thinkingmachines/Inkling), a multimodal model that accepts text and image inputs, with support for NVFP4 mixture-of-experts. Check out our Inkling configuration recipe for NVFP4 on 2x B200.

  • Added support for GLM-5.3 (zai-org/GLM-5.3), an LLM that's optimized for improved coding, agentic workloads, long-horizon tasks, and cyber capabilities compared to GLM-5.2. Check out our GLM-5.3 configuration recipes for FP8 and NVFP4 on 8x B200.

  • Added support for speculative decoding with DFlash and DSpark draft models on Gemma 4 31B. These drafters speed token generation without changing the model's output. Learn more in the speculative decoding guide.

  • The max.gpu package now includes everything previously provided in Mojo's std.gpu package, making it a complete entry point for accelerator programming.

Documentation​

  • Added a metrics reference page with all available Prometheus metrics, categorized by subsystem.

  • Added a reasoning guide, covering per-request reasoning, reasoning_effort, thinking temperature, parser overrides, and renaming the reasoning field.

  • Added documentation for audio generation:

    • Added an audio generation guide, covering serving a text-to-music model over /v1/audio/speech and /v1/responses, the request fields and their defaults, the lyric tag syntax, and the length a single render is capped at.
    • Added a music generation example that renders songs past that per-render cap by rendering sections and joining them, and checks the joins for audible seams.
    • Added reference docs for the /v1/audio/speech operation and the provider_options.audio attribute on /v1/responses.

  • Added the parallelism guide, explaining how to distribute models across multiple GPUs.

MAX models​

  • Added support for MiniMax-Music3 (MiniMaxAI/MiniMax-Music3), the first architecture on the audio_generation pipeline: a text-to-music model that renders a style caption plus lyrics into 44.1 kHz stereo audio. The five component networks exceed a 24 GB card together, so the pipeline builds and releases each stage in turn within a request; the first request after a cold start pays a multi-minute compile that later ones replay from the compilation cache.

  • Added support for Inkling (thinkingmachines/Inkling), a hybrid short-convolution and relative-bias attention decoder with an NVFP4 mixture-of-experts.

    • Added vision support to Inkling: image prompts run through the model's hMLP vision tower and replace image placeholder tokens in the decoder prompt.
    • Added an Inkling tool-call parser and constrained decoding for tool arguments, so tool invocations are returned in tool_calls and follow the declared JSON schema.
    • Added an Inkling reasoning parser that records an explicit <|content_thinking|> span and leaves direct answers out of the reasoning field.
    • Added multi-token prediction (MTP) speculative decoding for Inkling (UnifiedMTPInklingForConditionalGeneration), serving the checkpoint's chained dense draft depths; enabled automatically for Inkling checkpoints that ship mtp_config with --speculative-method mtp.

  • Added support for nvidia/Kimi-K2.7-Code-NVFP4 on the Kimi K2.5 ModuleV3 architecture, with an 8-GPU B200 recipe.

  • Expanded Qwen support:

    • Added Qwen/Qwen3.8-27B support in bfloat16 on the existing Qwen3_5ForConditionalGeneration architecture, covered by logit verification against the PyTorch reference.
    • Qwen3_5ForConditionalGeneration now serves across multiple GPUs. Tensor parallelism splits the attention heads, the gated-DeltaNet key and value heads, and the per-device linear-attention state pools; both mixers reject a device count that would not divide their head counts evenly.
    • Qwen3_5ForConditionalGeneration now supports device graph capture.
    • Added multi-token prediction (MTP) speculative decoding for Qwen3.8 (UnifiedMTPQwen3_5ForConditionalGeneration), fusing the target, the baked-in MTP head, and a recurrent-state rollback into one graph, selected for Qwen3.5-family checkpoints that ship an MTP head with --speculative-method mtp.
    • Added --state-pool-dtype, which overrides the storage dtype of a hybrid model's recurrent state pools (SSM and linear-attention conv and recurrent state). It defaults to the model's compute dtype. float32 makes a speculated generation follow the same state trajectory as an unspeculated one—the recurrence rounds to the pool dtype at each call boundary, so a lossy pool makes the trajectory depend on how speculation chunked the sequence—at roughly double the per-request state memory (Qwen3.8-27B: 74.8 to 149.6 MiB per seated request).
    • Added mixed-precision NVFP4 and FP8 checkpoint loading for Qwen3_5ForConditionalGeneration, so RadixArk/Qwen3.8-27B-NVFP4 loads with per-module quantization instead of being silently treated as uniform NVFP4.
    • Qwen3_5ForConditionalGeneration now uses a shared vision encoder cache, and can serve image requests with an FP8 KV cache.

  • Expanded Gemma 4 support:

    • Added DSpark speculative decoding for Gemma 4 31B (UnifiedDSparkGemma4_31BForCausalLM), serving google/gemma-4-31B-it with the vLLM speculators-format draft RedHatAI/gemma-4-31B-it-speculator.dspark (llama-style causal draft block, pruned 32k draft vocabulary mapped through the checkpoint's d2t table). Enabled with the gemma4_31b_dspark.yaml recipe or --draft-model-path RedHatAI/gemma-4-31B-it-speculator.dspark --speculative-method dflash. An explicit --num-speculative-tokens is honored: values below the trained 7 truncate the causal draft block prefix-stably, values above run as extrapolation with a warning and degrading acceptance; unset defaults to the trained 7.
    • Added DFlash speculative decoding for Gemma 4 31B (UnifiedDflashGemma4_31BForCausalLM), serving google/gemma-4-31B-it with the z-lab block-diffusion drafter z-lab/gemma-4-31B-it-DFlash: a 5-layer noncausal draft block drafts 15 tokens per step from six target hidden-state taps. Enabled with the gemma4_31b_dflash.yaml recipe or --draft-model-path z-lab/gemma-4-31B-it-DFlash --speculative-method dflash. The draft width is pinned to the drafter's trained block_size - 1; a mismatching --num-speculative-tokens is overridden with a warning. NVFP4 target checkpoints (nvidia/Gemma-4-31B-IT-NVFP4) are supported via the gemma4_31b_dflash_nvfp4.yaml recipe.
    • Gemma 4 31B DSpark now supports structured output (JSON schemas and tool-call grammars, enforced on the target verify pass; a grammar-violating draft is rejected at its position) and Gemma 4 thinking: reasoning content is split out of responses, and relaxed acceptance during the thinking phase can be enabled with use_relaxed_acceptance_for_thinking.
    • Renamed the Gemma 4 12B DSpark architecture to UnifiedDSparkGemma4_12BForCausalLM (module max.pipelines.architectures.unified_dspark_gemma4_12b), so the two Gemma 4 DSpark architectures are named by model line.
    • Added a Gemma 4 ModuleV3 eager architecture (Gemma4ForConditionalGeneration_ModuleV3) so Gemma 4 checkpoints can run through the eager API with --prefer-module-v3.
    • Gemma 4 31B DSpark (UnifiedDSparkGemma4_31BForCausalLM) now accepts NVFP4 target checkpoints such as nvidia/Gemma-4-31B-IT-NVFP4, via the gemma4_31b_dspark_nvfp4.yaml recipe.
    • Fixed unbounded host-memory usage in Gemma 4 video pre-processing.
    • Fixed the Gemma 4 12B DSpark draft applying full RoPE instead of the checkpoint's partial rotary factor (0.25), which was costing roughly 10% of the draft acceptance rate.

  • Expanded GLM support:

    • The GLM-5.2 B200 recipe now serves the checkpoint's full 1M-token context window (max_length: 1048576, previously pinned to 163840).
    • Added production FP8 and NVFP4 serving recipes for GLM-5.3 on 8x B200.
      • The NVFP4 recipe enables MTP and the checkpoint's 1M-token context.
      • Raised the NVFP4 recipe's max_batch_size from 8 to 32.
    • GLM chat templates now default clear_thinking to True and map reasoning_effort=low onto a template's low rung when it has one (GLM-5.3).
    • Masked GLM-5.x's padded vocabulary tail to -inf so untrained lm_head rows past the tokenizer can no longer be sampled as out-of-vocabulary tokens.

  • Improved tensor-parallel + expert-parallel serving for DeepSeek V3 / V3.2 and GLM 5.1 / 5.2:

    • Fused each DeepSeek V3 / V3.2 and GLM 5.1 / 5.2 decoder block's post-MLP all-gather with the next layer's input RMSNorm via ops.allgather_rms_norm on the tensor-parallel + expert-parallel path.
    • Restored graph-capture KV-cache headroom in the DeepSeek V3 memory planner to improve throughput in models that plan through it (DeepSeek V3/V3.2 and GLM-5.x MTP).

  • Switched MiniMax M2 tool calling from an llguidance Lark grammar to the xgrammar minimax structural tag.

  • Gave Gemma 3, Step-3.5-Flash, and Inkling hybrid Jenga KV groups, so sliding-window pages plateau independently of full-attention pages. Enabled Jenga hybrid groups by default for GPT-OSS and Olmo 3.

  • Fixed DeepSeek V3 losing tool-call parsing when served with Eagle3 or MTP speculative decoding. The fused spec-decode architectures now inherit the base architecture's settings instead of redeclaring them, so --tool-parser no longer has to be passed by hand.

MAX framework​

  • Improvements to host-side and graph profiling:

    • Host-side profiling spans (max.profiler.Tracer, @traced, and Mojo Trace scopes) now annotate external profiler tools on release builds: with MODULAR_ENABLE_PROFILING set, spans appear as NVTX ranges in NVIDIA Nsight Systems captures and as roctx ranges in rocprofv3 captures, with no build flags required. Previously these spans were only emitted in special profiling builds.
    • Added Graph.profile_scope, a context manager that labels every op for profiling. The scope name is appended to the op name in profile output. Profiler ranges are also created from sequential ops with the same scope, which is enabled with MODULAR_MAX_DEBUG_PROFILE_SCOPE_TRACING=1.

  • Improvements to the AMD GPU runtime:

    • Made UCCL the default NIXL transfer backend on AMD GPUs. Set MODULAR_NIXL_TRANSFER_BACKEND to ucx or libfabric to opt out.
    • Forced the virtual-memory allocator off on AMD GPUs, where it caused persistent VRAM leaks. Requesting memory_manager_vmm now logs a warning and is ignored.
    • Improved AMD decode latency on ROCm 7.14 and later by issuing batched device copies through hipMemcpyBatchAsync instead of one hipMemcpy per buffer.

Inference server​

  • Added the audio_generation pipeline task, for models that render audio rather than text or pixels. Its request options (lyrics, duration, denoising steps, guidance scale, output format) arrive as the audio provider options of an OpenResponses request, and an architecture on the task serves over /v1/audio/speech and /v1/responses. Responses report usage the way image generation does: token counts stay at 0 and a usage.audio_generation_details block carries duration_seconds, sample_rate, channels, num_samples, and steps, measured from the audio actually produced rather than the duration that was asked for.

  • Added a request body size limit. MAX_SERVE_MAX_REQUEST_BYTES (default 100 MiB) caps the size of an accepted HTTP request body; a larger request is rejected with HTTP 413 before the body is buffered, so a client cannot exhaust host memory with an oversized payload. The cap is enforced both from an oversized Content-Length and by counting the bytes actually received, so a chunked or mislabeled body cannot evade it. Raise it for larger inline (base64) multimodal payloads, or set it to 0 to disable the limit.

  • Improvements to the vision encoder cache:

    • The block-based vision encoder cache now shards its storage across devices instead of replicating every entry on each one. The same --vision-cache-utilization fraction buys the same cache capacity while reserving only 1/n_devices of it per device; the remainder stays with the KV cache. Cache hits gather rows to each device in one batched submission.
    • The vision encoder cache now stores embeddings in fixed-size blocks. Capacity is a byte budget carved into 128-token blocks—a video spans many blocks and an image spans a few—so a video-capable model no longer collapses the cache to a handful of worst-case-video slots that starve image workloads. The budget is set with the new --vision-cache-utilization flag, a fraction of the KV cache pool budget (default 0.05; 0 disables caching). The previous entry-count cache and its --max-vision-cache-entries flag are removed.
    • Vision embedding assembly during chunked prefill is now bounded by the active window: each step copies only the embedding rows whose placeholder tokens fall inside the chunk, with dense scatter indices, instead of rebuilding every image's rows with out-of-bounds sentinels. Per-chunk copy cost now scales with the chunk size rather than the request's total image tokens.

  • --kv-connector-config '{"type": "rust_tiered", "disk_offload_max_gb": 0}' now runs the tiered connector with no disk last level: offloaded blocks stop at the pinned host tier and no offload directory is created. Leaving disk_offload_max_gb unset still sizes the disk tier from the device page pool, and a negative budget is now rejected instead of silently accepted.

  • Enabled the Jenga KV cache by default for Gemma and Llama. Set MODULAR_USE_LEGACY_KV_CACHE=1 to opt out. Disaggregated inference and the dKV connector still use the legacy cache.

  • Improvements to speculative acceptance and draft defaults:

    • Greedy speculative acceptance (greedy_acceptance_sampler, AcceptanceSampler in greedy mode) now applies the structured-output grammar bitmask to the target logits (with a -inf fill) before the argmax, so a grammar-invalid draft is always rejected and recovered and bonus tokens always satisfy the constraint—matching the stochastic path. Unconstrained batches are unchanged.
    • stochastic_acceptance_sampler and AcceptanceSampler also accept a rank-1 [batch_size] per-row seed tensor in stochastic argmax mode: each row's acceptance sampling is then keyed off its own seed instead of row 0's, so a row samples independently of its co-residents. A single-row batch is bit-identical to the scalar-seed behavior. The Gemma 4 and Qwen3.5 unified MTP graphs now pass their per-row seed tensors through.
    • Added --enable-spec-decode-mixed-batches (default off): with speculative decoding, prefill requests are batched into decode steps (implies in-flight batching) and the riding decode rows keep verifying their draft tokens instead of advancing one token at a time. Supported by the Gemma 4 DSpark, DFlash, and MTP architectures; other architectures fall back to plain in-flight batching, as do batches containing grammar-constrained rows or rows with images awaiting vision encoding.
    • --num-speculative-tokens is now unset by default, and each speculative method resolves its own default: eagle and mtp keep drafting 2 tokens per step, while dflash-style block drafters (DFlash, DSpark) derive the draft checkpoint's trained block width. Explicit values are honored as before. Previously the flag defaulted to 2 for every method and block drafters overrode it at load time with a warning; a bare DFlash run now also sizes its KV cache draft headroom at the trained width instead of the old default.

  • Added opt-in knobs to release host memory after load:

    • Added MODULAR_MAX_RELEASE_FREE_HOST_MEMORY, an opt-in serving knob that returns free host-allocator pages to the OS once model compilation finishes, before graph capture. Graph compilation leaves tens of GiB free-but-unreturned in glibc's per-thread arenas, which glibc never reclaims on its own; setting this variable to any non-empty value calls malloc_trim(0) at that point. On Gemma 4 31B this returns ~24 GiB of anonymous RSS per model worker in ~1.4s. Unset by default, and a no-op on platforms without malloc_trim.
    • Setting the MODULAR_MAX_RELEASE_HOST_WEIGHTS environment variable to 1 frees the host copies of checkpoint weights once the GPU holds them, returning the full checkpoint size in host RSS. GPU deployments of graph-API architectures only; weights that execute on CPU must not be released.

  • Fixed response_format schema normalization skipping containers the grammar backends compile: an untyped object-shaped subschema under additionalProperties, unevaluatedProperties, unevaluatedItems, or dependentSchemas is now anchored to an object, as one under properties already was. Such a subschema previously compiled to a grammar admitting an unbounded value, letting a looping model run to max_length.

  • Chat completions now honor reasoning_effort; previously only an explicit chat_template_kwargs.reasoning_effort had any effect and the standard fields were silently ignored. An effort of none disables thinking, and values set directly in chat_template_kwargs still win.

  • Hardened fetching of client-supplied media URLs:

    • /v1/responses now fetches client-supplied input_image URLs through the same media resolver as /v1/chat/completions, so the two paths share one byte cap and one error mapping. Previously the responses path had its own downloader with no size limit, meaning an arbitrarily large image could be fetched and base64-expanded in memory, and its failures echoed the underlying network error back to the client. The inlined data: URI's MIME type is now sniffed from the fetched bytes instead of guessed from the URL, and content that is not a decodable image is rejected with a 400 rather than inlined as an image.
    • Hardened the server-side fetch of client-supplied image_url / video_url references against SSRF: the host is now validated and hosts that resolve to internal or reserved addresses are rejected before the fetch. On by default (MAX_SERVE_MEDIA_URL_SSRF_PROTECTION_ENABLED); a per-host allowlist (MAX_SERVE_MEDIA_URL_ALLOWED_HOSTS, hostnames or CIDRs) permits trusted internal hosts.

  • Improvements to structured-output and JSON Schema compilation:

    • Structured-output grammars are now compiled once, in the model worker. Removing the duplicate lowers time to first token for structured requests by 12-22% (Gemma 4 31B, concurrency 32); decode latency and requests without structured output are unchanged. An uncompilable grammar is still rejected with the same HTTP 400, streaming requests included, and a disaggregated prefill node now reports the failure to the decode node instead of leaving the request to time out.
    • Structured-output grammar compilation now runs off both serving hot paths. A new request's grammar matcher (from response_format JSON schemas or tool-call grammars) is built on a worker thread while the request waits for admission instead of on the scheduler's decode thread, and the API server's admission-time schema validation runs off the event loop instead of freezing in-flight streaming responses. A cold multi-second compile of a complex schema now delays only that request instead of stalling inter-token latency for every active request.
    • Structured-output JSON grammars can be made whitespace-tolerant, per architecture, via default_structured_output_any_whitespace. GLM 5 models default to whitespace-tolerant response_format grammars.
    • A JSON schema that composes with allOf is now enforced instead of refused. response_format and tool-call schemas previously returned 400 for any allOf with more than one member, or with a sibling object keyword. The members now fold into one schema before compilation, including members nested in another member's allOf and members that are a bare local $ref, so the common "shared definition plus an extension" shape compiles. A conjunction that cannot be folded exactly still returns 400 naming the keyword pair at fault, rather than compiling to a looser grammar.
    • A JSON schema using oneOf is now enforced when its branches can be proven pairwise disjoint, instead of being refused outright. Disjoint branches make the union exactly-one, which is what oneOf means. Branch types and const/enum value sets carry the proof, covering nullable values, scalar unions, enum partitions, and unions discriminated by a constant property. A union that cannot be proven disjoint still returns 400, as does a const/enum branch carrying a keyword the lowering drops. The refusals apply when unsupported-schema rejection (reject_unsupported) is enabled.
    • Fixed a union (anyOf/oneOf) folding its sibling keywords into each branch too widely, which could accept values the schema forbids. These shapes now return 400 instead, when unsupported-schema rejection (reject_unsupported) is enabled: a closing additionalProperties, items, or unevaluatedProperties beside a union, a base constraint beside $ref, const, or enum—whether folded in from a union or written in the same object—and $ref beside a sibling union.
    • Compiling deeply nested JSON schemas is substantially faster and uses less memory by avoiding repeated subtree copies while constructing cache keys. Emitted grammars are unchanged.
    • Fixed strict JSON Schema compilation silently dropping string length bounds when a pattern or format is present. Redundant bounds now compile, while unsatisfiable or partially overlapping constraints return 400. Equivalent direct, allOf, and union-folded schemas receive the same result. Regex length analysis has a per-schema work limit, so oversized patterns return 400 promptly.
    • Fixed JSON Schema compilation resolving a local $ref against the wrong resource when the document declares a resource identifier ($id, or id in Draft 4) below its root. A fragment names a place inside the resource it is resolved against, and every fragment was resolved against the whole document, so a definition name that appeared in both an embedded resource and at the root bound the root's copy in silence. When unsupported-schema rejection (reject_unsupported) is enabled, such a document now returns 400 naming the declaration, rather than compiling a grammar the author never wrote. A document whose only resource identifier sits at the root, or that has none at all, is one resource and is unaffected.
    • JSON Schema compilation now recognizes the $schema dialects it models: Draft 4, whose resource identifier is id, and Drafts 6, 7, 2019-09, and 2020-12, whose identifier is $id. When unsupported-schema rejection (reject_unsupported) is enabled, any other $schema returns 400 rather than being read as a modern document, because assuming the wrong dialect walks past the resources a document declares and resolves its fragments against the wrong one. Omitting $schema, as most tool schemas do, still means the current draft and is unaffected.
    • Fixed structured output under disaggregated prefill/decode serving. When --enable-structured-output or --enable-tool-call-constrained-decode is set, decode now re-forwards the handoff as a one-token continuation and samples token 1 under its own grammar matcher.
    • Removed the llguidance tool-call path from Kimi K2.5 and turned on reject_unsupported by default for every xgrammar JSON schema. Unsupported schema shapes now return HTTP 400 instead of compiling a looser grammar.

  • Improvements to speculative decoding controls:

    • Speculative decoding can now verify only some of the draft tokens it generates, varying that count with the decode batch size via the new num_speculative_tokens_per_batch_size speculative-config field. Each entry names an inclusive batch-size range and a count through the keys batch_start, batch_end, and num_tokens, so a two-range schedule is [{"batch_start": 1, "batch_end": 16, "num_tokens": 3}, {"batch_start": 17, "batch_end": 64, "num_tokens": 1}]. The first range must start at batch size 1 so every batch size resolves to a count; gaps and the tail carry the previous count forward. Drafting is cheap, but every draft the target verifies is another query position in its forward pass, so at high concurrency those positions compete with real tokens for the same compute and a rejected draft is compute spent for nothing. Whether narrowing pays off therefore depends on how well the drafts are being accepted, which is a property of the workload rather than of the batch size. Measure your own workload before adopting a schedule. The field is off by default, and unset behavior is unchanged. It applies to every speculative method. A block drafter (dflash) still drafts its whole checkpoint-fixed block every step, so a schedule narrows only how much of that block the target verifies; the saving comes from the target's verify pass, never from drafting less.

      It is most useful for a block drafter, whose draft depth is fixed by its checkpoint, making the verified count the only runtime lever on step cost. Where the draft depth is itself configurable, as it is for eagle and mtp, lowering num_speculative_tokens is the better tool: it removes the draft passes as well as the verify positions, while a schedule pays for drafts it then discards. A count of 0 is accepted and disables verification for that batch-size range.

    • Speculative decoding takes --draft-proposal sampled (default argmax, unchanged). The draft model samples its proposal under the request's temperature/top-k/top-p and keeps the distribution it drew from, so verification runs true speculative sampling—accept on the p_target/q_draft ratio, recover from max(p_target - q_draft, 0)—rather than the typical-acceptance approximation, and the emitted tokens follow the target model's distribution.

    • Added --num-speculative-tokens-mixed-batch, which sets how many drafted tokens the target verifies on a mixed prefill+decode batch. Unset keeps mixed batches on the num_speculative_tokens_per_batch_size schedule.

  • GLM models now map reasoning_effort onto the two thinking levels their chat template can express, instead of forwarding it verbatim. The template reads only high as a distinct level and treats every other value as maximum effort, so passing the value through inverted the scale: low and medium requested maximum reasoning while high requested less than they did. Every effort other than none (which disables thinking), max (the template's own top level, still addressable directly) and xhigh (OpenRouter's name for that same top level) now selects the lower level, so an unrecognized value degrades to less reasoning instead of silently maxing out. Requests that set no effort are unaffected.

  • Added MAX_SERVE_HTTP_KEEPALIVE_TIMEOUT_S (default 120 seconds) so MAX holds idle HTTP connections open longer than a pooling client's idle timeout. The previous 5-second uvicorn default closed first and surfaced as TCP resets on the next pooled request.

  • Unhandled server errors on the OpenAI-compatible endpoints now return the OpenAI JSON error envelope instead of a bare text/plain HTTP 500 that closed the connection.

  • Added OpenTelemetry distributed tracing: a TracerProvider with an OTLP HTTP span exporter that honors OTEL_EXPORTER_OTLP_ENDPOINT, a max.request span on the chat-completions, completions, and embeddings handlers, and max.phase.prefill / max.phase.decode / max.batch spans. Inbound W3C traceparent / tracestate headers are propagated. Added MAX_SERVE_KERNEL_TRACE_LEVEL (off / batch / op / kernel) to gate libkineto GPU-trace capture; the default off path has zero overhead.

  • The rust_tiered KV connector now supports Jenga hybrid sliding-window groups, so sliding-window pages can plateau instead of being treated as full-attention pages.

  • Per-request cache_salt now isolates prefix-cache entries under the default ahash64 algorithm; it was previously dropped. Client-supplied salt (X-Cache-Salt or the request body) is honored only when MAX_SERVE_USE_CLIENT_CACHE_SALT is enabled.

  • An explicit --chat-template override is now applied even when a trust_remote_code tokenizer hardcodes apply_chat_template and ignores the chat_template attribute.

Server metrics​

  • maxserve_cache_hits_tokens_total now carries a tier label naming what served each token: g0 for the on-device prefix cache (including cross-replica device-to-device copies), external for the KV connector. The per-tier series sum to the untagged total, so an existing single-series query that doesn't group by tier returns the same numbers as before. Misses stay unlabeled, which means a PromQL binary operation pairing hits against misses (a hit-rate expression) now matches on mismatched label sets and returns empty: add ignoring(tier), or wrap the hits side in sum without(tier) (...). The in-tree Datadog dashboard aggregates the tag away and is unaffected; external Prometheus consumers are the ones affected. Previously the on-device share could only be derived by subtracting the external tier's own server-side counters, which measure what that tier holds rather than what a request could use and so overstate reuse. Note that the untagged series is replaced rather than extended, so a rate() window spanning the upgrade sees the old series go stale and the labeled ones start from zero.

  • Added and fixed dKV metrics and logs:

    • Added maxserve_dkv_read_blocks_total, the count of KV blocks that landed in device memory from the dKV tier. Only confirmed-complete transfers count, so it measures delivered reuse. It is emitted only on dKV deployments, while maxserve_cache_hits_tokens_total{tier="external"} is stamped for any KV connector, so a missing counter means "not dKV" rather than "nothing landed." On a dKV deployment the two track each other for every load that lands, and comparing them needs the server's --kv-cache-page-size, since one is in blocks and the other in tokens.
    • maxserve_dkv_rpc_read_latency and maxserve_dkv_rpc_acquire_latency now report. Both were declared and published on a positive value, but nothing ever measured the underlying RPCs, so neither series ever appeared and the per-batch server log printed acquire 0.0ms, pin 0.0ms on every line, which reads as an instant lookup rather than an unmeasured one. The connector now times both round trips. They bracket the RPC rather than the transfer, so they include work the transfer latencies cannot see, most importantly the disk-tier restage the server awaits inside its read handler.
    • Added maxserve_dkv_nixl_read_latency_max, the slowest single dKV read in the window a batch samples, next to the existing maxserve_dkv_nixl_read_latency average. An average cannot separate one slow read from a uniformly slow batch, and it is the slow read that costs a request its time to first token. The peak also appears on the per-batch server log line and in the structured log, and it combines across data-parallel replicas by taking the maximum rather than by summing.
    • The per-batch dKV log clause now reports the blocks that landed and the bytes read, alongside the read average and the new peak. The block count was already in the structured log but missing from the human-readable line, and the byte count was not recoverable from either: the reported throughput divides by the transfer-time total, both surfaces carry only the average, and the sample count that bridges them is published nowhere. The clause is also emitted whenever a batch transferred blocks, where it was previously emitted only when a latency sample survived, so a read whose timing sample was dropped no longer drops the whole clause, and its block count with it. Such a batch reports its counts without the read timings rather than beside a row of zeros, which would read as an instant read.

  • Every number on the scheduler's batch-metrics log line is now also emitted as a structured log field, so log backends can facet and alert on them.

  • Fixed the speculative-decoding per-position acceptance-rate histogram (maxserve_spec_decode_acceptance_rate_per_position) understating acceptance: decode batches that performed zero verifications published a full row of 0% observations, diluting every position's average. Such batches now contribute nothing, matching the acceptance-length histogram's population. The batch log line also shows the acceptance length including the bonus token next to the accepted-drafts-per-step value, since the two conventions are easy to confuse.

  • Utilization histograms now bucket the top decile in 1% steps (91-100) instead of 2%, so a nearly-full KV cache tier resolves more precisely. Dashboards or alerts that hardcode le bucket edges above 90 need updating.

max CLI​

  • Improvements to cache warming:

    • max warm-interpreter-cache now shows a live progress row per op family.
    • Fixed max warm-interpreter-cache failing with a ValueError on a machine where an op family supports none of the available devices (for example, a GPU-only op family on a CPU-only machine). Such a family now warms as a no-op instead of aborting the whole command.
    • Fixed max warm-cache failing on diffusion models with No main model configured. The command now resolves the pipeline task from the architecture instead of defaulting to text generation.

  • Fixed LoRA and denoising-cache CLI flags replacing, rather than overriding, the matching --config-file section; --enable-lora=false now also disables LoRA that a recipe enabled, instead of being ignored.

  • Added --indexer-kv-cache-format (config key indexer_kv_cache_format) so MiniMax IndexK can use scale-free float8_e4m3fn independently of --kv-cache-format. The default stays bfloat16; FP8 IndexK is AMD-only.

  • max serve with no --model or --model-path now fails with a one-line usage error that names --model, instead of falling through to Hugging Face validation on an empty path and printing a traceback.

  • Added the experimental --experimental-device-graph-synthesis flag (PipelineRuntimeConfig.experimental_device_graph_synthesis): compiles model graphs with device-graph synthesis, so the compiled model records its kernels into a device graph and replays it on execute. Honored only by architectures that opt in (currently Gemma 4's language graph), and mutually exclusive with device_graph_capture.

Python API​

  • Eager mode tensors now use the JIT by default. This unlocks fusion and shape specialization optimizations even for eager code.

  • Unified the four types that described a tensor argument in max.experimental (TensorType, BufferType, DistributedTensorType, and DistributedBufferType) into two, both built from a dtype, a shape, and a device: max.experimental.sharding.TensorLayout, which a tensor backs with a graph value, and BufferLayout, which it backs with a buffer a compiled callable may store through.

  • Reworked pipeline memory planning around MemoryPlan:

    • Added max.pipelines.lib.MemoryPlan, the result of memory planning when a pipeline is loaded: the effective planned_max_length, planned_max_batch_size, planned_max_batch_total_tokens, KV-cache budget, and device specs the pipeline and its schedulers consume.
    • PipelineModel now requires the memory_plan constructor argument (keyword-only; constructing a pipeline model without a plan raises a TypeError), and PipelineModel.max_seq_len is a read-only view of the plan's planned_max_length rather than a stored copy with a config fallback.
    • Renamed MemoryEstimator.estimate_memory_footprint to MemoryEstimator.plan_from_sizes, after the MemoryPlan it returns. Use MemoryEstimator.plan instead to plan from a PipelineConfig alone; plan_from_sizes is for callers that have already computed the weight, activation, and signal-buffer sizes.
    • The sequence-length rule now runs once, when the config is built: config.model.max_length holds the resolved length and PipelineArgs.max_length keeps what the user asked for. ArchConfig.initialize receives that length instead of deriving it (max_seq_len is now a required keyword argument), and memory planning may only lower it, on the plan.
    • Removed PipelineModel.calculate_max_seq_len, ArchConfigWithAttentionKVCache.user_provided_max_length, and model_max_seq_len; architectures own the rule, so Mistral, Mistral3, and Pixtral now bound max_length on their configs.
    • Memory planning no longer writes its planned max_length and max_batch_total_tokens back onto the pipeline config. After startup, PipelineConfig.model.max_length keeps the construction-resolved value and PipelineConfig.runtime.max_batch_total_tokens keeps the user-provided value (None when unset); the effective values live on MemoryPlan.
    • Made MemoryEstimator.free_memory, static_memory_size, available_kv_cache_memory, and max_supported_sequence_length private. They are steps within a memory plan rather than useful on their own, and the values they produced are now available on MemoryPlan.
    • Removed leftover graph-capture headroom from architecture memory planners (up to 8 GiB per device). Once graph capture ran before planning, the reserved slack reduced on-device KV-cache capacity.

  • max.nn.kernels.msa_sparse_attention_ragged and msa_sparse_attention_ragged_mxfp8 take a required sparse_block_size: the KV block size in tokens from the model's sparse_attention_config. It must equal the KV cache page size, and the kernel now asserts that rather than inferring a block size from the attention tile-width default.

  • max.experimental.nn.Module.compile reuses precompiled MEFs when the session has them, so a ModuleV3 model can be compiled where no accelerator is attached and initialized where one is. max.experimental.support.set_export_mefs records each compiled graph into a directory, and max.experimental.support.set_precompiled_mefs initializes those artifacts instead of compiling. Both helpers accept more than one directory, so artifacts from separate compile jobs can be consumed together. InferenceSession.compile_reusing_mefs is the same half-step for callers that trace a graph and initialize it themselves.

  • max.experimental.sharding.NamedMapping takes its mesh from the enclosing mesh_context() when none is passed, so a layer can name the axis it shards along without being handed a mesh. Its original_spec and original_unreduced properties are removed.

  • Added max.experimental.tree_utils, pytree utilities over nested list / tuple / namedtuple / dict and any class declaring the tree protocol: __tree_flatten__ with either __tree_unflatten__ or __tree_empty__, and an optional __tree_setattr__. There is no registry and no decorator, so a type opts in by declaring the methods. flatten and unflatten carry a value across a flat boundary, leaves, paths, and nodes read it, map builds a new tree, and update writes path-keyed values into an existing one in place. Every walk takes leaf, saying where it stops, and shared, saying whether a value reachable by two paths is one object or two. Import the module as a namespace: from max.experimental import tree_utils as tree.

  • Added max.experimental.compilation, three transforms over plain callables.

    • stage(fn)(*args, **kwargs) traces fn into a max.graph that can be printed and inspected as MLIR. The arguments are fn's own, except that each tensor is given as a TensorType. This partially evaluates fn: the tensor types become graph inputs, and every other argument is evaluated during tracing.
    • compile(fn, weights=...)(*args, **kwargs) stages the same way and compiles the graph; the result is callable on real tensors. Weights and device memory load only on the first call, so export_mef can save the compiled graph to a file without loading either.
    • as_subgraph(fn) returns a drop-in replacement for fn that, during tracing, calls one shared subgraph instead of inlining its body, so a stack of identical layers compiles once.

  • Extended fused collective graph ops with group_size and residuals:

    • max.graph.ops.reduce_scatter_rms_norm takes an optional group_size argument, matching max.graph.ops.reducescatter.sum: the devices split into contiguous groups of that many, each reducing independently, so the fused op also works under tensor-parallel-within-data-parallel topologies. It was previously full-world only and silently disabled itself whenever the tensor-parallel degree was smaller than the device count.
    • max.graph.ops.reduce_scatter_rms_norm now takes an optional residuals argument so the post-attention residual add folds into the collective.
    • max.graph.ops.allgather_rms_norm takes an optional group_size argument, matching max.graph.ops.allgather: the devices split into contiguous groups of that many, each gathering independently, so the fused op also works under tensor-parallel-within-data-parallel topologies. It was previously full-world only.
    • Added max.graph.ops.allgather_rms_norm_quant_mxfp8 and max.graph.ops.allgather_rms_norm_quant_mxfp6, fused all-gather + RMSNorm + MXFP8/MXFP6 quantize graph ops.
    • max.graph.ops.allreduce.sum takes an optional group_size argument so tensor-parallel replicas under data parallelism reduce within contiguous device groups.

  • Changes to Buffer allocation and host access:

    • max.driver.Buffer now implements __str__, so str(buffer) and print(buffer) show the buffer's data formatted like a NumPy array, followed by its dtype, shape, and device. repr(buffer) still returns the metadata-only representation.
    • Added max.driver.Usage, an allocation-intent flag for Buffer. usage=Usage.STAGING requests host memory for staging transfers to and from the given device, which may be page-locked. Usage.UNTRACKED opts out of host-access ordering and is only meaningful with STAGING; DevicePinnedBuffer reports both. Buffer.usage reports the intent, Buffer.pinned whether the memory is page-locked.
    • Host access to a staging buffer (usage=Usage.STAGING on an accelerator) now waits for the copy MAX recorded against that buffer, not for the whole device. Covers to_numpy(), __dlpack__, item(), str(buffer), contiguous(), and indexed assignment.
    • Added max.driver.HostHazardError, raised by a host access when the device work that produced the buffer failed after being queued.

  • max.nn.sampling.AcceptanceSampler and max.nn.sampling.stochastic_acceptance_sampler take a draft_proposal argument. The default, "argmax", is unchanged: the draft proposes deterministically and verification runs typical acceptance. With "sampled", the caller passes the distribution the draft sampled from, so verification runs the real p_target / q_draft ratio test and recovers rejected positions from max(p_target - q_draft, 0); temperature, top-k, and top-p then all apply to the draft-verification distribution, where "argmax" applies only temperature. Sampled mode is GPU-only, needs a static vocab_size, and cannot be combined with relaxed thinking-phase acceptance, whose rule assumes the drafted token is the draft's argmax.

  • enable_dp_cross_replica_prefix_copy now takes effect on the Jenga KV cache, which previously logged that it was ignoring the flag. Under data parallelism a prefix cached on one replica is copied to the replica serving the request, in one batched device-to-device transfer, instead of being recomputed or fetched back through the host tier. The flag defaults to on, so this changes behavior for every data-parallel deployment on this cache: on a multi-turn workload it cut time-to-first-token by around a quarter and left the host tier unused, at a cost in decode latency that shrinks as offered load rises. Set it to false to restore the previous behavior.

  • Added max.driver.begin_launch_trace(), max.driver.take_launch_trace(), and a max.driver.launch_trace() context manager. They record kernel launches, copies, and memsets across all CUDA and HIP streams as an enqueue-ordered list of max.driver.LaunchTraceEntry values.

  • Added max.nn.kernels.mtp_eh_norm, which RMS-normalizes an MTP draft layer's token embedding and target hidden state in one pass.

  • Added DType.float6_e2m3fn and DType.float6_e3m2fn, the OCP MX FP6 storage formats, to max.dtype. Registered MXFP6 block-scaled matmul, grouped matmul, dynamic quantize, and dequantize graph ops.

  • Fixed Graph MLIR text I/O corrupting float64 constants and crashing or writing malformed float32 literals by routing parse and print through max.mlir.

  • An architecture can set checkpoint_draft_width on its registration to supply the draft width its checkpoint was trained for, so users of those models do not have to pass --num-speculative-tokens. A width that disagrees with the checkpoint is replaced, with a warning.

Kernels and GPU programming​

  • The max.gpu package now includes everything previously provided in Mojo's std.gpu package, making it a complete entry point for accelerator programming. A small number of modules need to remain included in the Mojo standard library, but are now private in the standard library (in std._gpu) and public in the MAX accelerator library (max.gpu).

  • Added APIs for unified host and device memory:

    • Added Device.is_host_unified (Python max.driver) and DeviceContext.is_host_unified() (Mojo): return whether a device and the host draw from one physical memory pool. Reports hardware topology, so it does not imply any given buffer is host-readable. Driver plugins answer it through the new optional host_unified device property.
    • Added DeviceBuffer.unsafe_host_ptr() to the Mojo max.gpu.host API. On devices with unified memory (Apple silicon), it returns a CPU-addressable pointer to the buffer, so the host can read a kernel's output after DeviceContext.synchronize() without an enqueue_copy round trip. Reads through it are uncached, so it suits small control records rather than bulk readback. A CPU device returns the buffer's own pointer, since its allocations are host memory already; devices whose memory is not CPU-addressable raise.

  • Improvements to DeviceContext events and Apple GPU copies:

    • DeviceContext.create_event() and DeviceEvent are now supported on Apple GPUs, backed by MTLSharedEvent. Event queries and waits track actual GPU completion instead of command-buffer submission order, and waiting on an event from another context's queue no longer blocks the host thread.
    • DeviceContext.create_event() on NVIDIA GPUs now honors the default disable_timing flag (previously inverted) and recycles events through the driver's event cache instead of growing it on every create/destroy cycle.
    • Device-to-device copies on Apple GPUs no longer race when the source was written on another DeviceStream.
    • MODULAR_DEBUG=device-sync-mode now works on Apple GPUs, where it previously did nothing.

  • Capturing DeviceContext.enqueue_function() now encodes the closure through DevicePassable before launch, matching explicit kernel arguments. Host handles such as DevicePointer reach the device as device addresses rather than raw host bytes.

  • extensibility.foreach now accepts its elementwise body as a runtime argument as well as a compile-time parameter, so a custom op can pass a unified closure with a capture list instead of a capturing one. Call it as foreach[target=target](body, output, ctx), with the body as the first argument. The compile-time-parameter overload is unchanged, so no existing caller needs edits.

  • Improvements to SM100 matmul:

    • SM100 matmuls with an elementwise epilogue no longer leave output columns unwritten when N is not a multiple of 16, such as N=136 or N=776.
    • SM100 bf16 and fp8-input matmuls whose N leaves the output row stride short of TMA's 16-byte alignment, such as a 258-wide MoE router projection, now take the split-K GEMV at up to 64 rows instead of falling back to vendor BLAS.
    • Raised the SM100 matmul automatic pipeline-stage cap from 16 to 24, improving decode-shaped matmuls by a median 4.4% at M≤32 on B200.

  • Improvements to the MLA sparse-attention indexer:

    • The SM100 MLA decode dispatch now enumerates 12, 24, and 48 query heads alongside the powers of two it already covered, so a model whose per-device head count is not a power of two can bind its dispatch metadata.
    • The MLA sparse-attention indexer (DeepSeek V3.2, GLM 5.x) now does work proportional to each row's actual key count instead of the batch's max_cache_length metadata. At the GLM 5.2 MTP decode shape (batch 8, width 6, 76k-token context, 4 heads per rank) with metadata frozen at 1M, one indexer layer drops from 0.89 ms to 0.10 ms on B200, matching its cost at a bound sized to the runtime lengths; shapes without a metadata gap are unchanged except a small fixed per-call cost for the row-bounds clamp (~4% on a batch-256, 4k-context decode).
    • Tuned the SM100 FP8 sparse-attention indexer used by GLM-5.2 and GLM-5.3, about 1.12-1.33x faster on long-context decode and up to about 2.1x faster on chunked continuation behind a cached prefix.
    • Replaced the indexer top-k's per-tile bitonic sort with a histogram radix select, 1.4-6.6× faster at decode and up to 13.9× at prefill. The DSA indexer TopK now runs unordered and non-deterministic.
    • Fused LayerNorm and ragged RoPE on the MLA sparse-attention indexer path (DeepSeek V3.2, GLM 5.x) into a single layer_norm_rope_ragged kernel.
    • Added native 512-wide NoPE row support to the SM100 sparse MLA kernels so GLM-5.3-Flash no longer pads latent KV rows to 576 in global memory.
    • Chunked the MLA FP8 indexer score matrix to a fixed memory budget (512 MB by default, overridable with MLA_INDEX_SCORES_BUDGET_MB).

  • KDA prefill now runs on the chunk-parallel pipeline. The pipeline existed as a Mojo kernel with no graph-op registration, so every prefill fell back to the token-sequential decode recurrence: O(total_seq_len) sequential steps per sequence, with no parallelism to spend on a long prompt. Registering kda_chunk as its own graph op takes that to O(total_seq_len / CHUNK_SIZE).

  • Added MODULAR_APPLE_M5_ALLOW_LOSSY_F32_ATTENTION. Set it to 0 to keep fp32 attention off the Apple M5 MMA, which truncates operands to fp19. It defaults to the fast (lossy) path, matching MODULAR_APPLE_M5_ALLOW_LOSSY_F32_MATMUL.

  • Improvements to AMD MXFP and matmul:

    • Improved MXFP8 block-scaled matmul decode latency for attention output-projection shapes at M=4, M=32, M=64, and M=128 on MI355.
    • Improved MXFP8 block-scaled fused QKV projection decode latency at M=4 on MI355.
    • Retuned the MI355X dispatch table for a grouped block-scaled MoE matmul (gate-up and down projections) at the estimated-total-M > 2048 band that real serving traffic hits, plus the down projection's estimated-total-M <= 2048 band. Gate-up projection speeds up 7.4-10.1% and down projection 18.2-19.6% (etm > 2048) and 6.9-23.3% (etm <= 2048) across real ragged-M, skewed routing scenarios.
    • Added OCP MXFP6 (float6_e2m3fn and float6_e3m2fn) block-scaled GEMM, preshuffled-B, and grouped matmul on AMD CDNA4. Dense decode uses a small split-K tile for M <= 64; decode latency improved up to 6.6x on attention projection shapes.
    • Enabled split-K and re-enabled LDS swizzle for MXFP8 dense matmul on AMD GPUs. Extended split-K past M=64 in the MXFP4/MXFP8 dense dispatcher.
    • Sped up the CDNA4 MXFP8 block-scaled matmul by about 44% on attention output-projection shapes and 21-30% on QKV. Sped up the AMD MXFP8 grouped GEMM prefill band on gfx950 by about 8%.
    • Fused the MXFP8 QKV projection for dense and sparse-attention layers on AMD CDNA4 into one block-scaled GEMM. At TP4 on MI355X the fused GEMM is 1.4-4.8× faster than the previous multi-GEMM path.
    • MXFP4/MXFP8 split-K on AMD can now carry a fused epilogue. Folded the MXFP8 activation quantize into the fused all-gather + RMSNorm kernel.
    • Switched the AMD MXFP8 MoE grouped-GEMM prefill bands to a persistent grid sized by the machine, cutting gate/up and down projection launch time by about 37% and 46%.
    • Added a small-M streaming bf16 matmul for MI355X vocabulary heads (M≤16, large N). Widened the AMD router mixed-GEMV decode path to M≤64.

  • Improvements to token sampling:

    • Sped up GPU token sampling by about 4% per output token when the largest top_k in the batch is below 10, by removing a device synchronize from fused_token_sampling_gpu. The synchronize backed a check that raised on an all-NaN logits row. Such a row now yields an arbitrary in-range token rather than an error. Set max-debug.assert-level to all to restore the check, or use max-debug.nan-check to locate NaN logits.
    • The joint top-k/top-p sampling kernel can now also return the masked, renormalized distribution it drew from, exposed as max.nn.kernels.topk_fused_sampling_with_dist. Speculative decoding needs that distribution to build a rejection residual, and reads the sampled token's own probability out of it—a value that has to agree with the sampler's accept decision, so it comes from the sampling kernel rather than a separate softmax. When top-k, top-p, and min-p are disabled, the distribution-producing path now skips its cutoff search. The existing single-output path is unchanged. On AMD GPUs, the distribution output also serves as temporary storage for exponentiated logits during sampling.
    • Added max.nn.kernels.topk_topp_masked_probs, which computes a row's top-k/top-p masked renormalized softmax without sampling and without a sort. Speculative decoding verification reads the target's masked probability of each drafted token and builds its rejection residual from this one tensor, in the same form the draft sampler emits its proposal distribution. When top-k and top-p are disabled, the kernel now skips the cutoff search because every positive-probability token already survives. On AMD GPUs, it also caches exponentiated logits in the output buffer so cutoff-search passes do not recompute them. Rows with top-k disabled also omit positive-value counting from the initial mass reduction and cutoff search.
    • Top-p-only distribution kernels bias cutoff-search pivots toward lower weights when the retained-mass budget is large relative to the mass still above the search's low bound, so the gain follows the bracket state rather than the requested top_p.
    • The fused gumbel-argmax sampling kernel takes a from_probs parameter, exposed as max.nn.kernels.gumbel_argmax_from_probs: each row's score is ln(p) + gumbel over unnormalized probabilities, drawn with noise the kernel generates from a per-row seed. This enables sampling a speculative decoding rejection residual max(p_target - q_draft, 0) that the caller builds in graph ops. GPU-only, non-Apple.
    • Improved wide-row FP32 Gumbel sampling performance on AMD GPUs.
    • Improved AMD GPU top-k/top-p sampling at decode batch sizes by splitting each row across a block group sized to the device.

  • Improvements to MoE and expert-parallel kernels:

    • Fixed expert-parallel dispatch dropping half of every token belonging to an expert that only one communication SM serves, which surfaced as NaN logits.
    • The SM100 grouped block-scaled matmul accepts MXFP4 weights against MXFP8 activations (W4A8), so a quantized MoE can feed its packed 4-bit experts straight to the tensor cores rather than dequantizing them to bfloat16 first. This removes MAX's per-forward mxfp4_dequant over the routed expert stack, and it keeps the weights at their 4-bit footprint in global memory, which matters most at expert counts where a bfloat16 copy of the stack does not fit. A new unpack_fp4 option on the NVIDIA TMA descriptor helpers, backed by the TensorMapDataType.PACKED_FP4_ALIGN16B tensor-map type, pads the weights into the byte-addressed form the tensor cores read as the copy engine lands them in shared memory.
    • Rebuilt MoE routing-index construction as a single-CTA kernel, making the routing step 1.8× to 7.9× faster.
    • Enabled the fused SwiGLU+NVFP4 grouped matmul on interleaved tensor-parallel MoE.
    • Sped up grouped MoE NVFP4 quantization by up to 16× at low batch on B200.
    • Fused expert-parallel dispatch with the MegaFFN body into one launch, then wrote received tokens directly into the final contiguous expert layout. Faster than the prior two-kernel PDL baseline across EP serving cells (about 1.3% to 7.8%).

  • Improvements to relative-logits and Gemma 4 attention:

    • Sped up RelativeLogitsMask attention decode by up to 23× by allowing the SM100 FA4 one-query path to use split-K.
    • Added a relative-logits mask to ragged paged flash attention. flash_attention_ragged now takes an optional rel_logits bias table selected by query-key distance.
    • Sped up Gemma 4 decode attention on B200 by forwarding d=256 and d=512 heads through FA4, about 2.7-4.8x faster across representative decode shapes.

  • Improvements to AMD attention:

    • AMD attention paths now support an FP8 KV cache. The AMD MHA decode token fold now admits FP8 at head depth 128 and widens to 7 speculative tokens.
    • Flattened AMD MHA decode cost across speculative verify widths 3-8, cutting kernel time by about 10-60% at those widths on MI355.
    • Sped up AMD MSA decode scoring by staging index-K through wave-private LDS tiles, reducing production width-4 latency by about 22%.

  • Sped up reductions:

    • Sped up GPU reductions over short non-innermost axes—for example summing a [16, 8, 2048] tensor over the middle axis—from 30.26 µs to 3.40 µs on B200.
    • Sped up the rowwise split-K reduction tier on wide rows, making vocabulary-sized argmax 2.3x to 6.1x faster on MI355X.

  • Added a relay-assisted grouped allgather and reduce-scatter that uses otherwise-idle inter-group links, speeding up those collectives by about 1.4-1.8x on 8x MI355X.

Breaking changes​

  • Removed the pinned= argument to Buffer(...) and Buffer.zeros(...). Use usage=Usage.STAGING instead.

  • Removed the NPU device class from max.driver and the corresponding DeviceRef.NPU(), DeviceRef.is_npu(), and DeviceKind.NPU from max.graph, along with the M_newNPUDevice() C API entry point. NPU was a thin subclass of Accelerator that differed only in the device label it stamped on the graph; it had no callers, and accelerator backends reached through a driver plugin are already served by Accelerator. Construct Accelerator() (or DeviceRef.GPU()) for any non-CPU device, and read the Accelerator.api property to tell the concrete backends apart.

  • The tile-tensor storage policy is renamed to an engine, and the layout.tensor_storage module is renamed layout.tensor_engine. The TensorStorage trait becomes TensorEngine, TileTensor's Storage parameter becomes Engine, and the conforming policies PointerStorage, DevicePointerStorage, and StaticOffsetStorage become DefaultEngine, DevicePointerEngine, and StaticOffsetEngine. The trait describes the operations a tile tensor performs on its handle (load, store, bitcast, elementwise) rather than the memory it points at, so the old name described the wrong thing. Update Storage= keyword arguments to Engine= and any tensor.Storage accesses to tensor.Engine. The TensorOps trait and the associated StorageType handle keep their names, since they still describe the borrowed memory itself.

    Kernel signatures follow. Every comptime parameter bound to TensorEngine or TensorOps now ends in Engine, replacing the three spellings that were in use: OutputStorage and XStorage become OutputEngine and XEngine, QStorageType and SeedStorageType become QEngine and SeedEngine, and the snake_case q_storage and x_store become q_engine and x_engine. Callers passing any of these by keyword need to update the name.

  • Changes to the KV cache connector:

    • The KV connector's external host and disk tiers now report occupancy and transfer volume in bytes rather than in blocks. Those tiers are byte budgets the operator sizes in bytes (host_offload_max_gb, disk_offload_max_gb), their block width need not match the device's, and bytes rate directly against PCIe and disk bandwidth. The device (G0) cache is unchanged and still reports blocks.

      KVConnector replaces host_block_count / disk_block_count with host_byte_count / disk_byte_count, returning a new ByteCount (the same free / total / used / used_pct / free_pct surface as BlockCount, measured in bytes). The KV cache managers make the same swap; block_count() is untouched. KVCacheMetrics renames h2d_blocks_copied, d2h_blocks_copied, disk_blocks_read, and disk_blocks_written to h2d_bytes_copied, d2h_bytes_copied, disk_bytes_read, and disk_bytes_written.

      The exported metrics follow: maxserve.cache.h2d_blocks_copied, maxserve.cache.d2h_blocks_copied, maxserve.cache.disk_blocks_read, and maxserve.cache.disk_blocks_written become h2d_bytes_copied, d2h_bytes_copied, disk_bytes_read, and disk_bytes_written, with unit bytes. maxserve.cache.used_host_kv_pct and maxserve.cache.used_disk_kv_pct keep their names and are now computed over bytes. Dashboards and alerts on the old tier counter names need updating.

    • The KV cache connector is now configured as a single object: its type moved onto --kv-connector-config as a type field, and the separate --kv-connector flag is removed. Replace --kv-connector rust_tiered with --kv-connector-config '{"type": "rust_tiered"}', and in a recipe set model.kv_cache.kv_connector_config.type. host_kvcache_swap_space_gb is renamed host_offload_max_gb to match disk_offload_max_gb, and both now default to sizing their tier from the device page pool (1.5 times it on host, twice on disk) rather than to a fixed 50 GiB. Dict-valued kv_cache flags now merge field-wise over a config file's value instead of replacing it, so overriding one connector field on the command line keeps the rest—previously a partial override reset the connector type and silently disabled offloading.

    • Removed the Python local and tiered KV connectors (LocalConnector, TieredConnector). tiered now aliases rust_tiered; the local connector type is gone.

    • Removed KVCacheConfig.allow_kv_head_replication, the architecture registration field requires_kv_head_replication, and the --allow-kv-head-replication flag. An architecture now asks for KV head replication in its construct_kv_params().

  • Config objects are now immutable after construction:

    • The pipeline configs are now immutable: PipelineArgs, PipelineConfig, PipelineRuntimeConfig, SamplingConfig, MAXModelConfig, KVCacheConfig, and its nested KVConnectorConfig, LoRAConfig, and ProfilingConfig. Assigning to a field after construction raises a Pydantic ValidationError. Construct them with the values you need.

    • SpeculativeConfig is now immutable: assigning to a field after construction raises a Pydantic ValidationError. Construct it with the values you need. A failed speculative target-architecture rewrite now raises from PipelineConfig.from_args() instead of being logged and ignored.

    • ModelManifest is now immutable from construction: mutating the mapping (item assignment, update, pop, and so on) raises a TypeError, and ModelManifest.resolve() is removed—a manifest is complete when built. Construct it with the component configs you need. The unused total_weights_size property is also removed.

  • Reworked pipeline construction onto a single nested config path and a MemoryPlan:

    • Reworked max.pipelines.PipelineArgs and PipelineConfig construction around a single path and a single (nested) shape:

      • PipelineArgs now nests its runtime, sampling, and profiling fields in runtime, sampling, and profiling sub-configs (PipelineRuntimeConfig, SamplingConfig, and ProfilingConfig), matching the nested shape already used by recipes and PipelineConfig. Flat constructor kwargs for those fields (for example max_batch_size=1) are rejected; pass runtime=PipelineRuntimeConfig(max_batch_size=1) instead, and use the nested keys in config files validated into PipelineArgs. PipelineArgs.from_flat_kwargs (the CLI path) still accepts the flat spellings and routes them to the sub-configs.
      • Removed PipelineConfig.from_flat_kwargs and PipelineArgs.from_pipeline_config; PipelineConfig.from_args is the single way to construct a PipelineConfig from user input. Replace PipelineConfig.from_flat_kwargs(...) with PipelineConfig.from_args(PipelineArgs.from_flat_kwargs(...)).
      • PipelineConfig.from_args now also applies the model generation config's sampling defaults, applies --model-override entries, and resolves the speculative draft architecture, so programmatically constructed PipelineArgs behave the same as CLI invocations.
      • PipelineRuntimeConfig is now exported from max.pipelines.

    • Removed PipelineConfig.resolve(). Resolution now runs inside PipelineConfig.from_args(); a constructed config is already complete.

    • Constructing a MAXModelConfig directly now only validates the fields you pass. It no longer fills in the weight and model paths or loads the Hugging Face config. Configs the pipeline builds are unchanged.

    • PipelineRegistry.retrieve_factory now returns a RetrievedPipeline dataclass with tokenizer, factory, and memory_plan fields instead of a (tokenizer, factory) tuple, so callers can reach the memory plan computed during retrieval. Replace tuple unpacking with attribute access. PipelineRegistry.retrieve is unchanged.

    • The serving surface now reads the planned sequence length and batch token budget from the memory plan instead of re-reading them from the pipeline config. TokenGenerationSchedulerConfig.from_pipeline_config, start_model_worker, the scheduler loaders, and the startup log helpers (log_basic_config, log_pipeline_info) take the memory plan as a parameter. Resolved values are unchanged.

    • Renamed MemoryPlan.max_length to MemoryPlan.planned_max_length to distinguish the plan's value from the user intent on PipelineArgs.max_length and the construction-resolved PipelineConfig.model.max_length, which keep their names. Also renamed MemoryPlan.max_batch_size to planned_max_batch_size and MemoryPlan.max_batch_total_tokens to planned_max_batch_total_tokens.

    • ArchConfig.calculate_max_seq_len() no longer takes pipeline_config, and model_config is now required.

  • Renamed max.driver.DeviceStream to DeviceQueue and Device.default_stream to Device.default_queue; the old names were removed. The driver models work submission as a command queue; a stream is one backend's implementation of that queue. Method, property, and argument names (Buffer.stream, stream=, native_stream_handle) are unchanged.

  • --max-vision-cache-entries is replaced by --vision-cache-utilization, a fraction of the KV cache pool budget for the vision encoder cache (default 0.05; 0 disables caching). The cache is block-based, so an entry count no longer describes its capacity; configs setting the old flag must convert to a pool fraction.

  • The legacy alias-buffer LoRA path has been removed. ModuleV3 LoRA (adapters passed as graph inputs) is now the only supported LoRA implementation. Serving a non-ModuleV3 architecture with --lora-paths now raises a clear error at startup instead of building a manager that never applies the adapters; serve the model's ModuleV3 variant (for example, --prefer-module-v3) to use LoRA adapters.

  • Denoising-cache input is now a frozen DenoisingCacheSettings on PipelineArgs (denoising_cache; in config files this section moves from runtime.denoising_cache to the top level). Construction fills unset fields from the architecture's TaylorSeer defaults into a frozen DenoisingCacheConfig. Enabling TaylorSeer without resolvable tuning fails at construction, as does enabling TaylorSeer and first-block caching together.

  • Removed the experimental --fold-sampler-into-graph option and the max-pending-futures schedule-ahead config from the overlap serve path.

  • Closures are now passed as runtime values instead of compile-time capturing parameters:

    • Removed the parametric max.benchmark.bencher_iter_custom[fn](bencher, ctx) overloads and unused bencher_iter_custom_multicontext(). Pass the launch closure as a value: bencher_iter_custom(bencher, fn, ctx).

    • Removed max.algorithm.reduce_boolean(), which took its reduce_fn and continue_fn as capturing compile-time parameters and had no callers. Use max.algorithm.reduce() with a boolean accumulator, or write the early-exit loop directly.

    • Removed the parametric max.algorithm.parallelize[func](num_work_items, ...) and max.algorithm.parallelize_over_rows[func](shape, axis, grain_size, ...) overloads that took a capturing closure as a compile-time parameter. Pass the body as a unified closure in the first runtime argument instead: parallelize(func, num_work_items, ...) and parallelize_over_rows(func, shape, axis, grain_size, ...). Closure bodies drop @__parameter / @__copy_capture in favor of an explicit capture list, for example def body(start: Int, end: Int) {imm}:.

    • Removed the parametric max.algorithm.sync_parallelize[func](num_work_items, ...) overload that took a capturing closure as a compile-time parameter. Pass the body as a unified closure in the first runtime argument instead: sync_parallelize(func, num_work_items, ...). The remaining overload accepts def(Int) raises -> None, so both raising and non-raising closures bind. Closure bodies drop @__parameter / @__copy_capture in favor of an explicit capture list, for example def body(i: Int) {imm}:.

    • Removed the parametric max.benchmark.bench_multicontext[fn](bench, ctxs, ...) overload. Pass the body as a unified closure in the second runtime argument: bench_multicontext(bench, fn, ctxs, ...). Nested closures passed this way drop @__parameter in favor of an explicit capture list such as {imm} or {mut buf, imm}.

    • Removed the parametric capturing overloads of DeviceContext.execution_time[fn](num_iters), DeviceContext.execution_time_iter[fn](num_iters), and DeviceContext.enqueue_cpu_function[fn](). Pass the closure as a runtime argument instead: execution_time(fn, num_iters), execution_time_iter(fn, num_iters), and enqueue_cpu_function(fn). Nested closures passed this way are unified closures, so replace @__parameter and @__copy_capture(x) with an explicit capture list such as {imm} or {var x, imm}.

    • Removed the parametric capturing layout.int_tuple.apply[func](t), reduce[reducer](t, initializer), and capturing apply_zip[func](...) overloads. Pass the closure as a runtime value: apply(t, func), reduce(t, initializer, reducer), and apply_zip(t1, t2, func) (or apply_zip(t1, t2, t3, func)). Nested closures passed this way are unified closures, so replace @__parameter with an explicit capture list such as {} or {imm}. Thin apply_zip[func](t1, t2) function-pointer overloads are unchanged.

    • DeviceGraphBuilder.add_function[kernel](*args, ...) takes a thin function pointer (func: def(...) thin -> None), the same identity as DeviceContext.compile_function[kernel]().

Fixes​

  • Fixed an out-of-range top_logprobs killing the model worker and taking the server down with it. Both the chat top_logprobs and the legacy /v1/completions logprobs count are now bounded at the request boundary and return HTTP 400 with the supported range.

  • Fixed a pre-tokenized prompt longer than --max-length killing the model worker instead of being rejected.

  • Fixed constrained decoding producing invalid output when combined with speculative decoding on AMD GPUs.

  • Fixed a model worker crash when constrained decoding and speculative decoding were enabled together.

  • Fixed constrained decoding applying a grammar bitmask in a way that could restore tokens the model had already masked, including ids outside the tokenizer's range.

  • Fixed a race that enforced structured-output grammars during a reasoning model's thinking span.

  • Fixed structured output and constrained tool calling being silently ignored on the Kimi K2.5-family pipelines when serving with DFlash speculative decoding (--speculative-method dflash).

  • Fixed tool calls being returned as raw markup in the assistant's content when a request did not declare a tools array. A tool established only by the conversation history, such as retrying a call that previously failed, now comes back as a structured tool_calls entry. Parsing runs whenever the model has a tool parser configured; tool_choice="none" still opts out.

  • Fixed tool-call requests failing with HTTP 400 (anyOf branch and base schema both set "description") on models whose grammar compiles in strict mode (GLM-5.x, Gemma 4). The xgrammar JSON-schema converter's anyOf base-merge now skips annotation-only keywords (description, title, default, examples, $comment, deprecated, readOnly, writeOnly) instead of rejecting them as branch/base conflicts; they carry no grammar constraint.

  • Fixed max-debug.source-tracebacks (for example MODULAR_DEBUG=source-tracebacks or Graph.debug.source_tracebacks = True) being silently ignored when it was enabled after max.graph was first imported.

  • Fixed the disk_bytes_written KV cache metric counting blocks the tiered connector's disk tier declined to write because they were already saved or had a write pending.

  • Fixed overlap-scheduling batch logs and the maxserve.batch_terminated_reqs metric attributing the completed batch's token counts and type to the newly enqueued batch.

  • Fixed a CUDA_ERROR_MISALIGNED_ADDRESS crash that could occur when concatenating tensors with byte sizes that do not satisfy the device's preferred alignment, including rank-1 index vectors with odd lengths.

  • Fixed ops.slice views claiming an alignment their non-innermost stride did not satisfy.

  • Fixed generate_async raising KeyError: Request ID not found in replica batch when requests in one batch finish on different steps, which happens whenever they are given different max_new_tokens.

  • Fixed the offline generate() and generate_async() APIs releasing only a finished request's KV cache blocks, and never the pipeline itself.

  • Fixed DeviceExternalFunction crashing on Metal instead of launching, so separately compiled kernels now load and launch there as they already did on other GPU backends.

  • Fixed run-to-run nondeterminism of layer_norm, rms_norm, and other Row-API rowwise reductions on Apple silicon GPUs: Model outputs on Metal (for example FLUX.2 image generation) are now byte-identical across runs; NVIDIA and AMD codegen is unchanged.

  • On Apple silicon, a missing Metal toolchain (a separate download since Xcode 16) now surfaces xcrun's own error, which names the fix (xcodebuild -downloadComponent MetalToolchain), instead of the opaque "Please submit a bug report." message.

  • Fixed bfloat16 math ops (sqrt, rsqrt, pow, sin, cos, log10, and log1p) failing to compile on Apple GPUs.

  • Fixed device buffer allocation no longer being pooled on GPUs without GPUDirect RDMA support, such as GeForce cards.

  • Fixed abandoned image, video, and audio generation requests still being rendered. A request cancelled before it starts is now dropped and answered as cancelled; one already in flight still runs to completion, since a render is a single uninterruptible call.

  • Fixed reductions over a zero-extent axis—for example ops.sum(x, axis=1) where that axis has length 0—leaving their output unwritten, along with anything fused into the reduction's epilogue. Each now writes its identity: 0 for sum, 1 for prod, the dtype's minimum for max and its maximum for min, index 0 for argmax and argmin, and NaN for floating-point mean (as numpy.mean reports). Integer mean returns 0. Note that max, min, argmax, and argmin return an identity here rather than raising the way NumPy does.

  • Fixed CPU argmax/argmin reductions returning a wrong index for reduce axes of 256K+ elements, for example an argmax over a [1, 2097152] tensor, where the row's reduction fans out across multiple CPU workers.

  • Fixed max benchmark --base-url failing before the first request against remote OpenAI-compatible endpoints: the server-readiness probe and the prefix-cache flush now target the --base-url endpoint (instead of http://<host>:<port>) and send Authorization: Bearer $OPENAI_API_KEY, matching the benchmark requests themselves.

  • Fixed GPU discovery inside a container granted only MIG compute instances, which made MAX and Mojo unusable on MIG-sliced clusters. Discovery reported GPU is not present, and a container holding several instances carved from the same GPU saw only one of them. Where NVML answers for the parent GPU, discovery now defers to CUDA, which describes the instance: for a device's memory when NVML rejects the query, and for the device count when MIG is enabled. (Issue #6896)

  • Fixed GPT-OSS, OLMo 3, and OLMo 2 ignoring --max-length.

  • Fixed DeepSeek-V3.2 and GLM-5.x pipelines ignoring --max-length. These models now size their rotary-embedding tables from the resolved maximum sequence length instead of the checkpoint's max_position_embeddings.

  • Fixed ops.group_norm() raising NotImplementedError in eager mode on CPU. group_norm previously had a GPU-only kernel; it now has a CPU compute path too, so eager group_norm runs on CPU the same way layer_norm/rms_norm already do.

  • Fixed the BF16 Expert Parallelism (EP) dispatch path failing to compile. The BF16 branch now sets dispatch_scale_dtype = float32 to match the kernel signature.

  • Fixed the distribution the top-k/top-p sampler emits for speculative decoding (emit_dist) being under-normalized when a min_p mask removes weight and the row passes top-p at the first trial.

  • Fixed top-p sampling at top_p=1 dropping tokens when rounding made retained mass look slightly over budget.

  • Fixed speculative decoding reusing the same sampling seed across draft steps and residual recovery when draft_proposal is sampled.

  • Fixed compiling FLUX.2 for a virtual device (as max warm-cache does) rejecting a valid checkpoint. The VAE BatchNorm guard no longer treats uninitialized virtual-device weights as all-zero statistics.

  • Fixed chat-completion usage.completion_tokens reporting 0 when a reasoning model's only generated token was a stripped delimiter at max_tokens=1.

  • Fixed --custom-architectures being overwritten when a built-in architecture of the same name registered lazily later.

Mojo language​

For all the updates to the Mojo language, standard library, and tools, see the Mojo release notes.

Models

Cookbook

Puzzles

Blog

Community