For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
MAX nightly
This version is still a work in progress.
Highlights
Documentation
Models
MAX framework
Inference server
Server metrics
max CLI
Python API
C API
Kernels and GPU programming
Breaking changes
-
Deprecated
LayoutTensorandLayoutTensorIterin thelayoutpackage. UseTileTensorandTileTensorIterinstead. -
Removed the
LayoutTensoroverloads ofsum()andmax()fromlayout.mathand ofcreate_tensor_tile()fromlayout.tma_async. Use theTileTensoroverloads instead. -
TMATensorTile,TMATensorTileArray,TMATensorTileIm2col, andSplitLastDimTMATensorTilenow take their shapes as flatCoordvalues instead ofIndexList[rank], and no longer take a separaterankparameter. -
The
rms_normfunctions and GPU kernels innn.normalization, the KV cacherms_normops, andgemv_and_partial_normnow takeepsilonandweight_offsetasFloat32compile-time parameters instead of runtime arguments. -
Removed the
extensibility.foreach[func, target=...](output, ctx)overloads that took acapturingclosure as a compile-time parameter. Pass the body as a closure in the first runtime argument instead:foreach[target=target](body, output, ctx). -
Removed the
--rejection-sampling-strategyflag, theSpeculativeConfigfield it set, and theRejectionSamplingStrategytype. The flag had no effect, because each architecture's acceptance sampler chooses the acceptance rule. -
Renamed the
--eplb-replicas-per-gpuoption to--eplb-redundant-experts-per-gpu(eplb_redundant_experts_per_gpuin config files). The value and its effect are unchanged. -
Removed the
mojo_tieredKV connector type. Userust_tieredinstead. A--kv-connector-configthat setsmojo_tierednow fails with an error. -
The legacy KV cache manager (
MODULAR_USE_LEGACY_KV_CACHE=1) no longer supports KV connectors or disaggregated inference, and requesting either now fails at startup. Disaggregated inference with a cache that has non-full-attention layers also fails instead of falling back to the legacy manager. -
max benchmarkno longer strips a leading@from a file path, whether it is passed to--response-format,--tools,--extra-bodyor--agentic-tool-profilesor named in a--workload-configYAML. Pass the bare path instead:--tools path/to/tools.json. -
Removed the
DeviceQueueoverloads ofmax.driver.Buffer.to()andBuffer.copy(). Pass aDevice(or, forto(), a sequence of devices) instead:buffer.to(device)andbuffer.copy(device=device). -
DSpark speculative decoding now has its own method: serve a DSpark draft with
--speculative-method dsparkinstead ofdflash. DSpark drafts a block at a time like DFlash but runs a different draft step, so a DSpark draft under any other method now fails at startup with a message naming the method to use. -
Changed shape functions registered with
@extensibility.register_shape_function()to require tensor-trait arguments. UseSome[Tensor]for shapes,Some[TileTensorable]for element values, orTensorTuplefor variadic inputs. Legacy tensor aliases remain supported onexecute(). -
Removed
max.pipelines.GenerateMixinand thegenerate()andgenerate_async()methods it gave text generation pipelines. Build contexts withpipeline.tokenizer.new_context()and run them withGenerateIterator.from_contexts(), or pass token ids tomax.pipelines.lib.generate.generate().TextGenerationPipelineInterfacenow declarespipeline_configandtokenizeras abstract properties, so custom subclasses must implement them. -
max.gpu.primitives.warp.reduce()andlane_group_reduce()now take the reduction function as a runtime closure argument instead of a compile-timecapturingparameter. The shuffle function stays a compile-time parameter:def add[dtype: DType, width: SIMDLength]( x: SIMD[dtype, width], y: SIMD[dtype, width] ) -> SIMD[dtype, width]: return x + y warp.reduce[shuffle_down](val, add) # was: warp.reduce[shuffle_down, add](val) -
The reduction callbacks in the
max.algorithmpackage now take aCoordinstead of anIndexList[rank], and no longer carry arankparameter. This changes theinput_fnandoutput_fncontracts ofsum(),min(),max(),mean()andproduct(), along withrow_reduce()andreduce_launch()in the GPU backend. The coordinate holds the same values as before, so results are unchanged:# was: def input_fn[width: Int, rank: Int](idx: IndexList[rank]) -> ... def input_fn[width: Int](idx: Coord) -> SIMD[dtype, width]: return input.load[width=width](idx) -
Text generation now stops on every
eos_token_idin a model'sgeneration_config.json, as Hugging Facegeneratedoes, in addition to the tokenizer's EOS token and theeos_token_idinconfig.json. This applies toTextTokenizer,TextAndVisionTokenizer, and the architecture tokenizers built on them. Before, those extra ids were only a default forstop_token_ids, so a request that set its ownstop_token_ids, or that bypassed the model's sampling defaults, could generate past them. Models whosegeneration_config.jsonlists more EOS ids thanconfig.json(Llama 3.1 Instruct, for example) may now end some responses sooner.stop_token_idsstill adds to the set, andignore_eosstill disables all of it. -
TensorEnginedrops itsloadandstorerequirements, along with theDefaultEngine,DevicePointerEngine, andStaticOffsetEngineimplementations. Loads and stores go through the raw pointerunsafe_ptrreturns:Engine.load[width=w, alignment=a](storage, offset)becomesEngine.unsafe_ptr(storage).load[width=w, alignment=a](offset), andTileTensor.unsafe_ptr()exposes the same pointer at the tensor level. -
TileTensor's two runtime slicing methods are replaced by subscript syntax, and theAllmarker they used is removed along with them.t.slice(batch, All, All)becomest[batch, :, :]andt.slice((0, h), (0, w))becomest[0:h, 0:w].Allwas aCoordLikethat stood for a dimension rather than a coordinate, which is whyCoordcarried acontains_slicesmember to reject it; both are gone. Where the indices are compile-time values, preferslice[](), which keeps the view fully static. -
MAX_SERVE_TRACE_PREFILL_BATCHis renamedMAX_SERVE_TRACE_BATCHand now brackets the model-execute call on the decode scheduler as well as prefill. A dispatch line with no matching completion identifies which worker is stuck inside one call, which the prefill-only form could not do for a decode-side stall. Log lines name the role, asDispatching decode batch: ..., and an execute that raises closes its bracket withFailed ... batchso it is distinguishable from one that never returned. -
KVCacheMetricsdropsnixl_read_blocks_local,nixl_read_blocks_remote, and theremote_read_ratioproperty computed over them. No code path ever populated either field, so the ratio returned0.0for every caller. Six populated counters are added in their place:dkv_peer_attaches,dkv_peer_attach_failures,dkv_peers_dropped,dkv_peer_loads,dkv_peer_load_failures, anddkv_hints_rejected. They are zero unless the external KV-cache connector is in use. -
max.nn.moe.MoEandMoEQuantizedtake the expert activation as oneactivation=argument, aGatedActivationfrommax.nn.activation, in place ofuse_swigluoai,swiglu_alpha,swiglu_limit, andgated_activation_fn. Replaceuse_swigluoai=True, swiglu_alpha=a, swiglu_limit=lwithactivation=GatedActivation("swigluoai", alpha=a, limit=l), andgated_activation_fn=fnwithactivation=GatedActivation(gated_activation_fn=fn). The fused SwiGLU kernel wrappersmax.nn.kernels.grouped_matmul_blocked_swigluandgrouped_dynamic_block_scaled_matmul_amd_swiglu_quant, andmax.nn.comm.ep.ep_kernels.fused_siluandfused_silu_quantized, take the sameactivation=argument. -
max.driver.DevicePinnedBufferis removed. AllocateBuffer(dtype, shape, device=gpu, usage=Usage.STAGING | Usage.UNTRACKED)instead;UNTRACKEDkeeps the old behavior, where reads never wait on device copies. Replaceisinstance(b, DevicePinnedBuffer)withb.pinnedto check for page-locked memory, orUsage.UNTRACKED in b.usageto check whether reads skip the hazard wait. -
MAX Serve now exports spans only when
OTEL_EXPORTER_OTLP_TRACES_ENDPOINTis set. It no longer sends them to Modular's collector by default, andOTEL_EXPORTER_OTLP_ENDPOINTalone no longer turns tracing on. To keep exporting spans, setOTEL_EXPORTER_OTLP_TRACES_ENDPOINT, for example tohttp://collector:4318/v1/traces. -
The
//operator onTensorValueandmax.experimental.tensor.Tensornow matchesops.floor_div: floor division of two integer tensors returns an integer tensor of the operand dtype instead offloat64. Float operands are unaffected. Code that relied on the float result should cast explicitly, for example(x // n).cast(DType.float64). -
MAX now builds against NIXL 1.5.0, whose libfabric (EFA) transport adds a mandatory connection handshake that a NIXL 1.3.0 peer never answers. A MAX built on 1.5.0 cannot exchange KV cache transfers over libfabric with one built on 1.3.0, so disaggregated prefill and decode have to be upgraded together.
-
The deprecated
max.kv_cachemodule is removed. It only re-exportedmax.pipelines.kv_cacheand emitted aDeprecationWarningon import. ImportPagedKVCacheManager,DummyKVCache,load_kv_manager, and the other names it re-exported frommax.pipelines.kv_cacheinstead. -
The
sha256_64value ofkv_cache_hash_algois removed, and a config that sets it now fails validation at startup. It kept the first 8 bytes of each block's SHA-256 digest but chained the next block onto the full digest, so a request whose hash chain grew after its first block (decode past a block boundary, chunked prefill) raised an error. Usesha256instead. The dKV connector keyssha256blocks by the first 8 bytes of the digest, so dKV keys are unchanged.
C API
Kernels and GPU programming
- The deprecated
LayoutTensorandLayoutTensorItertypes have been removed from thelayoutpackage, along with theto_layout_tensor()andlt_to_tt()conversions. UseTileTensorandTileTensorIterinstead.ThreadScopenow lives inlayout.tile_io.
Tools
- Added the ability to install
max-serveandmax-benchmarkas Conda extras, usingmax[extras=[serve]]andmax[extras=[benchmark]]respectively.max[extras=[all]]installs both plusmojo.
Fixes
Mojo language
For all the updates to the Mojo language, standard library, and tools, see the Mojo release notes.