For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Preloads and compiles the model to optimize initialization time by:

• Pre-compiling models before deployment
• Warming up the Hugging Face cache

This command is useful to run before serving a model.

For example, compile and cache a model hosted on Hugging Face:

max warm-cache \
  --model google/gemma-3-12b-it

To compile for a target API and architecture without requiring matching physical hardware, pass --target (for example, cuda, cuda:sm_90, or hip:gfx942). MAX uses virtual devices for the compilation, which is useful when building MEF caches on a CI host that doesn't have the deployment hardware attached:

max warm-cache \
  --model google/gemma-3-12b-it \
  --target cuda:sm_90

The Modular Executable Format (MEF) is platform independent, but the serialized cache (MEF files) produced during compilation is platform-dependent. This is because:

• Platform-dependent optimizations happen during compilation.
• Fallback operations assume a particular runtime environment.

Weight transformations and hashing during MEF caching can impact performance. While efforts to improve this through weight externalization are ongoing, compiled MEF files remain platform-specific and are not generally portable.

max warm-cache

Usage

max warm-cache [OPTIONS]

Options

• --allow-extra-request-fields, --no-allow-extra-request-fields

  When True, unknown top-level fields on OpenAI-compatible request bodies are dropped with a warning before pydantic validation, instead of producing a 400.

• --allow-kv-head-replication, --no-allow-kv-head-replication

  Allow TP wider than the KV head count by replicating each KV head across a group of devices. Used as the default for to_params(allow_kv_head_replication=...) so it reaches base-class paths that don't thread the flag. Only for architectures whose attention shards K/V projections to match.

• --allow-unsupported-logprobs, --no-allow-unsupported-logprobs

  When True, OpenAI-compatible requests that ask for logprobs against a runtime configuration that cannot honor them will raise a warning, and served as if logprobs were not requested. Each response chunk carries logprobs: null. When False (default), such requests are rejected with a 400.

• --ce-delay-ms <ce_delay_ms>

  Duration of scheduler sleep prior to starting a prefill batch.

• --chat-template <chat_template>

  Optional custom chat template to override the one shipped with the Hugging Face model config. If a path is provided, the file is read lazily by the registry when building the tokenizer. If None, the model's default chat template is used.

• --chunked-prefill-min-chunk-size <chunked_prefill_min_chunk_size>

  Floor, in tokens, on any chunk created by chunked prefill. When splitting a request against the CE token budget, the cut is moved earlier so that neither the chunk nor the remainder is smaller than this; if no legal cut point exists within the remaining budget, the request is left unsplit for a later step. 0 (default) disables the floor: cuts land exactly on the budget boundary, which can produce very small chunks. Values above max_batch_input_tokens / 2 forbid most splits; a sane range is roughly 64-1024.

• --config-file <config_file>

• --custom-architectures <custom_architectures>

  Custom architecture implementations to register. Each input is either a path to a single custom-architecture module directory or an IMPORT_PATH:MODULE_NAME colon-form. Each module must expose a top-level ARCHITECTURES list of SupportedArchitecture instances.

• --data-parallel-degree <data_parallel_degree>

  Data-parallelism parameter. The degree to which the model is replicated is dependent on the model type.

• --debug-verify-replay, --no-debug-verify-replay

  When device_graph_capture is enabled, execute eager launch-trace verification before replay. Intended for debugging only.

• --decode-request-ttl-s <decode_request_ttl_s>

  Per-request TTL in seconds for the decode-side prefill_reqs and inflight_transfers dicts. Entries older than this are evicted individually (KV blocks released, failure surfaced to the client) before the stall watchdog fires. None (the default) disables eviction. Set with the MODULAR_DECODE_REQUEST_TTL_S environment variable.

• --decode-stall-timeout-s <decode_stall_timeout_s>

  Seconds of no-batch-activity after which the decode worker exits to trigger a pod restart. None (the default) disables the watchdog. Set with the MODULAR_DECODE_STALL_TIMEOUT_S environment variable.

• --device-graph-capture, --no-device-graph-capture

  Enable device graph capture and replay for graph execution. If unset, automatically enabled for some selected architectures. Use --no-device-graph-capture to explicitly disable.

• --device-memory-utilization <device_memory_utilization>

  The fraction of available device memory that the process should consume. The remaining headroom holds the KV cache: kv_cache_workspace = (total_free_memory * device_memory_utilization) - model_weights_size.

• --devices <devices>

  Whether to run the model on CPU (--devices=cpu), GPU (--devices=gpu), every visible GPU (--devices=gpu:all), or a list of GPUs (--devices=gpu:0,1). An ID value can be provided optionally to indicate the device ID to target. If not provided, the model or config default is used.

• --dp-ce-balance-enable-dynamic-chunk-size, --no-dp-ce-balance-enable-dynamic-chunk-size

  Whether a below-threshold CE step with work on 2+ replicas runs immediately with each replica's chunk size reduced to the balance level, deferring only the excess. When False, such steps are held whole until the threshold is met, a deadline expires, or there is nothing else to run. Only consulted when dp_ce_balance_timeout_ms > 0.

• --dp-ce-balance-threshold <dp_ce_balance_threshold>

  Per-step CE active-token occupancy across DP replicas (mean/max, 0-1) at or above which CE work is scheduled without further deferral. Only consulted when dp_ce_balance_timeout_ms > 0.

• --dp-ce-balance-timeout-ms <dp_ce_balance_timeout_ms>

  Max time in milliseconds a context-encoding request's work may be deferred, from arrival, while awaiting token-balanced scheduling across data-parallel replicas. -1 disables the balancer (requests bind to a replica on arrival; current default behavior); 0 enables post-cache-weighted placement with late binding but never defers; > 0 additionally defers unbalanced CE work until dp_ce_balance_threshold is met, the deadline expires, or there is nothing else to run.

• --draft-chat-template <draft_chat_template>

  Optional custom chat template to override the one shipped with the Hugging Face model config. If a path is provided, the file is read lazily by the registry when building the tokenizer. If None, the model's default chat template is used.

• --draft-config-file <draft_config_file>

• --draft-data-parallel-degree <draft_data_parallel_degree>

  Data-parallelism parameter. The degree to which the model is replicated is dependent on the model type.

• --draft-devices <draft_devices>

  Devices for the draft model in speculative decoding. If not provided, inherits from --devices. Accepts the same format as --devices.

• --draft-enable-echo, --no-draft-enable-echo

  Whether the model should be built with echo capabilities.

• --draft-force-download, --no-draft-force-download

  Whether to force download a given file if it's already present in the local cache.

• --draft-huggingface-model-revision <draft_huggingface_model_revision>

  Branch or Git revision of Hugging Face model repository to use.

• --draft-huggingface-weight-revision <draft_huggingface_weight_revision>

  Branch or Git revision of Hugging Face model repository to use.

• --draft-max-length <draft_max_length>

  Maximum sequence length the model can process. If not specified, defaults to the model's max_position_embeddings. May be clamped during resolution based on available memory.

• --draft-model-path <draft_model_path>

  Accepts either a Hugging Face repository ID or a local path to the model.

• --draft-pool-embeddings, --no-draft-pool-embeddings

  Whether to pool embedding outputs.

• --draft-proposal <draft_proposal>

  How the draft model proposes tokens. 'argmax' (default) proposes deterministically. 'sampled' makes the draft sample its own proposal and keep the distribution it drew from, so verification runs true speculative sampling instead of typical acceptance. Incompatible with use_relaxed_acceptance_for_thinking. Inert unless the serving architecture supports it.

  Options:

  argmax | sampled

• --draft-quantization-encoding <draft_quantization_encoding>

  Weight encoding type. For GGUF models, the encoding is auto-detected from the repository when unset; if set, it must match an available encoding. When the repository contains multiple quantization formats, set this to choose one.

  Options:

  float32 | float16 | bfloat16 | q4_k | q4_0 | q6_k | float8_e4m3fn | float4_e2m1fnx2 | float6_e2m3fn | gptq

• --draft-rope-type <draft_rope_type>

  Force using a specific rope type. Only matters for GGUF weights.

  Options:

  none | normal | neox | longrope | yarn

• --draft-section-name <draft_section_name>

• --draft-served-model-name <draft_served_model_name>

  Optional override for client-facing model name. Defaults to model_path.

• --draft-sliding-window <draft_sliding_window>

  If set, overrides the model's attention to use a sliding-window causal mask of this many tokens. None (the default) defers to the HuggingFace config's sliding_window field, or full causal attention if the model doesn't advertise one.

• --draft-subfolder <draft_subfolder>

  Subdirectory within the HuggingFace repo to load config and weights from (for example, vae or text_encoder). When set, config.json and weights are resolved from {model_path}/{subfolder}/.

• --draft-trust-remote-code, --no-draft-trust-remote-code

  Whether or not to allow for custom modeling files on Hugging Face.

• --draft-use-subgraphs, --no-draft-use-subgraphs

  Whether to use subgraphs for the model. This can significantly reduce compile time, especially for large models with identical blocks. Default is true.

• --draft-vision-config-overrides <draft_vision_config_overrides>

  Model-specific vision configuration overrides. For example, for InternVL: {"max_dynamic_patch": 24}.

• --draft-weight-path <draft_weight_path>

  Optional path or URL of the model weights to use. Overrides default weight discovery.

• --emit-reasoning-content, --no-emit-reasoning-content

  When True, chat completion responses emit a thinking model's chain-of-thought under reasoning_content only (reasoning is omitted). The reasoning_content alias is used by vLLM, SGLang, and the DeepSeek API; some clients require it. When False (default), responses emit reasoning under reasoning only.

• --enable-chunked-prefill, --no-enable-chunked-prefill

  Enable chunked prefill to split context encoding requests into multiple chunks based on max_batch_input_tokens.

• --enable-dp-cross-replica-prefix-copy, --no-enable-dp-cross-replica-prefix-copy

  Whether a prefix-cache block resident on another data-parallel (DP) replica's GPU may be copied device-to-device onto the request's replica to serve a cache hit. When disabled, cross-replica reuse is only served from the shared host/disk tier via the KV connector (or recomputed). Only relevant when data_parallel_degree > 1 and prefix caching is enabled.

• --enable-echo, --no-enable-echo

  Whether the model should be built with echo capabilities.

• --enable-in-flight-batching, --no-enable-in-flight-batching

  When enabled, prioritizes token generation by batching it with context encoding requests.

• --enable-lora, --no-enable-lora

  Enables LoRA on the server.

• --enable-min-tokens, --no-enable-min-tokens

  Whether to enable min_tokens, which blocks the model from generating stopping tokens before the min_tokens count is reached.

• --enable-overlap-scheduler, --no-enable-overlap-scheduler

  Whether to enable the overlap scheduler. This feature allows the scheduler to run alongside GPU execution. This helps improve GPU utilization. This is an experimental feature which may crash and burn. This feature will be enabled by default for some selected architectures. You can forcibly disable this by setting --no-enable-overlap-scheduler --force.

• --enable-penalties, --no-enable-penalties

  Whether to apply frequency and presence penalties to the model's output.

• --enable-prefix-caching, --no-enable-prefix-caching

  Whether to enable prefix caching for the paged KVCache.

• --enable-prioritize-first-decode, --no-enable-prioritize-first-decode

  When enabled, the scheduler always runs a TG batch immediately after a CE batch with the same requests. This may reduce time-to-first-chunk latency.

• --enable-structured-output, --no-enable-structured-output

  Enable structured generation/guided decoding for the server. This allows the user to pass a JSON schema in the response_format field, which the LLM will adhere to.

• --enable-tool-call-constrained-decode, --no-enable-tool-call-constrained-decode

  Whether tool-call requests are constrained to a server-generated grammar during decoding. When enabled (the default), a configured runtime.tool_parser both produces a decode-time grammar and parses the resulting output. Set to False to keep the parser (tool calls are still parsed out of generated text) while skipping the constrained-decode/bitmask path for tool calls -- useful when the grammar path is undesirable but tool-call parsing is still wanted. With this disabled, tool_choice=required or a named function can no longer force a tool call. Independent of enable_structured_output, which gates user-supplied response_format JSON schemas.

• --enable-variable-logits, --no-enable-variable-logits

  Enable the sampling graph to accept a ragged tensor of different sequences as inputs, along with their associated logit_offsets. This is needed to produce additional logits for echo and speculative decoding purposes.

• --ep-size <ep_size>

  The expert parallelism size. Needs to be 1 (no expert parallelism) or the total number of GPUs across nodes.

• --ep-use-allreduce, --no-ep-use-allreduce

  Whether to use allreduce for the cross-device communication in expert parallelism.

• --eplb-profile, --no-eplb-profile

  When True, enables expert-parallel load balancing (EPLB) MoE routing histogram profiling in the pipeline. Mirrors Settings.eplb_profile for pipeline code that doesn't have access to Settings.

• --eplb-replicas-per-gpu <eplb_replicas_per_gpu>

  Number of redundant expert replicas to add per GPU when EPLB is active. 0 (default) means no replication. k > 0 adds k extras per GPU; total redundant slots = k * ep_size (so num_redundant is always a multiple of the device count, which the rebalance algorithm requires).

• --execute-empty-batches, --no-execute-empty-batches

  When enabled, the scheduler runs the model's forward pass even for an empty batch, so expert-parallel and data-parallel replicas still reach their collective barrier points; output processing is skipped. The architecture must support empty batches.

• --experimental-vision-cache-utilization <experimental_vision_cache_utilization>

  Fraction of the KV cache pool budget (not total device memory) reserved for the experimental block-based vision encoder cache; the remainder stays with the KV cache. Greater than 0 activates block mode on architectures whose memory planner reports a vision row spec; 0 (the default) keeps the entry-count cache. Set via the MAX_EXPERIMENTAL_VISION_CACHE_UTILIZATION environment variable. Only used by VLMs.

• --export-mefs <export_mefs>

  Directory to write a compiled-graph artifact into for every graph this run compiles, for a later run to reuse via --precompiled-mefs. Compilation itself is unaffected.

• --first-block-caching, --no-first-block-caching

  Enable First-Block Cache (FBCache) for step-cache denoising. When enabled, the transformer skips remaining blocks if the first-block residual is similar to the previous step.

• --fold-sampler-into-graph, --no-fold-sampler-into-graph

  Fold greedy token selection (argmax) into the captured forward graph so a single device-graph replay materializes the sampled token, avoiding a separate sampler submission and its blocking readback. Only takes effect for all-greedy decode batches on architectures that emit the folded token output (Nemotron-H); any non-greedy request falls back to the separate sampler. Default on.

• --force, --no-force

  Skip validation of user provided flags against the architecture's required arguments.

• --force-download, --no-force-download

  Whether to force download a given file if it's already present in the local cache.

• --gpu-profiling <gpu_profiling>

  Whether to enable GPU profiling of the model.

  Options:

  off | on | detailed

• --huggingface-model-revision <huggingface_model_revision>

  Branch or Git revision of Hugging Face model repository to use.

• --huggingface-weight-revision <huggingface_weight_revision>

  Branch or Git revision of Hugging Face model repository to use.

• --kv-cache-format <kv_cache_format>

  Override the default data type for the KV cache. Supported values: float32, bfloat16, float8_e4m3fn.

• --kv-cache-hash-algo <kv_cache_hash_algo>

  Hash algorithm used for KV-cache block identity. ahash64 (default) is fast and non-cryptographic; sha256 is a cryptographic 256-bit hasher; both support an optional seed/salt for prefix-cache isolation. sha256_64 truncates the SHA-256 chain to 64 bits for protocol compatibility.

  Options:

  ahash64 | sha256 | sha256_64

• --kv-cache-hash-seed <kv_cache_hash_seed>

  Optional 64-character hex string (32 bytes), a cluster-wide seed for kv_cache_hash_algo. If omitted, sha256/sha256_64 generate a random seed at startup; ahash64 does not, so existing deployments are unaffected unless set explicitly.

• --kv-cache-page-size <kv_cache_page_size>

  The number of tokens in a single page in the paged KVCache.

• --kv-connector-config <kv_connector_config>

  KV cache connector configuration as inline JSON or a path to a YAML/JSON file. The connector type is the type field, e.g. '{"type": "rust_tiered"}'. Defaults to the null connector (no external caching); each type has sensible defaults for its remaining fields. Merges field-wise over a config file's value, so overriding one field on the command line preserves the rest.

• --lora-paths <lora_paths>

  List of statically defined LoRA paths.

• --max-batch-input-tokens <max_batch_input_tokens>

  The target number of un-encoded tokens to include in each batch. This value is used for chunked prefill and memory estimation.

• --max-batch-size <max_batch_size>

  Maximum batch size to execute with the model. When not specified (None), this value is determined dynamically. For server launches, set this higher based on server capacity.

• --max-batch-total-tokens <max_batch_total_tokens>

  Ensures the sum of page-aligned context lengths in a batch does not exceed max_batch_total_tokens. Alignment uses the KV cache page size. If None, the sum is not limited.

• --max-length <max_length>

  Maximum sequence length the model can process. If not specified, defaults to the model's max_position_embeddings. May be clamped during resolution based on available memory.

• --max-lora-rank <max_lora_rank>

  Maximum rank of all possible LoRAs.

• --max-media-preprocess-cache-idle-seconds <max_media_preprocess_cache_idle_seconds>

  How long a preprocessed image or video may go unused before it becomes eligible to be dropped from the tokenizer's cache. This is a reclaim policy rather than a lifetime: sweeps are periodic, so an entry can outlive its deadline, and a request that arrives meanwhile is served from it and resets the clock -- an entry is keyed on media content, so it never goes stale. Without this, the byte budget is the only bound, so a burst of distinct media holds its whole resident set for the rest of the process's life -- host memory the model worker's own allocations compete for. An entry is only worth keeping while the conversation that sent it might send the next turn, which is seconds to minutes, and re-preprocessing a wrongly dropped image costs a few milliseconds. Set to 0 to keep entries until the budget evicts them. Only used by VLMs.

• --max-num-loras <max_num_loras>

  The maximum number of active LoRAs in a batch. This controls how many LoRA adapters can be active simultaneously during inference. Lower values reduce memory usage but limit concurrent adapter usage.

• --max-pending-futures <max_pending_futures>

  Maximum number of unrealized future-token placeholders a request may hold at once. The default of 1 is the classic overlap-scheduler depth: one forward in flight per request. A value of 2 enables experimental schedule-ahead decoding in the overlap pipeline: two forwards stay in flight and each step's outputs are consumed one step late, for pure-greedy token-generation batches only (other batches drain to the classic depth). Not supported with speculative decoding; prefill-only workers pin to 1.

• --max-queue-size-tg <max_queue_size_tg>

  Maximum number of requests in decode queue. By default, this is max_batch_size.

• --max-video-preprocess-cache-bytes <max_video_preprocess_cache_bytes>

  Host-memory budget, in bytes, for caching preprocessed video tensors in the tokenizer. Unlike images, videos are not decoded at admission, so a hit skips the whole decode -- sampling, resize and patchify of every sampled frame. Budgeted separately from max_vision_preprocess_cache_bytes because a video entry is an order of magnitude larger than an image one, so a shared budget would let a single video evict many images. Set to 0 to disable. Only used by VLMs that accept video.

• --max-vision-cache-entries <max_vision_cache_entries>

  Maximum number of images cached in the vision encoder cache. Each entry stores the vision encoder output for one image, avoiding re-encoding across chunks and requests. Set to 0 to disable caching. Only used by VLMs.

• --max-vision-preprocess-cache-bytes <max_vision_preprocess_cache_bytes>

  Host-memory budget, in bytes, for caching preprocessed image tensors in the tokenizer. A hit skips the resize, rescale and patchify for a repeated image -- for example the same image resent on every turn of a conversation -- which the vision encoder cache cannot avoid, because it is consulted only after preprocessing has already run. This is a ceiling on resident host memory in the API server process, not a reservation: the cache grows to it under load and evicts least-recently-used entries to stay within it. Set to 0 to disable. Only used by VLMs.

• --min-batch-size-tg <min_batch_size_tg>

  Soft floor on the decode batch size. If the TG batch size is larger, the scheduler continues TG batches; if it falls below, the scheduler prioritizes CE. This is not a strict minimum. By default, this is max_queue_size_tg.

• --model, --model-path <model_path>

  Accepts either a Hugging Face repository ID or a local path to the model.

• --model-override <model_override>

  Per-component overrides for the ModelManifest, in the format component.field=value. Applied before resolution. Repeatable. Example: transformer.quantization_encoding=float4_e2m1fnx2.

• --models <models>

  The model manifest containing all model configs keyed by role.

• --num-speculative-tokens <num_speculative_tokens>

  The number of speculative tokens. Unset selects a per-method default: 2 for eagle/mtp, and the draft checkpoint's trained width for dflash.

• --pipeline-role <pipeline_role>

  Whether the pipeline should serve both a prefill or decode role or both.

  Options:

  prefill_and_decode | prefill_only | decode_only

• --pool-embeddings, --no-pool-embeddings

  Whether to pool embedding outputs.

• --precompiled-mefs <precompiled_mefs>

  Directory of compiled-graph artifacts written by an earlier run's --export-mefs. Every graph is initialized from its artifact instead of being compiled, so the compiling and the executing run can happen on different machines. The runs must build the same graphs; a mismatch is an error rather than a silent recompile.

• --prefer-module-v3, --no-prefer-module-v3

  Whether to prefer the eager API architecture over the graph API architecture. When False (default), the inference server uses the graph API architecture. When True, the server uses the eager API architecture when available and falls back to the graph API architecture.

• --quantization-encoding <quantization_encoding>

  Weight encoding type. For GGUF models, the encoding is auto-detected from the repository when unset; if set, it must match an available encoding. When the repository contains multiple quantization formats, set this to choose one.

  Options:

  float32 | float16 | bfloat16 | q4_k | q4_0 | q6_k | float8_e4m3fn | float4_e2m1fnx2 | float6_e2m3fn | gptq

• --reasoning-parser <reasoning_parser>

  Name of the reasoning output parser. The parser extracts thinking blocks to populate the reasoning field in chat completion responses. When unset, the server applies the architecture's default reasoning parser, if any. Pass "none" (case-insensitive) to explicitly disable reasoning parsing even when the architecture declares a default.

• --rejection-sampling-strategy <rejection_sampling_strategy>

  Rejection sampling strategy for verifying draft tokens. Defaults to typical-acceptance for eagle/mtp.

  Options:

  greedy | residual | typical-acceptance | logit-comparison

• --relaxed-delta <relaxed_delta>

  Probability gap below the top-1 candidate inside which candidates remain eligible for relaxed acceptance. A draft token is accepted if it matches any top-N candidate whose probability is at least top1_prob - relaxed_delta. Ignored when use_relaxed_acceptance_for_thinking is False.

• --relaxed-topk <relaxed_topk>

  Top-N candidates from the target distribution to consider when relaxed acceptance is active. Ignored when use_relaxed_acceptance_for_thinking is False.

• --rope-type <rope_type>

  Force using a specific rope type. Only matters for GGUF weights.

  Options:

  none | normal | neox | longrope | yarn

• --sample-on-host, --no-sample-on-host

  Run the token sampler on the host CPU instead of the model device. The last-token logits are copied device-to-host and sampling (top-k/argmax) runs on CPU. Default is to sample on the model device.

• --section-name <section_name>

• --served-model-name <served_model_name>

  Optional override for client-facing model name. Defaults to model_path.

• --sliding-window <sliding_window>

  If set, overrides the model's attention to use a sliding-window causal mask of this many tokens. None (the default) defers to the HuggingFace config's sliding_window field, or full causal attention if the model doesn't advertise one.

• --speculative-method <speculative_method>

  The speculative decoding method to use.

  Options:

  eagle | mtp | dflash

• --structured-output-any-whitespace, --no-structured-output-any-whitespace

  Whether structured-output (response_format) grammars accept whitespace between JSON tokens. False (the resolved default) constrains generation to compact JSON -- no whitespace, ','/':' separators -- which mitigates runaway generation on some models but also masks the newline/indentation tokens models prefer at structural boundaries. True uses the grammar engine's whitespace-tolerant JSON. When unset (None), resolved at config construction to the architecture's default if it declares one, else False. An explicit value always wins. Tool-call grammars are unaffected.

• --structured-output-backend <structured_output_backend>

  Grammar backend for constrained decoding. One of xgrammar or llguidance. When unset (None), resolved at config construction to the architecture's default if it declares one, else the global default xgrammar. An explicit value always wins.

• --subfolder <subfolder>

  Subdirectory within the HuggingFace repo to load config and weights from (for example, vae or text_encoder). When set, config.json and weights are resolved from {model_path}/{subfolder}/.

• --synthetic-acceptance-rate <synthetic_acceptance_rate>

  Synthetic acceptance rate for benchmarking (0.0 to 1.0). When set, the rejection sampler bypasses the real draft/target comparison and accepts each draft position with a calibrated probability so the mean joint acceptance across num_speculative_tokens positions matches this value.

• --target <target>

  Target API and architecture to compile for (e.g., cuda, cuda:sm_90, hip:gfx942, metal). When specified, uses virtual devices for compilation without requiring physical hardware.

• --task <task>

  The pipeline task to run (e.g. text_generation, embeddings_generation). Used to disambiguate architectures registered under the same name for multiple tasks.

  Options:

  PipelineTask.TEXT_GENERATION | PipelineTask.EMBEDDINGS_GENERATION | PipelineTask.PIXEL_GENERATION | PipelineTask.UNDEFINED

• --taylorseer, --no-taylorseer

  Enable TaylorSeer cache optimization. Uses Taylor series prediction to skip full transformer passes on certain denoising steps.

• --taylorseer-cache-interval <taylorseer_cache_interval>

  Steps between full TaylorSeer computations. None uses the model-specific default (typically 5).

• --taylorseer-max-order <taylorseer_max_order>

  Taylor expansion order (1 or 2). Higher order uses second derivatives for more accurate prediction. None uses the model-specific default (typically 1).

• --taylorseer-warmup-steps <taylorseer_warmup_steps>

  Number of warmup steps before TaylorSeer prediction begins. None uses the model-specific default (typically 4).

• --temperature <temperature>

  Default sampling temperature. Controls randomness of token selection—higher values (e.g. 1.0) produce more random outputs, lower values (e.g. 0.2) produce more deterministic outputs. When set, this server-level default applies to all requests that do not explicitly provide temperature.

• --thinking-temperature <thinking_temperature>

  Default temperature override for tokens inside <think>...</think> blocks. When set, this server-level default applies to all requests that do not explicitly provide thinking_temperature. Requires a reasoning parser to be configured; ignored otherwise.

• --tool-parser <tool_parser>

  Name of the tool call parser. The parser extracts tool calls from model output in chat completion responses. When unset, the server applies the architecture's default tool parser, if any. Pass "none" (case-insensitive) to explicitly disable tool parsing even when the architecture declares a default.

• --top-k <top_k>

  Default top-k sampling limit. When set, this server-level default applies to all requests that do not explicitly provide top_k.

• --trust-remote-code, --no-trust-remote-code

  Whether or not to allow for custom modeling files on Hugging Face.

• --use-experimental-kernels <use_experimental_kernels>

  Enables using experimental Mojo kernels with max serve. The kernels could be unstable or incorrect.

• --use-greedy-acceptance, --no-use-greedy-acceptance

  Use greedy (argmax) draft acceptance instead of the stochastic sampler. The greedy path has no mid-graph allocation, so the fused speculative graph can be CUDA-graph captured. Valid only for greedy serving (temperature 0, top_k 1); incompatible with relaxed and synthetic acceptance.

• --use-relaxed-acceptance-for-thinking, --no-use-relaxed-acceptance-for-thinking

  Enables relaxed acceptance for speculative decoding draft positions inside a <think>...</think> block. The target's top-N candidates (filtered by a probability threshold top1_prob - relaxed_delta) are compared against the draft token; matching any candidate accepts the draft. Outside the thinking span, the existing strict acceptance rule still applies. Requires draft_proposal='argmax'.

• --use-subgraphs, --no-use-subgraphs

  Whether to use subgraphs for the model. This can significantly reduce compile time, especially for large models with identical blocks. Default is true.

• --use-vendor-blas <use_vendor_blas>

  Enables using vendor BLAS libraries (cublas, hipblas, etc.) with max serve. Currently, this just replaces matmul calls.

• --use-vendor-ccl <use_vendor_ccl>

  Enables using vendor CCL libraries (NCCL/RCCL) for collective operations such as allreduce in multi-GPU inference.

• --vision-config-overrides <vision_config_overrides>

  Model-specific vision configuration overrides. For example, for InternVL: {"max_dynamic_patch": 24}.

• --weight-path <weight_path>

  Optional path or URL of the model weights to use. Overrides default weight discovery.