IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

EPConfig

EPConfig​

class max.nn.ep.EPConfig(dispatch_dtype, combine_dtype, hidden_size, top_k, n_experts, max_tokens_per_rank, n_gpus_per_node, n_nodes, node_id=-1, dispatch_quant_config=None, nvfp4_dyn_global_scales=False, fused_shared_expert=False, shared_expert_side_stream=None, fuse_ffn_combine_send=False, moe_dim=0, fused_moe=False, fused_moe_ready=False, use_allreduce=False, tokens_split_evenly=False, max_batch_size=0, combine_mx_min_tokens=0, eplb_enabled=False, num_moe_layers=0, max_replicas=1, eplb_router_folded=False, eplb_count_expert_load=False, eplb_phy2log_plan=None, num_logical_experts=0)

source

Bases: object

Configuration for Expert Parallelism (EP) communication.

Parameters:

  • dispatch_dtype (DType)
  • combine_dtype (DType)
  • hidden_size (int)
  • top_k (int)
  • n_experts (int)
  • max_tokens_per_rank (int)
  • n_gpus_per_node (int)
  • n_nodes (int)
  • node_id (int)
  • dispatch_quant_config (QuantConfig | None)
  • nvfp4_dyn_global_scales (bool)
  • fused_shared_expert (bool)
  • shared_expert_side_stream (Literal['pre', 'post'] | None)
  • fuse_ffn_combine_send (bool)
  • moe_dim (int)
  • fused_moe (bool)
  • fused_moe_ready (bool)
  • use_allreduce (bool)
  • tokens_split_evenly (bool)
  • max_batch_size (int)
  • combine_mx_min_tokens (int)
  • eplb_enabled (bool)
  • num_moe_layers (int)
  • max_replicas (int)
  • eplb_router_folded (bool)
  • eplb_count_expert_load (bool)
  • eplb_phy2log_plan (Any)
  • num_logical_experts (int)

combine_dtype​

combine_dtype: DType

source

Data type used for combining expert outputs.

combine_mx_min_tokens​

combine_mx_min_tokens: int = 0

source

Steps of at least this many tokens across all ranks send the combine back as MXFP8 (E4M3 values with an E8M0 scale per 32 elements), about half the bytes of bfloat16, at some accuracy cost. 0 keeps every combine exact. Only the split ep_combine_async / ep_combine_wait path on a single node takes it.

dispatch_dtype​

dispatch_dtype: DType

source

Data type used for dispatching tokens to experts.

dispatch_quant_config​

dispatch_quant_config: QuantConfig | None = None

source

Quantization configuration used for dispatch token quantization.

eplb_count_expert_load​

eplb_count_expert_load: bool = False

source

When true, the exchange’s graph inputs include one [num_moe_layers, n_local_experts] uint64 tally per GPU (max.nn.ep.EPBuffers.expert_load), and every dispatch adds each local expert slot’s routed-token count to the row of its MoE layer. The tally is a running total the engine reads and differences on the host. Independent of eplb_enabled: a static placement can be profiled too.

eplb_enabled​

eplb_enabled: bool = False

source

When true, the exchange’s graph inputs include each GPU’s log2phy and logcnt maps (max.nn.ep.EPBuffers.eplb).

eplb_phy2log_plan​

eplb_phy2log_plan: Any = None

source

Optional pre-computed EPLB plan (phy2log numpy array, shape [num_moe_layers, num_phy]). Only ModuleV2 models read it: when non-None, the legacy EPBatchManager picks it up at construction time so that MoE.shard sees the plan during the model’s first __init__ pass (no re-shard needed). max.nn.ep does not read it.

eplb_router_folded​

eplb_router_folded: bool = False

source

When true, the router’s expert-indexed weights are stored in physical slot order, so the router emits physical expert ids and dispatch skips the EPLB remap. Only valid for a placement that is a pure permutation (max_replicas == 1).

estimate_memory_usage()​

estimate_memory_usage()

source

Estimate the EP communication memory usage per device per buffer group.

Returns:

Estimated memory usage in bytes per device per buffer group.

Return type:

int

fuse_ffn_combine_send​

fuse_ffn_combine_send: bool = False

source

Fuse the combine send into the MoE FFN’s own epilogue.

The FFN’s down-projection epilogue scatters each finished row straight into the owning rank’s receive buffer and releases the arrival signal itself, so the FFN’s output is never staged in a local tensor: the (max_recv_tokens, hidden_size) buffer the unfused chain has to materialize – the largest single activation in the MoE region – is not allocated. The caller drops its combine_async and goes straight to the wait, which reads only the symmetric receive buffers.

Opt-in per model rather than always-on: the send pays for itself at prefill-scale tokens per expert and is neutral below, so it is a recipe choice rather than a universal default. Incompatible with use_allreduce (no peer buffers to write) and with fused_shared_expert (which keeps the shared expert’s rows in the very tensor being elided); both raise rather than degrade silently.

fused_moe​

fused_moe: bool = False

source

Whether EP init sets up the backend’s one-launch fused MoE. A ModuleV3 pipeline sets it from MODULAR_EP_FUSED_MOE=1, which the legacy EPCommInitializer reads itself.

fused_moe_ready​

fused_moe_ready: bool = False

source

Whether EP init set up the backend’s one-launch fused MoE for this configuration: it was requested and the backend serves the configuration. Layers whose experts the fused op serves then run dispatch, FFN and combine as one launch.

fused_shared_expert​

fused_shared_expert: bool = False

source

Whether to fuse the shared expert computation with the routed experts.

get_max_recv_tokens()​

get_max_recv_tokens()

source

Get the maximum number of tokens that can be received by a single device.

Return type:

int

hidden_size​

hidden_size: int

source

Size of the hidden dimension in the model.

max_batch_size​

max_batch_size: int = 0

source

Decode concurrency cap sizing the AMD MXFP4 preb direct grid.y on the decode bands. 0 disables (full-stride fallback).

max_replicas​

max_replicas: int = 1

source

Largest replica count across all (layer, logical) cells. Also the trailing dim of log2phy.

max_tokens_per_rank​

max_tokens_per_rank: int

source

Maximum number of tokens processed per GPU rank.

moe_dim​

moe_dim: int = 0

source

Intermediate width of the routed experts’ FFN, when the model provides it. A config that gives it gets the fused MoE counters’ reserve in its EP sync-counter buffers. It does not request the fused MoE: see fused_moe.

n_experts​

n_experts: int

source

Total number of experts in the model.

n_gpus_per_node​

n_gpus_per_node: int

source

Number of GPUs available per node.

n_nodes​

n_nodes: int

source

Total number of nodes in the distributed setup.

node_id​

node_id: int = -1

source

ID of the current node, which EP init finds.

num_logical_experts​

num_logical_experts: int = 0

source

Number of logical experts per layer. When EPLB is disabled this equals n_experts. When EPLB is enabled, n_experts is inflated to the physical slot count and this preserves the original logical count, which is the leading dim of log2phy / logcnt.

num_moe_layers​

num_moe_layers: int = 0

source

Number of MoE layers; sizes the per layer EPLB buffer.

nvfp4_dyn_global_scales​

nvfp4_dyn_global_scales: bool = False

source

Quantize each NVFP4 dispatch token against its own global scale.

Instead of the checkpoint’s static input scale, the dispatch kernel first reduces each token’s absolute max, then quantizes the token against 2688 / rowmax. The scale’s BF16 inverse travels with the token, and the dispatch outputs gain output_rowwise_scales – one per received row, directly after the block scales – so a received row dequantizes as fp4 * block_scale * rowwise_scale. Requires NVFP4 dispatch.

shared_expert_side_stream​

shared_expert_side_stream: Literal['pre', 'post'] | None = None

source

Runs the unfused shared expert of an nvfp4_dyn_global_scales model on a side stream so it overlaps the EP collectives.

The per-token NVFP4 dispatch has no async split, so without this its shared expert runs inline and serializes with the dispatch. "pre" issues it before the dispatch, "post" before the combine. Requires nvfp4_dyn_global_scales; the other dispatch paths already overlap it.

tokens_split_evenly​

tokens_split_evenly: bool = False

source

Whether every device holds within one token of every other at each EP step, as tensor-parallel attention’s even reduce-scatter leaves them. The P2P small path needs it: each device decides from its own count whether the protocol kernels run, so devices with unrelated counts (data-parallel attention) would disagree and hang.

top_k​

top_k: int

source

Number of top experts to route each token to.

use_allreduce​

use_allreduce: bool = False

source

Whether to use allreduce for the cross-device communication.

uses_p2p_small_path​

property uses_p2p_small_path: bool

source

Whether the EP buffers carry the P2P small path’s flags and words.

The EP kernels take the small path for decode-sized steps only on AMD GPUs, on a single node, without allreduce and with a fused shared expert, and only when the sync-counter buffers end in its words. Every other configuration keeps the buffers it had without the small path. The GPU vendor is the graph’s target (see _ep_targets_amd()), not the host’s. It also needs tokens_split_evenly, and a dispatch format other than NVFP4, whose dispatch the small kernel cannot serve. The remaining terms mirror the bounds of ep_p2p_small_on in ep.mojo, so the buffers are sized for the small path only when the kernels will take it.