For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python function
mega_ffn_fused
mega_ffn_fused()
max.nn.ep.mega_ffn_fused(config, buffers, *, tokens, ids, weights, input_scales, gate_up_weight, gate_up_b_scales, down_weight, down_b_scales, gate_up_expert_scales, down_expert_scales, c_input_scales, activation=None, layer_idx=None)
Runs dispatch, the routed experts’ FFN and the combine in one launch.
Each device runs one kernel, which replaces dispatch(),
both grouped matmuls and combine(). Every device must
run it for every MoE layer, also with no tokens, because its peers wait
for its counts. The fused workspace that EP init sets up is shared by
every MoE layer on a device.
-
Parameters:
-
- config (EPConfig) – The expert-parallel configuration. EP init must have set
config.fused_moe_ready. - buffers (EPBuffers) – The exchange’s buffers.
- tokens (Tensor) – Each device’s BF16 tokens, of shape
(tokens, hidden)per device. - ids (Tensor) – Each device’s
int32top-k expert ids, of shape(tokens, top_k)per device. - weights (Tensor) – Each device’s
float32router weights, of shape(tokens, top_k)per device. - input_scales (Tensor) – Each device’s inverted NVFP4 dispatch input scale. The kernel reads element 0. It is 1 for MXFP8.
- gate_up_weight (Tensor) – The local experts’ gate/up weights, in the sigma-permuted NVFP4 or MXFP8 layout.
- gate_up_b_scales (Tensor) – The gate/up weight scales.
- down_weight (Tensor) – The local experts’ NVFP4 or MXFP8 down weights.
- down_b_scales (Tensor) – The down weight scales.
- gate_up_expert_scales (Tensor) – Each local expert’s gate/up output scale.
- down_expert_scales (Tensor) – Each local expert’s down output scale.
- c_input_scales (Tensor) – Each local expert’s inverted scale for the requantization between the two matmuls.
- activation (GatedActivation | None) – The gated activation that the kernel fuses. Defaults to SiLU.
- layer_idx (Tensor | None) – Each device’s
int32index of this MoE layer, as fordispatch(). Needed withconfig.eplb_count_expert_load, and ignored otherwise.
- config (EPConfig) – The expert-parallel configuration. EP init must have set
-
Returns:
-
Each token’s weighted sum of its routed experts’ BF16 outputs, mapped as
tokens. The caller adds the shared expert. -
Raises:
-
ValueError – If
check_supported()rejectsconfig, if EP init did not set up the fused MoE, or ifconfigcounts expert load withoutlayer_idxor withoutexpert_load. -
Return type: