For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python function
dispatch
dispatch()
max.nn.ep.dispatch(config, buffers, tokens, ids, *, input_scales=None, layer_idx=None)
Sends each token to the devices of its experts, in one launch.
A quantized dispatch quantizes the tokens in transit to the format of
config.dispatch_quant_config. The multi-device kernel cannot count
expert load, so with config.eplb_count_expert_load each device runs
its own fused dispatch kernel instead, as the legacy exchange does.
-
Parameters:
-
- config (EPConfig) – The expert-parallel configuration.
- buffers (EPBuffers) – The exchange’s buffers.
- tokens (Tensor) – Each device’s BF16 tokens, of shape
(tokens, hidden)per device. - ids (Tensor) – Each device’s top-k expert ids, of shape
(tokens, top_k)per device, in any integer dtype. - input_scales (Tensor | None) – Each device’s global input scale, which the dispatch inverts. NVFP4 needs it, unless it scales each token on its own. MXFP8 in the NVIDIA block-scaled layout uses it when given, and 1 otherwise. The other formats ignore it.
- layer_idx (Tensor | None) – Each device’s
int32index of this MoE layer, a scalar or of shape(1,): the row ofEPBuffers.expert_loadthat the dispatch adds to. Needed withconfig.eplb_count_expert_load, and ignored otherwise.
-
Returns:
-
The tokens each device received, and the route that
combine()sends them back by. -
Raises:
-
ValueError – If
check_supported()rejectsconfig, or ifconfigcounts expert load withoutlayer_idxor withoutEPBuffers.expert_load. -
Return type: