IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python function

dispatch

dispatch()​

max.nn.ep.dispatch(config, buffers, tokens, ids, *, input_scales=None, layer_idx=None)

source

Sends each token to the devices of its experts, in one launch.

A quantized dispatch quantizes the tokens in transit to the format of config.dispatch_quant_config. The multi-device kernel cannot count expert load, so with config.eplb_count_expert_load each device runs its own fused dispatch kernel instead, as the legacy exchange does.

Parameters:

  • config (EPConfig) – The expert-parallel configuration.
  • buffers (EPBuffers) – The exchange’s buffers.
  • tokens (Tensor) – Each device’s BF16 tokens, of shape (tokens, hidden) per device.
  • ids (Tensor) – Each device’s top-k expert ids, of shape (tokens, top_k) per device, in any integer dtype.
  • input_scales (Tensor | None) – Each device’s global input scale, which the dispatch inverts. NVFP4 needs it, unless it scales each token on its own. MXFP8 in the NVIDIA block-scaled layout uses it when given, and 1 otherwise. The other formats ignore it.
  • layer_idx (Tensor | None) – Each device’s int32 index of this MoE layer, a scalar or of shape (1,): the row of EPBuffers.expert_load that the dispatch adds to. Needed with config.eplb_count_expert_load, and ignored otherwise.

Returns:

The tokens each device received, and the route that combine() sends them back by.

Raises:

ValueError – If check_supported() rejects config, or if config counts expert load without layer_idx or without EPBuffers.expert_load.

Return type:

tuple[Grouped, Route]