IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python module

max.nn.ep

Expert parallelism over a device mesh.

Plain functions over mesh-sharded tensors. The exchange’s buffers are passed explicitly as one EPBuffers argument, which a decoder receives ready from the framework, and no call keeps state for a later one: dispatch() returns the Route that combine() takes. mega_ffn_fused() and mega_ffn_combine_send() fuse the routed experts’ FFN into the exchange. max.nn.ep.init is the host side the framework runs.

Configuration and buffers​

EPBuffersEvery buffer of the expert-parallel exchange.
EPConfigConfiguration for Expert Parallelism (EP) communication.

Exchange​

GroupedThe tokens a device received, grouped by local expert.
RouteWhere each received token came from, which combine sends it back to.
combineReturns the experts' outputs to their tokens, weighted, in one launch.
combine_finishWaits for each device's combine that combine_start() launched.
combine_localReturns the experts' outputs to their tokens, one kernel per device.
combine_startLaunches each device's combine; combine_finish() waits for it.
dispatchSends each token to the devices of its experts, in one launch.
dispatch_finishWaits for each device's dispatch that dispatch_start() launched.
dispatch_localSends each token to its experts, with one fused kernel per device.
dispatch_startLaunches each device's dispatch; dispatch_finish() waits for it.
grouped_matmul_boundsReturns the grouped matmul's host bounds for the received tokens.

Fused expert FFN​

mega_ffn_combine_sendRuns the routed experts' FFN and sends its rows back, one launch each.
mega_ffn_fusedRuns dispatch, the routed experts' FFN and the combine in one launch.

Functions​

check_supportedRaises if this exchange cannot run config on mesh.