For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python module
max.nn.ep
Expert parallelism over a device mesh.
Plain functions over mesh-sharded tensors. The exchange’s buffers are
passed explicitly as one EPBuffers argument, which a decoder
receives ready from the framework, and no call keeps state for a later one:
dispatch() returns the Route that combine() takes.
mega_ffn_fused() and mega_ffn_combine_send() fuse the routed
experts’ FFN into the exchange. max.nn.ep.init is the host side the
framework runs.
Configuration and buffers
EPBuffers | Every buffer of the expert-parallel exchange. |
|---|---|
EPConfig | Configuration for Expert Parallelism (EP) communication. |
Exchange
Grouped | The tokens a device received, grouped by local expert. |
|---|---|
Route | Where each received token came from, which combine sends it back to. |
combine | Returns the experts' outputs to their tokens, weighted, in one launch. |
|---|---|
combine_finish | Waits for each device's combine that combine_start() launched. |
combine_local | Returns the experts' outputs to their tokens, one kernel per device. |
combine_start | Launches each device's combine; combine_finish() waits for it. |
dispatch | Sends each token to the devices of its experts, in one launch. |
dispatch_finish | Waits for each device's dispatch that dispatch_start() launched. |
dispatch_local | Sends each token to its experts, with one fused kernel per device. |
dispatch_start | Launches each device's dispatch; dispatch_finish() waits for it. |
grouped_matmul_bounds | Returns the grouped matmul's host bounds for the received tokens. |
Fused expert FFN
mega_ffn_combine_send | Runs the routed experts' FFN and sends its rows back, one launch each. |
|---|---|
mega_ffn_fused | Runs dispatch, the routed experts' FFN and the combine in one launch. |
Functions
check_supported | Raises if this exchange cannot run config on mesh. |
|---|