For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python class
EPBuffers
EPBuffers
class max.nn.ep.EPBuffers(counters, dispatch, combine, eplb=None, expert_load=None)
Bases: object
Every buffer of the expert-parallel exchange.
A decoder receives it ready as DecoderInputs.ep and passes it to its
MoE layers. Dispatch and combine each send through their own pointer
tables, so that combine never writes to buffers that a dispatch still
reads. Both directions synchronize on the same atomic counters.
-
Parameters:
combine
combine: EPTables
The pointer tables combine uses.
counters
counters: Tensor
Each device’s atomic counters, sharded over the mesh.
dispatch
dispatch: EPTables
The pointer tables dispatch uses.
eplb
eplb: EPLBBuffers | None = None
The EPLB buffers, or None when EPLB is off.
expert_load
Each device’s [num_moe_layers, n_local_experts] uint64 tally
of the tokens routed to its experts, sharded over the mesh by expert, or
None when expert-load counting is off.
Every dispatch adds to the row of its MoE layer. Counting is independent of EPLB, so that a static placement can be profiled too.
on()
on(mesh)
Returns these buffers on mesh, moving no data.
The framework places the buffers on a mesh of one axis. A model with
its own mesh of the same devices re-places them on it, so that
dispatch() takes them with that mesh’s tokens. The
host pointer tables stay as they are.
-
Parameters:
-
mesh (DeviceMesh) – A mesh of the same devices, in the same order, of any shape.
-
Returns:
-
The same buffers, each placed on
meshas it is placed now. -
Raises:
-
ValueError – If
meshholds other devices. -
Return type: