IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

EPBuffers

EPBuffers​

class max.nn.ep.EPBuffers(counters, dispatch, combine, eplb=None, expert_load=None)

source

Bases: object

Every buffer of the expert-parallel exchange.

A decoder receives it ready as DecoderInputs.ep and passes it to its MoE layers. Dispatch and combine each send through their own pointer tables, so that combine never writes to buffers that a dispatch still reads. Both directions synchronize on the same atomic counters.

Parameters:

  • counters (Tensor)
  • dispatch (EPTables)
  • combine (EPTables)
  • eplb (EPLBBuffers | None)
  • expert_load (Tensor | None)

combine​

combine: EPTables

source

The pointer tables combine uses.

counters​

counters: Tensor

source

Each device’s atomic counters, sharded over the mesh.

dispatch​

dispatch: EPTables

source

The pointer tables dispatch uses.

eplb​

eplb: EPLBBuffers | None = None

source

The EPLB buffers, or None when EPLB is off.

expert_load​

expert_load: Tensor | None = None

source

Each device’s [num_moe_layers, n_local_experts] uint64 tally of the tokens routed to its experts, sharded over the mesh by expert, or None when expert-load counting is off.

Every dispatch adds to the row of its MoE layer. Counting is independent of EPLB, so that a static placement can be profiled too.

on()​

on(mesh)

source

Returns these buffers on mesh, moving no data.

The framework places the buffers on a mesh of one axis. A model with its own mesh of the same devices re-places them on it, so that dispatch() takes them with that mesh’s tokens. The host pointer tables stay as they are.

Parameters:

mesh (DeviceMesh) – A mesh of the same devices, in the same order, of any shape.

Returns:

The same buffers, each placed on mesh as it is placed now.

Raises:

ValueError – If mesh holds other devices.

Return type:

EPBuffers