IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

ExpertLoadTally

struct ExpertLoadTally

Running per-layer, per-local-expert token totals in device memory.

The tally is a [layers, n_local_experts] uint64 buffer that is never reset on device; the host reads it periodically and differences consecutive readings. The add is atomic so two overlapped steps or two data-parallel ranks sharing a GPU cannot lose an update.

Fields​

  • ​counts (MutPointer[UInt64, MutUntrackedOrigin]): Base of the [layers, n_local_experts] running totals.
  • ​layer_idx (ImmPointer[Int32, ImmUntrackedOrigin]): Device scalar holding the MoE layer index of this dispatch, the same tensor the EPLB router reads.

Implemented traits​

AnyType, Copyable, Deinitable, DevicePassable, ExpertLoadSink, ImplicitlyCopyable, Movable, RegisterPassable, TrivialRegisterPassable

comptime members​

device_type​

comptime device_type = ExpertLoadTally

Device-side type alias. ExpertLoadTally is trivially device-passable.

Methods​

__init__​

def __init__(counts: MutPointer[UInt64, MutUntrackedOrigin], layer_idx: ImmPointer[Int32, ImmUntrackedOrigin]) -> Self

Wraps the tally buffer and the layer-index scalar.

Args:

add​

def add(self, local_expert: Int, n_local_experts: Int, count: UInt32)

Atomically adds count to the expert's total for this layer.

Args:

  • ​local_expert (Int): Index of the expert among this rank's experts.
  • ​n_local_experts (Int): Number of routed experts on this rank.
  • ​count (UInt32): Tokens routed to the expert in this step.

get_type_name​

static def get_type_name() -> String

Returns the type name for runtime diagnostics.

Returns:

String: Always "ExpertLoadTally".

Was this page helpful?