For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
ExpertLoadTally
struct ExpertLoadTally
Running per-layer, per-local-expert token totals in device memory.
The tally is a [layers, n_local_experts] uint64 buffer that is never
reset on device; the host reads it periodically and differences
consecutive readings. The add is atomic so two overlapped steps or two
data-parallel ranks sharing a GPU cannot lose an update.
Fields
- counts (
MutPointer[UInt64, MutUntrackedOrigin]): Base of the[layers, n_local_experts]running totals. - layer_idx (
ImmPointer[Int32, ImmUntrackedOrigin]): Device scalar holding the MoE layer index of this dispatch, the same tensor the EPLB router reads.
Implemented traits
AnyType,
Copyable,
Deinitable,
DevicePassable,
ExpertLoadSink,
ImplicitlyCopyable,
Movable,
RegisterPassable,
TrivialRegisterPassable
comptime members
device_type
comptime device_type = ExpertLoadTally
Device-side type alias. ExpertLoadTally is trivially device-passable.
Methods
__init__
def __init__(counts: MutPointer[UInt64, MutUntrackedOrigin], layer_idx: ImmPointer[Int32, ImmUntrackedOrigin]) -> Self
Wraps the tally buffer and the layer-index scalar.
Args:
- counts (
MutPointer[UInt64, MutUntrackedOrigin]): Base of the[layers, n_local_experts]running totals. - layer_idx (
ImmPointer[Int32, ImmUntrackedOrigin]): Device scalar holding the MoE layer index.
add
def add(self, local_expert: Int, n_local_experts: Int, count: UInt32)
Atomically adds count to the expert's total for this layer.
Args:
get_type_name
static def get_type_name() -> String
Returns the type name for runtime diagnostics.
Returns:
String: Always "ExpertLoadTally".