For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

EplbStatsSnapshot

EplbStatsSnapshot

class max.pipelines.lib.eplb_stats.EplbStatsSnapshot(metadata, histogram, total_tokens=0, total_routed_tokens=0, hostname=None)

source

Bases: object

An immutable snapshot of per-layer logical-expert routing counts.

The histogram has shape (num_moe_layers, num_logical_experts) and dtype int64. Values are cumulative token-routings to each logical expert since the worker started accumulating.

Snapshots are produced by the model worker and consumed by the API server. They are intentionally pure-data: no locks, no references back into the accumulator.

The histogram is marked read-only after construction so a consumer holding a snapshot can never mutate it.

Parameters:

• metadata (EplbStatsMetadata)
• histogram (ndarray[tuple[Any, ...], dtype[int64]])
• total_tokens (int)
• total_routed_tokens (int)
• hostname (str | None)

draft_candidate_tokens

property draft_candidate_tokens: int

source

Extra tokens routed by verified speculative draft candidates.

0 on non-spec paths.

from_array()

classmethod from_array(metadata, histogram, *, total_tokens=0, total_routed_tokens=0, hostname=None)

source

Builds a snapshot, defensively copying the histogram.

Use this when the source array is owned by a live accumulator that may keep mutating after the snapshot is taken.

Parameters:

• metadata (EplbStatsMetadata) – Static shape descriptor.
• histogram (ndarray[tuple[Any, ...], dtype[int64]]) – 2D int64 array of cumulative counts. Will be copied.
• total_tokens (int) – Total number of tokens contributing to the histogram so far.
• total_routed_tokens (int) – Total number of tokens routed to experts, including speculative draft candidates.
• hostname (str | None) – Optional worker hostname to tag the snapshot with.

Returns:

A new class instance of EplbStatsSnapshot.

Raises:

AssertionError – If shape/dtype/values are invalid.

Return type:

EplbStatsSnapshot

from_dict()

classmethod from_dict(data)

source

Constructs a snapshot from a dict produced by method to_dict.

Parameters:

data (dict[str, Any]) – A dict produced by method to_dict.

Returns:

A new class instance of EplbStatsSnapshot.

Raises:

• ValueError – If a required field is missing, has the wrong type,
• AssertionError – If the histogram fails shape/dtype validation against the metadata.

Return type:

EplbStatsSnapshot

histogram

histogram: ndarray[tuple[Any, ...], dtype[int64]]

source

Int64 array of shape (num_layers, num_logical_experts). Read-only.

hostname

hostname: str | None = None

source

metadata

metadata: EplbStatsMetadata

source

Static shape/semantics descriptor.

to_dict()

to_dict()

source

Returns a JSON-safe dict representation.

The histogram is serialized as a nested list[list[int]] and int64 values are converted to Python int via ndarray.tolist().

Returns:

A dict that is safe to pass directly to json.dumps.

Return type:

dict[str, Any]

total_routed_tokens

total_routed_tokens: int = 0

source

Tokens actually routed through the target’s experts (confirmed + verified drafts). Any MoE row sums to total_routed_tokens * num_experts_per_token; equivalently total_routed_tokens == total_tokens + draft_candidate_tokens.

total_tokens

total_tokens: int = 0

source

Non-draft (confirmed) tokens fed to the target. In spec decode this EXCLUDES draft candidates, so it is < total_routed_tokens.