IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python class

Grouped

Grouped​

class max.nn.ep.Grouped(tokens, expert_start, expert_ids, bounds, scales=None, scales_offset=None, row_scales=None)

source

Bases: object

The tokens a device received, grouped by local expert.

This is the layout that the grouped matmuls read. The optional fields that a dispatch fills depend on its format:

Parameters:

bounds​

bounds: Tensor

source

A (2,) uint32 host tensor per device, read inside call_on_mesh(): see grouped_matmul_bounds(). The NVFP4 grouped matmul builds its own bounds, but the MegaFFN combine send reads the expert count from these.

dispatch() builds one host tensor that every device shares, and dispatch_finish() builds one per device, as the legacy exchange does. dispatch_local(), and a dispatch() that counts expert load, also build one per device. Outside call_on_mesh the two differ in placement.

expert_ids​

expert_ids: Tensor

source

The local expert of each group of rows.

expert_start​

expert_start: Tensor

source

The first row of each expert’s tokens, then the end of the last.

row_scales​

row_scales: Tensor | None = None

source

Each received row’s inverse global scale, for NVFP4 tokens quantized with per-token global scales, else None.

scales​

scales: Tensor | None = None

source

The scales of quantized tokens, or None for BF16 tokens.

scales_offset​

scales_offset: Tensor | None = None

source

Each expert’s first row in the padded scales of the NVIDIA block-scaled layout, or None when the scales are not padded.

tokens​

tokens: Tensor

source

The received tokens, the rows of each expert together.