For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python class
Grouped
Grouped
class max.nn.ep.Grouped(tokens, expert_start, expert_ids, bounds, scales=None, scales_offset=None, row_scales=None)
Bases: object
The tokens a device received, grouped by local expert.
This is the layout that the grouped matmuls read. The optional fields that a dispatch fills depend on its format:
- BF16: no scales.
- FP8, MXFP4 and MXFP6:
scales. - MXFP8:
scales, and alsoscales_offsetin the NVIDIA block-scaled layout. - NVFP4:
scalesandscales_offset. With per-token global scales, alsorow_scales.
-
Parameters:
bounds
bounds: Tensor
A (2,) uint32 host tensor per device, read inside
call_on_mesh(): see grouped_matmul_bounds().
The NVFP4 grouped matmul builds its own bounds, but the MegaFFN combine
send reads the expert count from these.
dispatch() builds one host tensor that every device shares, and
dispatch_finish() builds one per device, as the legacy exchange
does. dispatch_local(), and a dispatch() that counts expert
load, also build one per device. Outside call_on_mesh the two differ
in placement.
expert_ids
expert_ids: Tensor
The local expert of each group of rows.
expert_start
expert_start: Tensor
The first row of each expert’s tokens, then the end of the last.
row_scales
Each received row’s inverse global scale, for NVFP4 tokens quantized
with per-token global scales, else None.
scales
The scales of quantized tokens, or None for BF16 tokens.
scales_offset
Each expert’s first row in the padded scales of the NVIDIA
block-scaled layout, or None when the scales are not padded.
tokens
tokens: Tensor
The received tokens, the rows of each expert together.