For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python class
GateUpFormat
GateUpFormat
class max.nn.moe.GateUpFormat(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)
Bases: Enum
Specifies the format of the combined gate/up projection weights.
CONCATENATED
CONCATENATED = 'concatenated'
Gate and up projections concatenated as [gate | up].
Stored as [num_experts, hidden_dim, 2 * moe_dim].
Split at moe_dim: gate = output[:, :moe_dim],
up = output[:, moe_dim:]. Used by Llama4 and Qwen3VL.
INTERLEAVED
INTERLEAVED = 'interleaved'
Gate and up projections interleaved as [g0, u0, g1, u1, ...].
Stored as [num_experts, hidden_dim, 2 * moe_dim].
Split with stride: gate = output[:, 0::2],
up = output[:, 1::2]. Used by GptOss.