For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Python function

make_concatenated_gated_activation_fn

make_concatenated_gated_activation_fn()

max.nn.moe.make_concatenated_gated_activation_fn(activation_fn, limit=None)

source

Builds a gated activation for concatenated [gate | up] projections.

The returned callable splits gate_up at moe_dim, applies activation_fn to the gate half, and multiplies with the up half: activation_fn(gate_up[:, :moe_dim]) * gate_up[:, moe_dim:].

When limit is provided, both halves are clamped to [-limit, limit] before multiplying.

Parameters:

• activation_fn (Callable[[TensorValue], TensorValue])
• limit (float | None)

Return type:

Callable[[TensorValue, int], TensorValue]