For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Python function
make_stacked_gated_activation_fn
make_stacked_gated_activation_fn()
max.nn.moe.make_stacked_gated_activation_fn(activation_fn)
Builds a gated activation for split (gate, up) projections.
The returned callable applies activation_fn to the gate tensor
and multiplies with the up tensor:
activation_fn(gate) * up.
-
Parameters:
-
activation_fn (Callable[[TensorValue], TensorValue]) – Pointwise activation to apply to the gate tensor.
-
Returns:
-
A callable
(gate, up) -> activated. -
Return type: