IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

situ

def situ[alpha: Float32, limit: Float32](gate: Float32, up: Float32) -> Float32

Computes the SiTU gated activation.

With beta = alpha and linear_beta = limit, returns beta * tanh(gate / beta) * sigmoid(gate) * u, where u is linear_beta * tanh(up / linear_beta) when limit != 0 and up otherwise. Unlike the stock activations this uses IEEE fp32 division rather than recip, like the graph's elementwise SiTU chain: the quantize consumes the value's bfloat16 rounding, so it must track the chain within its transcendental ulps, which recip would exceed.

Parameters:

  • ​alpha (Float32): The gate's tanh scale (beta); non-zero, since the gate is divided by it.
  • ​limit (Float32): The up half's soft saturation (linear_beta); 0 passes up through unchanged.

Args:

  • ​gate (Float32): Gate projection value.
  • ​up (Float32): Up projection value.

Returns:

Float32: The activated value.

Was this page helpful?