IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

geglu_tanh

def geglu_tanh(gate: Float32, up: Float32) -> Float32

Computes GeGLU with the tanh approximation of GELU.

Returns gelu(gate) * up, where gelu(x) = 0.5 * x * (1 + tanh(0.7978845608028654 * (x + 0.044715 * x^3))) in the operation order of nn.activations.gelu_tanh.

Args:

  • ​gate (Float32): Gate projection value.
  • ​up (Float32): Up projection value.

Returns:

Float32: The activated value.

Was this page helpful?