For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
geglu_tanh
def geglu_tanh(gate: Float32, up: Float32) -> Float32
Computes GeGLU with the tanh approximation of GELU.
Returns gelu(gate) * up, where
gelu(x) = 0.5 * x * (1 + tanh(0.7978845608028654 * (x + 0.044715 * x^3)))
in the operation order of nn.activations.gelu_tanh.
Args:
Returns:
Float32: The activated value.