For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
swiglu_activation
Gated activations for the fused SwiGLU epilogues.
Kernels that fuse a gated activation take it as a comptime thin function, so
each activation compiles to its own specialization. get_swiglu_activation
builds one from a name and its f32 scalar constants.
comptime values
get_swiglu_activation
comptime get_swiglu_activation[name: StringSpan[ImmStaticOrigin], alpha: Float32 = 0, limit: Float32 = 0] = _named_swiglu_activation[name, alpha, limit]
The activation for a name, for a kernel's activation parameter. Unknown names fail to compile.
Parameters
swiglu_activation_type
comptime swiglu_activation_type = def(Float32, Float32) thin -> Float32
Gated activation (gate, up) -> out.
Thin so it can ride as a comptime parameter; any constants it uses must be comptime values.
Functions
-
apply_swiglu_activation: Applies a gated activation lane by lane. -
geglu_tanh: Computes GeGLU with the tanh approximation of GELU. -
swiglu: Computes the SwiGLU activationsilu(gate) * up. -
swigluoai: Computes the clampedswigluoaiactivation.