For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
swiglu_activation
Gated activations for the fused SwiGLU epilogues.
Kernels that fuse a gated activation take it as a comptime thin function, so
each activation compiles to its own specialization. get_swiglu_activation
builds one from a name and its f32 scalar constants.
comptime values
get_swiglu_activation
comptime get_swiglu_activation[name: StaticString, alpha: Float32 = 0, limit: Float32 = 0] = comptime #kgen.func.symbol<@linalg::@swiglu_activation::@"_named_swiglu_activation[::StringSpan[ImmStaticOrigin, ::Origin[False, ImmStaticOrigin]()],::SIMD[DType.float32, 1],::SIMD[DType.float32, 1]](::SIMD[DType.float32, 1],::SIMD[DType.float32, 1])"<:!kgen.param<:meta<!lit.struct<@std::@collections::@string::@string_span::@StringSpan<:origin<false> #lit.origin.field<#lit.static.origin : !lit.origin<false>, "__constants__">, :!lit.struct<@std::@origin::@__init__::@Origin<:!lit.struct<@std::@builtin::@bool::@Bool> {:scalar<bool> false}, :origin<false> #lit.origin.field<#lit.static.origin : !lit.origin<false>, "__constants__">>> sugar_alias(*"ImmStaticOrigin`0x11", {})>>> sugar_alias(*"StaticString`0x3", @std::@collections::@string::@string_span::@StringSpan<:origin<false> #lit.origin.field<#lit.static.origin : !lit.origin<false>, "__constants__">, :!lit.struct<@std::@origin::@__init__::@Origin<:!lit.struct<@std::@builtin::@bool::@Bool> {:scalar<bool> false}, :origin<false> #lit.origin.field<#lit.static.origin : !lit.origin<false>, "__constants__">>> sugar_alias(*"ImmStaticOrigin`0x11", {})>)> name, :!lit.struct<@std::@simd::@SIMD<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype f32}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}>> alpha, :!lit.struct<@std::@simd::@SIMD<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype f32}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}>> limit>>
The activation for a name, for a kernel's activation parameter. Unknown names fail to compile.
Parameters
- name (
StaticString): Activation name, as the Python graph API spells it. - alpha (
Float32): Constant for activations that take one; ignored otherwise. - limit (
Float32): Constant for activations that take one; ignored otherwise.
swiglu_activation_type
comptime swiglu_activation_type = def(Float32, Float32) thin -> Float32
Gated activation (gate, up) -> out.
Thin so it can ride as a comptime parameter; any constants it uses must be comptime values.
Functions
-
apply_swiglu_activation: Applies a gated activation lane by lane. -
geglu_tanh: Computes GeGLU with the tanh approximation of GELU. -
situ: Computes the SiTU gated activation. -
swiglu: Computes the SwiGLU activationsilu(gate) * up. -
swigluoai: Computes the clampedswigluoaiactivation.