For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
apply_swiglu_activation
def apply_swiglu_activation[dtype: DType, width: SIMDLength, //, activation: def(Float32, Float32) thin -> Float32](gate: SIMD[dtype, width], up: SIMD[dtype, width]) -> SIMD[dtype, width]
Applies a gated activation lane by lane.
Parameters:
- dtype (
DType): Element type of the operands. - width (
SIMDLength): SIMD width of the operands. - activation (
def(Float32, Float32) thin -> Float32): The activation to apply.
Args:
- gate (
SIMD[dtype, width]): Gate projection values. - up (
SIMD[dtype, width]): Up projection values.
Returns:
SIMD[dtype, width]: The activated values.