For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
moe_exit_act_concat
def moe_exit_act_concat[dtype: DType, simd_width: Int, //, LatentFnType: def[width: Int](Coord[**?]) -> SIMD[dtype, width] & RegisterPassable & ImplicitlyCopyable, GateUpFnType: def[width: Int](Coord[**?]) -> SIMD[dtype, width] & RegisterPassable & ImplicitlyCopyable](output: TileTensor[dtype, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type], latent_fn: LatentFnType, gate_up_fn: GateUpFnType, latent_cols: Int, half_cols: Int, beta: Float32, linear_beta: Float32, context: DeviceContext)
Writes a normed latent and a SiTU-and-mul beside each other, one launch.
The MoE exit reads [normed latent | shared activation] as one K axis:
the routed experts' normed latent followed by the shared expert's
SiTU-and-mul of its fused [gate ; up] projection. Computing the
activation into its own buffer and concatenating afterwards is two
launches over tens of KB, so this writes both halves of the destination
directly -- the latent half as a copy, the shared half activated.
Parameters:
- dtype (
DType): Element type of the destination and of both inputs. - simd_width (
Int): Elements per thread's vector; must divide the latent width and the shared half so no vector straddles them. - LatentFnType (
def[width: Int](Coord[**?]) -> SIMD[dtype, width]&RegisterPassable&ImplicitlyCopyable): The latent input's load closure. - GateUpFnType (
def[width: Int](Coord[**?]) -> SIMD[dtype, width]&RegisterPassable&ImplicitlyCopyable): The[gate ; up]input's load closure.
Args:
- output (
TileTensor[dtype, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type]):[num_tokens, latent_cols + half_cols], the exit GEMM's activation. - latent_fn (
LatentFnType): Loads[num_tokens, latent_cols], the normed latent. - gate_up_fn (
GateUpFnType): Loads[num_tokens, 2 * half_cols], the shared expert's fused gate/up projection, gate half first. - latent_cols (
Int): Latent width, the output's leading columns. - half_cols (
Int): Shared FFN width, the gate/up input's half width. - beta (
Float32):config.activation_situ_beta; the gate's tanh scale. - linear_beta (
Float32):config.activation_situ_linear_beta, or 0 to pass the up half through unchanged. - context (
DeviceContext): The device context.