IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

moe_exit_act_concat

def moe_exit_act_concat[dtype: DType, simd_width: Int, //, LatentFnType: def[width: Int](Coord[**?]) -> SIMD[dtype, width] & RegisterPassable & ImplicitlyCopyable, GateUpFnType: def[width: Int](Coord[**?]) -> SIMD[dtype, width] & RegisterPassable & ImplicitlyCopyable](output: TileTensor[dtype, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type], latent_fn: LatentFnType, gate_up_fn: GateUpFnType, latent_cols: Int, half_cols: Int, beta: Float32, linear_beta: Float32, context: DeviceContext)

Writes a normed latent and a SiTU-and-mul beside each other, one launch.

The MoE exit reads [normed latent | shared activation] as one K axis: the routed experts' normed latent followed by the shared expert's SiTU-and-mul of its fused [gate ; up] projection. Computing the activation into its own buffer and concatenating afterwards is two launches over tens of KB, so this writes both halves of the destination directly -- the latent half as a copy, the shared half activated.

Parameters:

  • ​dtype (DType): Element type of the destination and of both inputs.
  • ​simd_width (Int): Elements per thread's vector; must divide the latent width and the shared half so no vector straddles them.
  • ​LatentFnType (def[width: Int](Coord[**?]) -> SIMD[dtype, width] & RegisterPassable & ImplicitlyCopyable): The latent input's load closure.
  • ​GateUpFnType (def[width: Int](Coord[**?]) -> SIMD[dtype, width] & RegisterPassable & ImplicitlyCopyable): The [gate ; up] input's load closure.

Args:

  • ​output (TileTensor[dtype, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type]): [num_tokens, latent_cols + half_cols], the exit GEMM's activation.
  • ​latent_fn (LatentFnType): Loads [num_tokens, latent_cols], the normed latent.
  • ​gate_up_fn (GateUpFnType): Loads [num_tokens, 2 * half_cols], the shared expert's fused gate/up projection, gate half first.
  • ​latent_cols (Int): Latent width, the output's leading columns.
  • ​half_cols (Int): Shared FFN width, the gate/up input's half width.
  • ​beta (Float32): config.activation_situ_beta; the gate's tanh scale.
  • ​linear_beta (Float32): config.activation_situ_linear_beta, or 0 to pass the up half through unchanged.
  • ​context (DeviceContext): The device context.

Was this page helpful?