IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

fused_silu_mx_kernel

def fused_silu_mx_kernel[quant_dtype: DType, scales_dtype: DType, input_dtype: DType, output_layout: TensorLayout, scales_layout: TensorLayout, input_layout: TensorLayout, offsets_layout: TensorLayout, num_threads: Int, num_sms: Int, *, activation: def(Float32, Float32) thin -> Float32 = swiglu[SIMD(0)]](output_tensor: TileTensor[quant_dtype, output_layout, MutUntrackedOrigin], scales_tensor: TileTensor[scales_dtype, scales_layout, MutUntrackedOrigin], input_tensor: TileTensor[input_dtype, input_layout, ImmUntrackedOrigin], row_offsets: TileTensor[.uint32, offsets_layout, ImmUntrackedOrigin], max_padded_M: Int32)

This kernel performs the gated activation for all the MLPs in the EP MoE module. We need to manually implement the kernel here is because after the EP dispatch phase, the actual number of received tokens is not known to the host. This kernel will read the row offsets to determine the actual number of received tokens in the input tensor.

Once the activation is applied, the output tensor will be quantized to the MX format, with the E8M0 scales written into the grouped matmul's per-expert scale_4d slot layout (max_padded_M rows per expert). The activation is the comptime activation parameter (defaults to swiglu).

Arguments: output_tensor: The output tensor to store the result. scales_tensor: The per-expert scale_4d slot buffer for the scales. input_tensor: The input tensor to perform the SILU operation. row_offsets: The row offsets to determine the actual number of received tokens. max_padded_M: Per-expert slot stride in rows; a positive multiple of 32.

Was this page helpful?