IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

single_group_router_kernel

def single_group_router_kernel[scores_type: DType, ExpertIndicesLayoutType: TensorLayout, ExpertWeightsLayoutType: TensorLayout, ExpertScoresLayoutType: TensorLayout, n_routed_experts: Int, n_experts_per_tok: Int, norm_weights: Bool, num_threads: Int, ScoresInputFnType: def[width: Int](Coord[**?]) -> SIMD[scores_type, width] & RegisterPassable & ImplicitlyCopyable, BiasInputFnType: def[width: Int](Coord[**?]) -> SIMD[scores_type, width] & RegisterPassable & ImplicitlyCopyable, *, ExpertIndicesEngine: TensorEngine, ExpertWeightsEngine: TensorEngine, packed: Bool = False](expert_indices: TileTensor[.int32, ExpertIndicesLayoutType, MutAnyOrigin, Engine=ExpertIndicesEngine], expert_weights: TileTensor[scores_type, ExpertWeightsLayoutType, MutAnyOrigin, Engine=ExpertWeightsEngine], num_tokens: Int32, routed_scaling_factor: Float32, packed_p: Optional[Pointer[Scalar[scores_type], MutAnyOrigin]], scores_input_fn: ScoresInputFnType, bias_input_fn: BiasInputFnType)

Single-group MoE router kernel. One warp per token.

Fuses: corrected = scores + bias β†’ top-k selection β†’ weight = the selected scores β†’ optional normalize β†’ scale. Each lane holds a few adjacent experts of every warp-wide span, and each of the n_experts_per_tok rounds takes the warp's largest corrected score, ties to the lower expert. Scores and the selection bias are read only through scores_input_fn and bias_input_fn, closures that load the value at a (token, expert) index, so the bias may differ per token. With packed, each token's ids, as scores_type, then its weights are also written to its [2 * n_experts_per_tok] row of packed_p.

Was this page helpful?