IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

router_group_limited

def router_group_limited[scores_type: DType, bias_type: DType, //, n_routed_experts: Int, n_experts_per_tok: Int, n_groups: Int, topk_group: Int, norm_weights: Bool](expert_indices: TileTensor[.int32, Engine=expert_indices.Engine, address_space=expert_indices.address_space, linear_idx_type=expert_indices.linear_idx_type], expert_weights: TileTensor[scores_type, Engine=expert_weights.Engine, address_space=expert_weights.address_space, linear_idx_type=expert_weights.linear_idx_type], expert_scores: TileTensor[scores_type, Engine=expert_scores.Engine, address_space=expert_scores.address_space, linear_idx_type=expert_scores.linear_idx_type], expert_bias: TileTensor[bias_type, Engine=expert_bias.Engine, address_space=expert_bias.address_space, linear_idx_type=expert_bias.linear_idx_type], routed_scaling_factor: Float32, context: DeviceContext)

A manually fused MoE router with the group-limited strategy.

Reference: https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/model.py#L566.

Inputs: expert_indices: The indices of the routed experts for each token. Shape: [num_tokens, num_experts_per_tok]. expert_weights: The weights of the routed experts for each token. Shape: [num_tokens, num_experts_per_tok]. expert_scores: The scores for each expert for each token. Shape: [num_tokens, n_routed_experts]. expert_bias: The bias for each expert. Shape: [n_routed_experts]. routed_scaling_factor: The scaling factor for the routed expert weights. context: The device context.

Parameters:

  • ​scores_type (DType): The data type of the scores and the output weights.
  • ​bias_type (DType): The data type of the expert bias.
  • ​n_routed_experts (Int): The number of experts to route to.
  • ​n_experts_per_tok (Int): The number of experts to be selected per token.
  • ​n_groups (Int): The number of expert groups.
  • ​topk_group (Int): The number of expert groups to be selected per token.
  • ​norm_weights (Bool): Whether to normalize the selected weights.

def router_group_limited[scores_type: DType, bias_type: DType, ScoresInputFnType: def[width: Int](Coord[**?]) -> SIMD[scores_type, width] & RegisterPassable & ImplicitlyCopyable, //, n_routed_experts: Int, n_experts_per_tok: Int, n_groups: Int, topk_group: Int, norm_weights: Bool, packed: Bool = False](expert_indices: TileTensor[.int32, Engine=expert_indices.Engine, address_space=expert_indices.address_space, linear_idx_type=expert_indices.linear_idx_type], expert_weights: TileTensor[scores_type, Engine=expert_weights.Engine, address_space=expert_weights.address_space, linear_idx_type=expert_weights.linear_idx_type], scores_layout: T, scores_input_fn: ScoresInputFnType, expert_bias: TileTensor[bias_type, Engine=expert_bias.Engine, address_space=expert_bias.address_space, linear_idx_type=expert_bias.linear_idx_type], routed_scaling_factor: Float32, context: DeviceContext, packed_p: Optional[Pointer[Scalar[scores_type], MutUntrackedOrigin]] = None)

A manually fused MoE router with the group-limited strategy.

Reference: https://github.com/deepseek-ai/DeepSeek-V3/blob/9b4e9788e4a3a731f7567338ed15d3ec549ce03b/inference/model.py#L566.

Inputs: expert_indices: The indices of the routed experts for each token. Shape: [num_tokens, num_experts_per_tok]. expert_weights: The weights of the routed experts for each token. Shape: [num_tokens, num_experts_per_tok]. scores_layout: Layout of the routing scores. Shape: [num_tokens, n_routed_experts]. scores_input_fn: Closure that loads the score at a [token, expert] coordinate. It runs on the device, so it must copy what it captures. expert_bias: The bias for each expert. Shape: [n_routed_experts]. routed_scaling_factor: The scaling factor for the routed expert weights. context: The device context. packed_p: With packed, the [num_tokens, 2 * n_experts_per_tok] output of each token's ids, as scores_type, then its weights.

Parameters:

  • ​scores_type (DType): The data type of the scores and the output weights.
  • ​bias_type (DType): The data type of the expert bias.
  • ​ScoresInputFnType (def[width: Int](Coord[**?]) -> SIMD[scores_type, width] & RegisterPassable & ImplicitlyCopyable): Type of the closure that loads the routing scores (inferred).
  • ​n_routed_experts (Int): The number of experts to route to.
  • ​n_experts_per_tok (Int): The number of experts to be selected per token.
  • ​n_groups (Int): The number of expert groups.
  • ​topk_group (Int): The number of expert groups to be selected per token.
  • ​norm_weights (Bool): Whether to normalize the selected weights.
  • ​packed (Bool): Also write packed_p.

Was this page helpful?