For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
router_group_limited
def router_group_limited[scores_type: DType, bias_type: DType, //, n_routed_experts: Int, n_experts_per_tok: Int, n_groups: Int, topk_group: Int, norm_weights: Bool](expert_indices: TileTensor[.int32, Engine=expert_indices.Engine, address_space=expert_indices.address_space, linear_idx_type=expert_indices.linear_idx_type], expert_weights: TileTensor[scores_type, Engine=expert_weights.Engine, address_space=expert_weights.address_space, linear_idx_type=expert_weights.linear_idx_type], expert_scores: TileTensor[scores_type, Engine=expert_scores.Engine, address_space=expert_scores.address_space, linear_idx_type=expert_scores.linear_idx_type], expert_bias: TileTensor[bias_type, Engine=expert_bias.Engine, address_space=expert_bias.address_space, linear_idx_type=expert_bias.linear_idx_type], routed_scaling_factor: Float32, context: DeviceContext)
A manually fused MoE router with the group-limited strategy.
Inputs: expert_indices: The indices of the routed experts for each token. Shape: [num_tokens, num_experts_per_tok]. expert_weights: The weights of the routed experts for each token. Shape: [num_tokens, num_experts_per_tok]. expert_scores: The scores for each expert for each token. Shape: [num_tokens, n_routed_experts]. expert_bias: The bias for each expert. Shape: [n_routed_experts]. routed_scaling_factor: The scaling factor for the routed expert weights. context: The device context.
Parameters:
- scores_type (
DType): The data type of the scores and the output weights. - bias_type (
DType): The data type of the expert bias. - n_routed_experts (
Int): The number of experts to route to. - n_experts_per_tok (
Int): The number of experts to be selected per token. - n_groups (
Int): The number of expert groups. - topk_group (
Int): The number of expert groups to be selected per token. - norm_weights (
Bool): Whether to normalize the selected weights.
def router_group_limited[scores_type: DType, bias_type: DType, ScoresInputFnType: def[width: Int](Coord[**?]) -> SIMD[scores_type, width] & RegisterPassable & ImplicitlyCopyable, //, n_routed_experts: Int, n_experts_per_tok: Int, n_groups: Int, topk_group: Int, norm_weights: Bool, packed: Bool = False](expert_indices: TileTensor[.int32, Engine=expert_indices.Engine, address_space=expert_indices.address_space, linear_idx_type=expert_indices.linear_idx_type], expert_weights: TileTensor[scores_type, Engine=expert_weights.Engine, address_space=expert_weights.address_space, linear_idx_type=expert_weights.linear_idx_type], scores_layout: T, scores_input_fn: ScoresInputFnType, expert_bias: TileTensor[bias_type, Engine=expert_bias.Engine, address_space=expert_bias.address_space, linear_idx_type=expert_bias.linear_idx_type], routed_scaling_factor: Float32, context: DeviceContext, packed_p: Optional[Pointer[Scalar[scores_type], MutUntrackedOrigin]] = None)
A manually fused MoE router with the group-limited strategy.
Inputs:
expert_indices: The indices of the routed experts for each token.
Shape: [num_tokens, num_experts_per_tok].
expert_weights: The weights of the routed experts for each token.
Shape: [num_tokens, num_experts_per_tok].
scores_layout: Layout of the routing scores. Shape:
[num_tokens, n_routed_experts].
scores_input_fn: Closure that loads the score at a
[token, expert] coordinate. It runs on the device, so it
must copy what it captures.
expert_bias: The bias for each expert. Shape: [n_routed_experts].
routed_scaling_factor: The scaling factor for the routed expert weights.
context: The device context.
packed_p: With packed, the [num_tokens, 2 * n_experts_per_tok]
output of each token's ids, as scores_type, then its weights.
Parameters:
- scores_type (
DType): The data type of the scores and the output weights. - bias_type (
DType): The data type of the expert bias. - ScoresInputFnType (
def[width: Int](Coord[**?]) -> SIMD[scores_type, width]&RegisterPassable&ImplicitlyCopyable): Type of the closure that loads the routing scores (inferred). - n_routed_experts (
Int): The number of experts to route to. - n_experts_per_tok (
Int): The number of experts to be selected per token. - n_groups (
Int): The number of expert groups. - topk_group (
Int): The number of expert groups to be selected per token. - norm_weights (
Bool): Whether to normalize the selected weights. - packed (
Bool): Also writepacked_p.