IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

single_group_router

def single_group_router[scores_type: DType, bias_type: DType, //, n_routed_experts: Int, n_experts_per_tok: Int, norm_weights: Bool](expert_indices: TileTensor[.int32, Engine=expert_indices.Engine, address_space=expert_indices.address_space, linear_idx_type=expert_indices.linear_idx_type], expert_weight: TileTensor[scores_type, Engine=expert_weight.Engine, address_space=expert_weight.address_space, linear_idx_type=expert_weight.linear_idx_type], expert_scores: TileTensor[scores_type, Engine=expert_scores.Engine, address_space=expert_scores.address_space, linear_idx_type=expert_scores.linear_idx_type], expert_bias: TileTensor[bias_type, Engine=expert_bias.Engine, address_space=expert_bias.address_space, linear_idx_type=expert_bias.linear_idx_type], routed_scaling_factor: Float32, context: DeviceContext)

Launch the single-group MoE router on GPU.

One warp per token. Selects the top n_experts_per_tok experts by corrected score, ties to the lower expert.

Inputs: expert_indices: Output expert indices. Shape: [num_tokens, n_experts_per_tok]. expert_weights: Output expert weights. Shape: [num_tokens, n_experts_per_tok]. expert_scores: Input routing scores. Shape: [num_tokens, n_routed_experts]. expert_bias: Per-expert correction bias used for selection only. routed_scaling_factor: Scalar multiplied into every output weight. context: The device context.

Parameters:

  • ​scores_type (DType): DType of routing scores and output weights.
  • ​bias_type (DType): DType of the expert correction bias.
  • ​n_routed_experts (Int): Total number of experts.
  • ​n_experts_per_tok (Int): Experts selected per token.
  • ​norm_weights (Bool): If True, normalize selected weights to sum to 1 before applying routed_scaling_factor.

def single_group_router[scores_type: DType, bias_type: DType, ScoresInputFnType: def[width: Int](Coord[**?]) -> SIMD[scores_type, width] & RegisterPassable & ImplicitlyCopyable, //, n_routed_experts: Int, n_experts_per_tok: Int, norm_weights: Bool, packed: Bool = False](expert_indices: TileTensor[.int32, Engine=expert_indices.Engine, address_space=expert_indices.address_space, linear_idx_type=expert_indices.linear_idx_type], expert_weight: TileTensor[scores_type, Engine=expert_weight.Engine, address_space=expert_weight.address_space, linear_idx_type=expert_weight.linear_idx_type], scores_layout: T, scores_input_fn: ScoresInputFnType, expert_bias: TileTensor[bias_type, Engine=expert_bias.Engine, address_space=expert_bias.address_space, linear_idx_type=expert_bias.linear_idx_type], routed_scaling_factor: Float32, context: DeviceContext, packed_p: Optional[Pointer[Scalar[scores_type], MutAnyOrigin]] = None)

Launch the single-group MoE router on GPU, loading scores via a lambda.

The kernel reads scores only through scores_input_fn, so the scores tensor is described by its layout alone.

Inputs: expert_indices: Output expert indices. Shape: [num_tokens, n_experts_per_tok]. expert_weights: Output expert weights. Shape: [num_tokens, n_experts_per_tok]. scores_layout: Layout of the routing scores. Shape: [num_tokens, n_routed_experts]. scores_input_fn: Closure that loads the score at a [token, expert] coordinate. It runs on the device, so it must copy what it captures. expert_bias: Per-expert correction bias used for selection only. routed_scaling_factor: Scalar multiplied into every output weight. context: The device context. packed_p: With packed, the [num_tokens, 2 * n_experts_per_tok] output of each token's ids, as scores_type, then its weights.

Parameters:

  • ​scores_type (DType): DType of routing scores and output weights.
  • ​bias_type (DType): DType of the expert correction bias.
  • ​ScoresInputFnType (def[width: Int](Coord[**?]) -> SIMD[scores_type, width] & RegisterPassable & ImplicitlyCopyable): Type of the closure that loads the routing scores (inferred).
  • ​n_routed_experts (Int): Total number of experts.
  • ​n_experts_per_tok (Int): Experts selected per token, at most the warp size.
  • ​norm_weights (Bool): If True, normalize selected weights to sum to 1 before applying routed_scaling_factor.
  • ​packed (Bool): Also write packed_p (Muse Spark's decode routing).

def single_group_router[scores_type: DType, bias_type: DType, ids_type: DType, ScoresInputFnType: def[width: Int](Coord[**?]) -> SIMD[scores_type, width] & RegisterPassable & ImplicitlyCopyable, //, n_routed_experts: Int, n_experts_per_tok: Int, norm_weights: Bool](expert_indices: TileTensor[.int32, Engine=expert_indices.Engine, address_space=expert_indices.address_space, linear_idx_type=expert_indices.linear_idx_type], expert_weight: TileTensor[scores_type, Engine=expert_weight.Engine, address_space=expert_weight.address_space, linear_idx_type=expert_weight.linear_idx_type], scores_layout: T, scores_input_fn: ScoresInputFnType, expert_bias: TileTensor[bias_type, Engine=expert_bias.Engine, address_space=expert_bias.address_space, linear_idx_type=expert_bias.linear_idx_type], image_bias: TileTensor[bias_type, Engine=image_bias.Engine, address_space=image_bias.address_space, linear_idx_type=image_bias.linear_idx_type], token_ids: TileTensor[ids_type, Engine=token_ids.Engine, address_space=token_ids.address_space, linear_idx_type=token_ids.linear_idx_type], image_token_id: Int64, routed_scaling_factor: Float32, context: DeviceContext)

Launch the single-group MoE router on GPU, selecting with image_bias for tokens whose id is image_token_id and with expert_bias for the rest.

Inputs: expert_indices: Output expert indices. Shape: [num_tokens, n_experts_per_tok]. expert_weights: Output expert weights. Shape: [num_tokens, n_experts_per_tok]. scores_layout: Layout of the routing scores. Shape: [num_tokens, n_routed_experts]. scores_input_fn: Closure that loads the score at a [token, expert] coordinate. It runs on the device, so it must copy what it captures. expert_bias: Per-expert selection bias of the other tokens. image_bias: Per-expert selection bias of the image tokens. token_ids: Token ids. Shape: [num_tokens]. image_token_id: The id that marks an image token. routed_scaling_factor: Scalar multiplied into every output weight. context: The device context.

Parameters:

  • ​scores_type (DType): DType of routing scores and output weights.
  • ​bias_type (DType): DType of both selection biases.
  • ​ids_type (DType): DType of the token ids.
  • ​ScoresInputFnType (def[width: Int](Coord[**?]) -> SIMD[scores_type, width] & RegisterPassable & ImplicitlyCopyable): Type of the closure that loads the routing scores (inferred).
  • ​n_routed_experts (Int): Total number of experts.
  • ​n_experts_per_tok (Int): Experts selected per token.
  • ​norm_weights (Bool): If True, normalize selected weights to sum to 1 before applying routed_scaling_factor.

def single_group_router[scores_type: DType, ScoresInputFnType: def[width: Int](Coord[**?]) -> SIMD[scores_type, width] & RegisterPassable & ImplicitlyCopyable, BiasInputFnType: def[width: Int](Coord[**?]) -> SIMD[scores_type, width] & RegisterPassable & ImplicitlyCopyable, //, n_routed_experts: Int, n_experts_per_tok: Int, norm_weights: Bool, packed: Bool = False](expert_indices: TileTensor[.int32, Engine=expert_indices.Engine, address_space=expert_indices.address_space, linear_idx_type=expert_indices.linear_idx_type], expert_weight: TileTensor[scores_type, Engine=expert_weight.Engine, address_space=expert_weight.address_space, linear_idx_type=expert_weight.linear_idx_type], scores_layout: T, scores_input_fn: ScoresInputFnType, bias_input_fn: BiasInputFnType, routed_scaling_factor: Float32, context: DeviceContext, packed_p: Optional[Pointer[Scalar[scores_type], MutAnyOrigin]] = None)

Launch the single-group MoE router on GPU, loading scores and the selection bias via lambdas.

bias_input_fn takes the same [token, expert] coordinate as the scores, so the selection bias may differ per token.

Inputs: expert_indices: Output expert indices. Shape: [num_tokens, n_experts_per_tok]. expert_weights: Output expert weights. Shape: [num_tokens, n_experts_per_tok]. scores_layout: Layout of the routing scores. Shape: [num_tokens, n_routed_experts]. scores_input_fn: Closure that loads the score at a [token, expert] coordinate. It runs on the device, so it must copy what it captures. bias_input_fn: Closure that loads the selection bias at a [token, expert] coordinate, in scores_type. It runs on the device, so it must copy what it captures. routed_scaling_factor: Scalar multiplied into every output weight. context: The device context. packed_p: With packed, the [num_tokens, 2 * n_experts_per_tok] output of each token's ids, as scores_type, then its weights.

Parameters:

  • ​scores_type (DType): DType of routing scores and output weights.
  • ​ScoresInputFnType (def[width: Int](Coord[**?]) -> SIMD[scores_type, width] & RegisterPassable & ImplicitlyCopyable): Type of the closure that loads the routing scores (inferred).
  • ​BiasInputFnType (def[width: Int](Coord[**?]) -> SIMD[scores_type, width] & RegisterPassable & ImplicitlyCopyable): Type of the closure that loads the selection bias (inferred).
  • ​n_routed_experts (Int): Total number of experts.
  • ​n_experts_per_tok (Int): Experts selected per token.
  • ​norm_weights (Bool): If True, normalize selected weights to sum to 1 before applying routed_scaling_factor. An all-zero selection keeps zero weights.
  • ​packed (Bool): Also write packed_p (Muse Spark's decode routing).

Was this page helpful?