For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
single_group_router
def single_group_router[scores_type: DType, bias_type: DType, //, n_routed_experts: Int, n_experts_per_tok: Int, norm_weights: Bool](expert_indices: TileTensor[.int32, Engine=expert_indices.Engine, address_space=expert_indices.address_space, linear_idx_type=expert_indices.linear_idx_type], expert_weight: TileTensor[scores_type, Engine=expert_weight.Engine, address_space=expert_weight.address_space, linear_idx_type=expert_weight.linear_idx_type], expert_scores: TileTensor[scores_type, Engine=expert_scores.Engine, address_space=expert_scores.address_space, linear_idx_type=expert_scores.linear_idx_type], expert_bias: TileTensor[bias_type, Engine=expert_bias.Engine, address_space=expert_bias.address_space, linear_idx_type=expert_bias.linear_idx_type], routed_scaling_factor: Float32, context: DeviceContext)
Launch the single-group MoE router on GPU.
One warp per token. Selects the top n_experts_per_tok experts by corrected score, ties to the lower expert.
Inputs: expert_indices: Output expert indices. Shape: [num_tokens, n_experts_per_tok]. expert_weights: Output expert weights. Shape: [num_tokens, n_experts_per_tok]. expert_scores: Input routing scores. Shape: [num_tokens, n_routed_experts]. expert_bias: Per-expert correction bias used for selection only. routed_scaling_factor: Scalar multiplied into every output weight. context: The device context.
Parameters:
- scores_type (
DType): DType of routing scores and output weights. - bias_type (
DType): DType of the expert correction bias. - n_routed_experts (
Int): Total number of experts. - n_experts_per_tok (
Int): Experts selected per token. - norm_weights (
Bool): If True, normalize selected weights to sum to 1 before applying routed_scaling_factor.
def single_group_router[scores_type: DType, bias_type: DType, ScoresInputFnType: def[width: Int](Coord[**?]) -> SIMD[scores_type, width] & RegisterPassable & ImplicitlyCopyable, //, n_routed_experts: Int, n_experts_per_tok: Int, norm_weights: Bool, packed: Bool = False](expert_indices: TileTensor[.int32, Engine=expert_indices.Engine, address_space=expert_indices.address_space, linear_idx_type=expert_indices.linear_idx_type], expert_weight: TileTensor[scores_type, Engine=expert_weight.Engine, address_space=expert_weight.address_space, linear_idx_type=expert_weight.linear_idx_type], scores_layout: T, scores_input_fn: ScoresInputFnType, expert_bias: TileTensor[bias_type, Engine=expert_bias.Engine, address_space=expert_bias.address_space, linear_idx_type=expert_bias.linear_idx_type], routed_scaling_factor: Float32, context: DeviceContext, packed_p: Optional[Pointer[Scalar[scores_type], MutAnyOrigin]] = None)
Launch the single-group MoE router on GPU, loading scores via a lambda.
The kernel reads scores only through scores_input_fn, so the scores
tensor is described by its layout alone.
Inputs:
expert_indices: Output expert indices. Shape: [num_tokens, n_experts_per_tok].
expert_weights: Output expert weights. Shape: [num_tokens, n_experts_per_tok].
scores_layout: Layout of the routing scores. Shape:
[num_tokens, n_routed_experts].
scores_input_fn: Closure that loads the score at a
[token, expert] coordinate. It runs on the device, so it
must copy what it captures.
expert_bias: Per-expert correction bias used for selection only.
routed_scaling_factor: Scalar multiplied into every output weight.
context: The device context.
packed_p: With packed, the [num_tokens, 2 * n_experts_per_tok]
output of each token's ids, as scores_type, then its weights.
Parameters:
- scores_type (
DType): DType of routing scores and output weights. - bias_type (
DType): DType of the expert correction bias. - ScoresInputFnType (
def[width: Int](Coord[**?]) -> SIMD[scores_type, width]&RegisterPassable&ImplicitlyCopyable): Type of the closure that loads the routing scores (inferred). - n_routed_experts (
Int): Total number of experts. - n_experts_per_tok (
Int): Experts selected per token, at most the warp size. - norm_weights (
Bool): If True, normalize selected weights to sum to 1 before applying routed_scaling_factor. - packed (
Bool): Also writepacked_p(Muse Spark's decode routing).
def single_group_router[scores_type: DType, bias_type: DType, ids_type: DType, ScoresInputFnType: def[width: Int](Coord[**?]) -> SIMD[scores_type, width] & RegisterPassable & ImplicitlyCopyable, //, n_routed_experts: Int, n_experts_per_tok: Int, norm_weights: Bool](expert_indices: TileTensor[.int32, Engine=expert_indices.Engine, address_space=expert_indices.address_space, linear_idx_type=expert_indices.linear_idx_type], expert_weight: TileTensor[scores_type, Engine=expert_weight.Engine, address_space=expert_weight.address_space, linear_idx_type=expert_weight.linear_idx_type], scores_layout: T, scores_input_fn: ScoresInputFnType, expert_bias: TileTensor[bias_type, Engine=expert_bias.Engine, address_space=expert_bias.address_space, linear_idx_type=expert_bias.linear_idx_type], image_bias: TileTensor[bias_type, Engine=image_bias.Engine, address_space=image_bias.address_space, linear_idx_type=image_bias.linear_idx_type], token_ids: TileTensor[ids_type, Engine=token_ids.Engine, address_space=token_ids.address_space, linear_idx_type=token_ids.linear_idx_type], image_token_id: Int64, routed_scaling_factor: Float32, context: DeviceContext)
Launch the single-group MoE router on GPU, selecting with image_bias for tokens whose id is image_token_id and with expert_bias for the rest.
Inputs:
expert_indices: Output expert indices. Shape: [num_tokens, n_experts_per_tok].
expert_weights: Output expert weights. Shape: [num_tokens, n_experts_per_tok].
scores_layout: Layout of the routing scores. Shape:
[num_tokens, n_routed_experts].
scores_input_fn: Closure that loads the score at a
[token, expert] coordinate. It runs on the device, so it
must copy what it captures.
expert_bias: Per-expert selection bias of the other tokens.
image_bias: Per-expert selection bias of the image tokens.
token_ids: Token ids. Shape: [num_tokens].
image_token_id: The id that marks an image token.
routed_scaling_factor: Scalar multiplied into every output weight.
context: The device context.
Parameters:
- scores_type (
DType): DType of routing scores and output weights. - bias_type (
DType): DType of both selection biases. - ids_type (
DType): DType of the token ids. - ScoresInputFnType (
def[width: Int](Coord[**?]) -> SIMD[scores_type, width]&RegisterPassable&ImplicitlyCopyable): Type of the closure that loads the routing scores (inferred). - n_routed_experts (
Int): Total number of experts. - n_experts_per_tok (
Int): Experts selected per token. - norm_weights (
Bool): If True, normalize selected weights to sum to 1 before applying routed_scaling_factor.
def single_group_router[scores_type: DType, ScoresInputFnType: def[width: Int](Coord[**?]) -> SIMD[scores_type, width] & RegisterPassable & ImplicitlyCopyable, BiasInputFnType: def[width: Int](Coord[**?]) -> SIMD[scores_type, width] & RegisterPassable & ImplicitlyCopyable, //, n_routed_experts: Int, n_experts_per_tok: Int, norm_weights: Bool, packed: Bool = False](expert_indices: TileTensor[.int32, Engine=expert_indices.Engine, address_space=expert_indices.address_space, linear_idx_type=expert_indices.linear_idx_type], expert_weight: TileTensor[scores_type, Engine=expert_weight.Engine, address_space=expert_weight.address_space, linear_idx_type=expert_weight.linear_idx_type], scores_layout: T, scores_input_fn: ScoresInputFnType, bias_input_fn: BiasInputFnType, routed_scaling_factor: Float32, context: DeviceContext, packed_p: Optional[Pointer[Scalar[scores_type], MutAnyOrigin]] = None)
Launch the single-group MoE router on GPU, loading scores and the selection bias via lambdas.
bias_input_fn takes the same [token, expert] coordinate as the
scores, so the selection bias may differ per token.
Inputs:
expert_indices: Output expert indices. Shape: [num_tokens, n_experts_per_tok].
expert_weights: Output expert weights. Shape: [num_tokens, n_experts_per_tok].
scores_layout: Layout of the routing scores. Shape:
[num_tokens, n_routed_experts].
scores_input_fn: Closure that loads the score at a
[token, expert] coordinate. It runs on the device, so it
must copy what it captures.
bias_input_fn: Closure that loads the selection bias at a
[token, expert] coordinate, in scores_type. It runs on the
device, so it must copy what it captures.
routed_scaling_factor: Scalar multiplied into every output weight.
context: The device context.
packed_p: With packed, the [num_tokens, 2 * n_experts_per_tok]
output of each token's ids, as scores_type, then its weights.
Parameters:
- scores_type (
DType): DType of routing scores and output weights. - ScoresInputFnType (
def[width: Int](Coord[**?]) -> SIMD[scores_type, width]&RegisterPassable&ImplicitlyCopyable): Type of the closure that loads the routing scores (inferred). - BiasInputFnType (
def[width: Int](Coord[**?]) -> SIMD[scores_type, width]&RegisterPassable&ImplicitlyCopyable): Type of the closure that loads the selection bias (inferred). - n_routed_experts (
Int): Total number of experts. - n_experts_per_tok (
Int): Experts selected per token. - norm_weights (
Bool): If True, normalize selected weights to sum to 1 before applying routed_scaling_factor. An all-zero selection keeps zero weights. - packed (
Bool): Also writepacked_p(Muse Spark's decode routing).