IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

moe_create_indices_with_slots

def moe_create_indices_with_slots[input_type: DType, //, slot_stride: Int](token_expert_order: TileTensor[.uint32, Engine=token_expert_order.Engine, address_space=token_expert_order.address_space, linear_idx_type=token_expert_order.linear_idx_type], expert_start_indices: TileTensor[.uint32, Engine=expert_start_indices.Engine, address_space=expert_start_indices.address_space, linear_idx_type=expert_start_indices.linear_idx_type], restore_token_order: TileTensor[.int32, Engine=restore_token_order.Engine, address_space=restore_token_order.address_space, linear_idx_type=restore_token_order.linear_idx_type], expert_ids: TileTensor[.int32, Engine=expert_ids.Engine, address_space=expert_ids.address_space, linear_idx_type=expert_ids.linear_idx_type], expert_usage_stats: TileTensor[.uint32, Engine=expert_usage_stats.Engine, address_space=expert_usage_stats.address_space, linear_idx_type=expert_usage_stats.linear_idx_type], scales_offset_p: MutPointer[UInt32, MutAnyOrigin], slot_token_p: MutPointer[Int32, MutAnyOrigin], slot_weight_p: MutPointer[Float32, MutAnyOrigin], scale_slot_p: MutPointer[Int32, MutAnyOrigin], scale_rows: Int, topk_ids: TileTensor[input_type, Engine=topk_ids.Engine, address_space=topk_ids.address_space, linear_idx_type=topk_ids.linear_idx_type], slot_weights_p: MutPointer[Float32, MutAnyOrigin], expert_offset: Int, local_experts: Int, context: DeviceContext)

moe_create_indices plus the row maps of a slot-packed grouped GEMM.

For a grouped GEMM that reads the routed slots sorted by group, packed, and its A-side scales from 128-row tiles per group located through scales_offset (the layout of the with.scales.offset variant). The extra outputs replace the graph ops that would otherwise derive them from the CSR, and match them exactly:

  • slot_token[p] = order[p] // slot_stride: the token row of sorted slot p, with slot_stride slots per token.
  • slot_weight[p] = slot_weights[order[p]].
  • scale_slot[r] for each of scale_rows scale rows: with aligned_start the exclusive prefix sum of the group counts rounded up to 128, row r in [aligned_start[g], aligned_start[g + 1]) holds sorted slot start[g] + r - aligned_start[g] while that is below start[g + 1], else 0; rows past aligned_start[num_experts] hold 0.

Ids are mapped to groups first: id - expert_offset when that is in [0, local_experts), else local_experts. With num_experts == local_experts + 1 that last group gathers the slots of experts held elsewhere (expert parallel); with num_experts == local_experts and a zero offset, ids map to themselves.

restore_token_order is int32 here, as a gather index; the values are the same as the uint32 of moe_create_indices.

Parameters:

  • ​input_type (DType): DType of the topk_ids tensor.
  • ​slot_stride (Int): Slots per token (n_experts_per_tok).

Args:

Was this page helpful?