For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
moe_create_indices_with_slots
def moe_create_indices_with_slots[input_type: DType, //, slot_stride: Int](token_expert_order: TileTensor[.uint32, Engine=token_expert_order.Engine, address_space=token_expert_order.address_space, linear_idx_type=token_expert_order.linear_idx_type], expert_start_indices: TileTensor[.uint32, Engine=expert_start_indices.Engine, address_space=expert_start_indices.address_space, linear_idx_type=expert_start_indices.linear_idx_type], restore_token_order: TileTensor[.int32, Engine=restore_token_order.Engine, address_space=restore_token_order.address_space, linear_idx_type=restore_token_order.linear_idx_type], expert_ids: TileTensor[.int32, Engine=expert_ids.Engine, address_space=expert_ids.address_space, linear_idx_type=expert_ids.linear_idx_type], expert_usage_stats: TileTensor[.uint32, Engine=expert_usage_stats.Engine, address_space=expert_usage_stats.address_space, linear_idx_type=expert_usage_stats.linear_idx_type], scales_offset_p: MutPointer[UInt32, MutAnyOrigin], slot_token_p: MutPointer[Int32, MutAnyOrigin], slot_weight_p: MutPointer[Float32, MutAnyOrigin], scale_slot_p: MutPointer[Int32, MutAnyOrigin], scale_rows: Int, topk_ids: TileTensor[input_type, Engine=topk_ids.Engine, address_space=topk_ids.address_space, linear_idx_type=topk_ids.linear_idx_type], slot_weights_p: MutPointer[Float32, MutAnyOrigin], expert_offset: Int, local_experts: Int, context: DeviceContext)
moe_create_indices plus the row maps of a slot-packed grouped GEMM.
For a grouped GEMM that reads the routed slots sorted by group, packed,
and its A-side scales from 128-row tiles per group located through
scales_offset (the layout of the with.scales.offset variant). The
extra outputs replace the graph ops that would otherwise derive them from
the CSR, and match them exactly:
slot_token[p] = order[p] // slot_stride: the token row of sorted slotp, withslot_strideslots per token.slot_weight[p] = slot_weights[order[p]].scale_slot[r]for each ofscale_rowsscale rows: withaligned_startthe exclusive prefix sum of the group counts rounded up to 128, rowrin[aligned_start[g], aligned_start[g + 1])holds sorted slotstart[g] + r - aligned_start[g]while that is belowstart[g + 1], else 0; rows pastaligned_start[num_experts]hold 0.
Ids are mapped to groups first: id - expert_offset when that is in
[0, local_experts), else local_experts. With num_experts == local_experts + 1 that last group gathers the slots of experts held
elsewhere (expert parallel); with num_experts == local_experts and a
zero offset, ids map to themselves.
restore_token_order is int32 here, as a gather index; the values are the
same as the uint32 of moe_create_indices.
Parameters:
- input_type (
DType): DType of the topk_ids tensor. - slot_stride (
Int): Slots per token (n_experts_per_tok).
Args:
- token_expert_order (
TileTensor[.uint32, Engine=token_expert_order.Engine, address_space=token_expert_order.address_space, linear_idx_type=token_expert_order.linear_idx_type]): Asmoe_create_indices. - expert_start_indices (
TileTensor[.uint32, Engine=expert_start_indices.Engine, address_space=expert_start_indices.address_space, linear_idx_type=expert_start_indices.linear_idx_type]): Asmoe_create_indices. - restore_token_order (
TileTensor[.int32, Engine=restore_token_order.Engine, address_space=restore_token_order.address_space, linear_idx_type=restore_token_order.linear_idx_type]): Asmoe_create_indices, int32. - expert_ids (
TileTensor[.int32, Engine=expert_ids.Engine, address_space=expert_ids.address_space, linear_idx_type=expert_ids.linear_idx_type]): Asmoe_create_indices. - expert_usage_stats (
TileTensor[.uint32, Engine=expert_usage_stats.Engine, address_space=expert_usage_stats.address_space, linear_idx_type=expert_usage_stats.linear_idx_type]): Asmoe_create_indices. - scales_offset_p (
MutPointer[UInt32, MutAnyOrigin]):[num_experts]aligned scale offsets. - slot_token_p (
MutPointer[Int32, MutAnyOrigin]):[num_slots]token row of each sorted slot. - slot_weight_p (
MutPointer[Float32, MutAnyOrigin]):[num_slots]weight of each sorted slot. - scale_slot_p (
MutPointer[Int32, MutAnyOrigin]):[scale_rows]sorted slot of each scale row. - scale_rows (
Int): Scale rows to fill, at leastaligned_start[num_experts](num_slots + 128 * num_expertsalways is). - topk_ids (
TileTensor[input_type, Engine=topk_ids.Engine, address_space=topk_ids.address_space, linear_idx_type=topk_ids.linear_idx_type]): Input 1D tensor of expert IDs, one per slot. - slot_weights_p (
MutPointer[Float32, MutAnyOrigin]):[num_slots]routing weight of each slot, intopk_idsorder. - expert_offset (
Int): Global id of the first local expert. - local_experts (
Int): Experts held locally. - context (
DeviceContext): The device context.