IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

DistributedEPDispatch

struct DistributedEPDispatch

Registers the mo.distributed.ep.dispatch graph op with the graph compiler.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[OutTokens: TypeList[OutTokens.values], RowOffsets: TypeList[RowOffsets.values], ExpertIds: TypeList[ExpertIds.values], SrcInfos: TypeList[SrcInfos.values], InTokens: TypeList[InTokens.values], TopkIds: TypeList[TopkIds.values], SendPtrs: TypeList[SendPtrs.values], RecvPtrs: TypeList[RecvPtrs.values], RecvCountPtrs: TypeList[RecvCountPtrs.values], Counters: TypeList[Counters.values], hidden_size: Int, top_k: Int, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, fused_shared_expert: Bool, //, _trace_name: StaticString](output_tokens: TensorTuple[*OutTokens.values], row_offsets: TensorTuple[*RowOffsets.values], expert_ids: TensorTuple[*ExpertIds.values], src_info: TensorTuple[*SrcInfos.values], input_tokens: TensorTuple[*InTokens.values], topk_ids: TensorTuple[*TopkIds.values], send_ptrs: TensorTuple[*SendPtrs.values], recv_ptrs: TensorTuple[*RecvPtrs.values], recv_count_ptrs: TensorTuple[*RecvCountPtrs.values], atomic_counters: TensorTuple[*Counters.values], dev_ctxs: DeviceContextArray)

Multi-device fused Expert Parallelism BF16 dispatch.

Parameters:

  • ​OutTokens (TypeList[values]): The element types of output_tokens (inferred).
  • ​RowOffsets (TypeList[values]): The element types of row_offsets (inferred).
  • ​ExpertIds (TypeList[values]): The element types of expert_ids (inferred).
  • ​SrcInfos (TypeList[values]): The element types of src_info (inferred).
  • ​InTokens (TypeList[values]): The element types of input_tokens (inferred).
  • ​TopkIds (TypeList[values]): The element types of topk_ids (inferred).
  • ​SendPtrs (TypeList[values]): The element types of send_ptrs (inferred).
  • ​RecvPtrs (TypeList[values]): The element types of recv_ptrs (inferred).
  • ​RecvCountPtrs (TypeList[values]): The element types of recv_count_ptrs (inferred).
  • ​Counters (TypeList[values]): The element types of atomic_counters (inferred).
  • ​hidden_size (Int): Size of the model's hidden dimension (inferred).
  • ​top_k (Int): Number of experts each token is routed to (inferred).
  • ​n_experts (Int): Total number of experts across all GPUs (inferred).
  • ​max_token_per_rank (Int): Maximum number of tokens per GPU (inferred).
  • ​n_gpus_per_node (Int): Number of GPUs per node (inferred).
  • ​n_nodes (Int): Number of physical nodes (inferred).
  • ​fused_shared_expert (Bool): Whether a shared expert is fused into the dispatch kernel (inferred).
  • ​_trace_name (StaticString): Trace label for this op.

Args:

Was this page helpful?