IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

DistributedEPCombine

struct DistributedEPCombine

Registers the mo.distributed.ep.combine graph op with the graph compiler.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[Outs: TypeList[Outs.values], Tokens: TypeList[Tokens.values], SrcInfos: TypeList[SrcInfos.values], SendPtrs: TypeList[SendPtrs.values], RecvPtrs: TypeList[RecvPtrs.values], RecvCountPtrs: TypeList[RecvCountPtrs.values], RouterWeights: TypeList[RouterWeights.values], Counters: TypeList[Counters.values], hidden_size: Int, top_k: Int, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, fused_shared_expert: Bool, //, _trace_name: StaticString](output_tokens: TensorTuple[*Outs.values], input_tokens: TensorTuple[*Tokens.values], src_info: TensorTuple[*SrcInfos.values], send_ptrs: TensorTuple[*SendPtrs.values], recv_ptrs: TensorTuple[*RecvPtrs.values], recv_count_ptrs: TensorTuple[*RecvCountPtrs.values], router_weights: TensorTuple[*RouterWeights.values], atomic_counters: TensorTuple[*Counters.values], dev_ctxs: DeviceContextArray)

Multi-device fused Expert Parallelism combine with output fusion.

Parameters:

  • ​Outs (TypeList[values]): The element types of output_tokens (inferred).
  • ​Tokens (TypeList[values]): The element types of input_tokens (inferred).
  • ​SrcInfos (TypeList[values]): The element types of src_info (inferred).
  • ​SendPtrs (TypeList[values]): The element types of send_ptrs (inferred).
  • ​RecvPtrs (TypeList[values]): The element types of recv_ptrs (inferred).
  • ​RecvCountPtrs (TypeList[values]): The element types of recv_count_ptrs (inferred).
  • ​RouterWeights (TypeList[values]): The element types of router_weights (inferred).
  • ​Counters (TypeList[values]): The element types of atomic_counters (inferred).
  • ​hidden_size (Int): Size of the model's hidden dimension.
  • ​top_k (Int): Number of experts each token is routed to.
  • ​n_experts (Int): Total number of experts across all GPUs.
  • ​max_token_per_rank (Int): Maximum number of tokens per GPU.
  • ​n_gpus_per_node (Int): Number of GPUs per node.
  • ​n_nodes (Int): Number of physical nodes.
  • ​fused_shared_expert (Bool): Whether a shared expert is fused into the combine kernel.
  • ​_trace_name (StaticString): Trace label for this op.

Args:

Was this page helpful?