For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
DistributedEPCombine
struct DistributedEPCombine
Registers the mo.distributed.ep.combine graph op with the graph compiler.
Implemented traits
Methods
execute
static def execute[Outs: TypeList[Outs.values], Tokens: TypeList[Tokens.values], SrcInfos: TypeList[SrcInfos.values], SendPtrs: TypeList[SendPtrs.values], RecvPtrs: TypeList[RecvPtrs.values], RecvCountPtrs: TypeList[RecvCountPtrs.values], RouterWeights: TypeList[RouterWeights.values], Counters: TypeList[Counters.values], hidden_size: Int, top_k: Int, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, fused_shared_expert: Bool, //, _trace_name: StaticString](output_tokens: TensorTuple[*Outs.values], input_tokens: TensorTuple[*Tokens.values], src_info: TensorTuple[*SrcInfos.values], send_ptrs: TensorTuple[*SendPtrs.values], recv_ptrs: TensorTuple[*RecvPtrs.values], recv_count_ptrs: TensorTuple[*RecvCountPtrs.values], router_weights: TensorTuple[*RouterWeights.values], atomic_counters: TensorTuple[*Counters.values], dev_ctxs: DeviceContextArray)
Multi-device fused Expert Parallelism combine with output fusion.
Parameters:
- Outs (
TypeList[values]): The element types ofoutput_tokens(inferred). - Tokens (
TypeList[values]): The element types ofinput_tokens(inferred). - SrcInfos (
TypeList[values]): The element types ofsrc_info(inferred). - SendPtrs (
TypeList[values]): The element types ofsend_ptrs(inferred). - RecvPtrs (
TypeList[values]): The element types ofrecv_ptrs(inferred). - RecvCountPtrs (
TypeList[values]): The element types ofrecv_count_ptrs(inferred). - RouterWeights (
TypeList[values]): The element types ofrouter_weights(inferred). - Counters (
TypeList[values]): The element types ofatomic_counters(inferred). - hidden_size (
Int): Size of the model's hidden dimension. - top_k (
Int): Number of experts each token is routed to. - n_experts (
Int): Total number of experts across all GPUs. - max_token_per_rank (
Int): Maximum number of tokens per GPU. - n_gpus_per_node (
Int): Number of GPUs per node. - n_nodes (
Int): Number of physical nodes. - fused_shared_expert (
Bool): Whether a shared expert is fused into the combine kernel. - _trace_name (
StaticString): Trace label for this op.
Args:
- output_tokens (
TensorTuple[*Outs.values]): Fused output variadic tensors storing the combined expert outputs, one per device. - input_tokens (
TensorTuple[*Tokens.values]): Input variadic tensors of expert-processed tokens to combine, one per device. - src_info (
TensorTuple[*SrcInfos.values]): Source info tensors recording the originating rank and token index for each received token, one per device. - send_ptrs (
TensorTuple[*SendPtrs.values]): Send buffer pointers for the combine phase, one per device. - recv_ptrs (
TensorTuple[*RecvPtrs.values]): Receive buffer pointers for the combine phase, one per device. - recv_count_ptrs (
TensorTuple[*RecvCountPtrs.values]): Receive count buffer pointers tracking tokens received per expert, one per device. - router_weights (
TensorTuple[*RouterWeights.values]): Router weights used to scale each expert's contribution, one per device. - atomic_counters (
TensorTuple[*Counters.values]): Atomic counters coordinating work across thread blocks, one per device. - dev_ctxs (
DeviceContextArray): List of GPU device contexts, one per device.