IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

ep_combine_wait_kernel_api

def ep_combine_wait_kernel_api[AtomicCountersEngine: TensorEngine, RecvPtrsEngine: TensorEngine, RecvCountPtrsEngine: TensorEngine, StoreFnType: EPCombineStoreFn, //, combine_dtype: DType, hidden_size: Int, top_k: Int, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int](num_tokens: Int, atomic_counters: TileTensor[.int32, Engine=AtomicCountersEngine, address_space=atomic_counters.address_space, linear_idx_type=atomic_counters.linear_idx_type], recv_ptrs: TileTensor[.uint64, Engine=RecvPtrsEngine, address_space=recv_ptrs.address_space, linear_idx_type=recv_ptrs.linear_idx_type], recv_count_ptrs: TileTensor[.uint64, Engine=RecvCountPtrsEngine, address_space=recv_count_ptrs.address_space, linear_idx_type=recv_count_ptrs.linear_idx_type], context: DeviceContext, store_fn: StoreFnType)

Execute the Expert Parallelism combine completion kernel, unweighted.

Each routed expert's output is stored on its own. See the overload taking router_weights_fn.

Arguments: num_tokens: Number of output tokens. atomic_counters: EP kernel synchronization counters. Used to coordinate between different thread blocks. recv_ptrs: Receive buffer pointers for each local GPU. recv_count_ptrs: Receive count buffer pointers for each local GPU. context: Device context pointer. store_fn: Stores the output tokens, called with (token_idx, topk_idx, elem). It runs on the device, so it must copy what it captures.

Parameters:

  • ​AtomicCountersEngine (TensorEngine): Storage engine backing the atomic_counters tile.
  • ​RecvPtrsEngine (TensorEngine): Storage engine backing the recv_ptrs tile.
  • ​RecvCountPtrsEngine (TensorEngine): Storage engine backing the recv_count_ptrs tile.
  • ​StoreFnType (EPCombineStoreFn): The type of store_fn (inferred).
  • ​combine_dtype (DType): Data type for tokens during combine phase.
  • ​hidden_size (Int): Model hidden dimension size.
  • ​top_k (Int): Number of experts each token was routed to.
  • ​n_experts (Int): Total experts across all devices.
  • ​max_token_per_rank (Int): Maximum tokens any device can receive.
  • ​n_gpus_per_node (Int): GPUs per physical node.
  • ​n_nodes (Int): Number of physical nodes.

def ep_combine_wait_kernel_api[AtomicCountersEngine: TensorEngine, RecvPtrsEngine: TensorEngine, RecvCountPtrsEngine: TensorEngine, StoreFnType: EPCombineStoreFn, RouterWeightsFnType: RouterWeightsFn, //, combine_dtype: DType, hidden_size: Int, top_k: Int, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, has_router_weights: Bool = True, mx_msg: Bool = False](num_tokens: Int, atomic_counters: TileTensor[.int32, Engine=AtomicCountersEngine, address_space=atomic_counters.address_space, linear_idx_type=atomic_counters.linear_idx_type], recv_ptrs: TileTensor[.uint64, Engine=RecvPtrsEngine, address_space=recv_ptrs.address_space, linear_idx_type=recv_ptrs.linear_idx_type], recv_count_ptrs: TileTensor[.uint64, Engine=RecvCountPtrsEngine, address_space=recv_count_ptrs.address_space, linear_idx_type=recv_count_ptrs.linear_idx_type], context: DeviceContext, store_fn: StoreFnType, router_weights_fn: RouterWeightsFnType)

Execute the Expert Parallelism combine completion kernel.

This function launches the combine_wait_kernel from ep_comm.mojo to complete the token combine phase. It waits for all inter-device communication to complete, then computes the weighted sum of routed expert outputs for each token.

Arguments: num_tokens: Number of output tokens. atomic_counters: EP kernel synchronization counters. Used to coordinate between different thread blocks. recv_ptrs: Receive buffer pointers for each local GPU. recv_count_ptrs: Receive count buffer pointers for each local GPU. context: Device context pointer. store_fn: Stores the output tokens. Called with (token_idx, elem) when has_router_weights, and with (token_idx, topk_idx, elem) otherwise. It runs on the device, so it must copy what it captures. router_weights_fn: The router weights. It runs on the device, so it must copy what it captures.

Parameters:

  • ​AtomicCountersEngine (TensorEngine): Storage engine backing the atomic_counters tile.
  • ​RecvPtrsEngine (TensorEngine): Storage engine backing the recv_ptrs tile.
  • ​RecvCountPtrsEngine (TensorEngine): Storage engine backing the recv_count_ptrs tile.
  • ​StoreFnType (EPCombineStoreFn): The type of store_fn (inferred).
  • ​RouterWeightsFnType (RouterWeightsFn): The type of router_weights_fn (inferred).
  • ​combine_dtype (DType): Data type for tokens during combine phase.
  • ​hidden_size (Int): Model hidden dimension size.
  • ​top_k (Int): Number of experts each token was routed to.
  • ​n_experts (Int): Total experts across all devices.
  • ​max_token_per_rank (Int): Maximum tokens any device can receive.
  • ​n_gpus_per_node (Int): GPUs per physical node.
  • ​n_nodes (Int): Number of physical nodes.
  • ​has_router_weights (Bool): Whether the routed expert outputs are weighted by router_weights_fn and summed. The overload without router_weights_fn passes False.
  • ​mx_msg (Bool): The messages are MXFP8, as ep_combine_async_kernel_api sent them with mx_msg.

Was this page helpful?