For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
ep_combine_wait_kernel_api
def ep_combine_wait_kernel_api[AtomicCountersEngine: TensorEngine, RecvPtrsEngine: TensorEngine, RecvCountPtrsEngine: TensorEngine, StoreFnType: EPCombineStoreFn, //, combine_dtype: DType, hidden_size: Int, top_k: Int, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int](num_tokens: Int, atomic_counters: TileTensor[.int32, Engine=AtomicCountersEngine, address_space=atomic_counters.address_space, linear_idx_type=atomic_counters.linear_idx_type], recv_ptrs: TileTensor[.uint64, Engine=RecvPtrsEngine, address_space=recv_ptrs.address_space, linear_idx_type=recv_ptrs.linear_idx_type], recv_count_ptrs: TileTensor[.uint64, Engine=RecvCountPtrsEngine, address_space=recv_count_ptrs.address_space, linear_idx_type=recv_count_ptrs.linear_idx_type], context: DeviceContext, store_fn: StoreFnType)
Execute the Expert Parallelism combine completion kernel, unweighted.
Each routed expert's output is stored on its own. See the overload taking
router_weights_fn.
Arguments:
num_tokens: Number of output tokens.
atomic_counters: EP kernel synchronization counters.
Used to coordinate between different thread blocks.
recv_ptrs: Receive buffer pointers for each local GPU.
recv_count_ptrs: Receive count buffer pointers for each local GPU.
context: Device context pointer.
store_fn: Stores the output tokens, called with
(token_idx, topk_idx, elem). It runs on the device, so it must
copy what it captures.
Parameters:
- AtomicCountersEngine (
TensorEngine): Storage engine backing theatomic_counterstile. - RecvPtrsEngine (
TensorEngine): Storage engine backing therecv_ptrstile. - RecvCountPtrsEngine (
TensorEngine): Storage engine backing therecv_count_ptrstile. - StoreFnType (
EPCombineStoreFn): The type ofstore_fn(inferred). - combine_dtype (
DType): Data type for tokens during combine phase. - hidden_size (
Int): Model hidden dimension size. - top_k (
Int): Number of experts each token was routed to. - n_experts (
Int): Total experts across all devices. - max_token_per_rank (
Int): Maximum tokens any device can receive. - n_gpus_per_node (
Int): GPUs per physical node. - n_nodes (
Int): Number of physical nodes.
def ep_combine_wait_kernel_api[AtomicCountersEngine: TensorEngine, RecvPtrsEngine: TensorEngine, RecvCountPtrsEngine: TensorEngine, StoreFnType: EPCombineStoreFn, RouterWeightsFnType: RouterWeightsFn, //, combine_dtype: DType, hidden_size: Int, top_k: Int, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, has_router_weights: Bool = True, mx_msg: Bool = False](num_tokens: Int, atomic_counters: TileTensor[.int32, Engine=AtomicCountersEngine, address_space=atomic_counters.address_space, linear_idx_type=atomic_counters.linear_idx_type], recv_ptrs: TileTensor[.uint64, Engine=RecvPtrsEngine, address_space=recv_ptrs.address_space, linear_idx_type=recv_ptrs.linear_idx_type], recv_count_ptrs: TileTensor[.uint64, Engine=RecvCountPtrsEngine, address_space=recv_count_ptrs.address_space, linear_idx_type=recv_count_ptrs.linear_idx_type], context: DeviceContext, store_fn: StoreFnType, router_weights_fn: RouterWeightsFnType)
Execute the Expert Parallelism combine completion kernel.
This function launches the combine_wait_kernel from ep_comm.mojo to complete the token combine phase. It waits for all inter-device communication to complete, then computes the weighted sum of routed expert outputs for each token.
Arguments:
num_tokens: Number of output tokens.
atomic_counters: EP kernel synchronization counters.
Used to coordinate between different thread blocks.
recv_ptrs: Receive buffer pointers for each local GPU.
recv_count_ptrs: Receive count buffer pointers for each local GPU.
context: Device context pointer.
store_fn: Stores the output tokens. Called with (token_idx, elem)
when has_router_weights, and with
(token_idx, topk_idx, elem) otherwise. It runs on the device,
so it must copy what it captures.
router_weights_fn: The router weights. It runs on the device, so it
must copy what it captures.
Parameters:
- AtomicCountersEngine (
TensorEngine): Storage engine backing theatomic_counterstile. - RecvPtrsEngine (
TensorEngine): Storage engine backing therecv_ptrstile. - RecvCountPtrsEngine (
TensorEngine): Storage engine backing therecv_count_ptrstile. - StoreFnType (
EPCombineStoreFn): The type ofstore_fn(inferred). - RouterWeightsFnType (
RouterWeightsFn): The type ofrouter_weights_fn(inferred). - combine_dtype (
DType): Data type for tokens during combine phase. - hidden_size (
Int): Model hidden dimension size. - top_k (
Int): Number of experts each token was routed to. - n_experts (
Int): Total experts across all devices. - max_token_per_rank (
Int): Maximum tokens any device can receive. - n_gpus_per_node (
Int): GPUs per physical node. - n_nodes (
Int): Number of physical nodes. - has_router_weights (
Bool): Whether the routed expert outputs are weighted byrouter_weights_fnand summed. The overload withoutrouter_weights_fnpassesFalse. - mx_msg (
Bool): The messages are MXFP8, asep_combine_async_kernel_apisent them withmx_msg.