For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
ep_fused_combine_kernel_api
def ep_fused_combine_kernel_api[combine_dtype: DType, AtomicCountersEngine: TensorEngine, InputTokensEngine: TensorEngine, SrcInfoEngine: TensorEngine, SendPtrsEngine: TensorEngine, RecvPtrsEngine: TensorEngine, RecvCountPtrsEngine: TensorEngine, StoreFnType: EPCombineStoreFn, //, hidden_size: Int, top_k: Int, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, fused_shared_expert: Bool = False, skip_a2a: Bool = False, use_shmem: Bool = (n_nodes > Int(1)), allreduce_world_size: Int = Int(1), p2p_small_max_tokens: Int = Int(32), *, OutputTokensEngine: TensorEngine = DefaultEngine](num_tokens: Int, atomic_counters: TileTensor[.int32, Engine=AtomicCountersEngine, address_space=atomic_counters.address_space, linear_idx_type=atomic_counters.linear_idx_type], input_tokens: TileTensor[combine_dtype, Engine=InputTokensEngine, address_space=input_tokens.address_space, linear_idx_type=input_tokens.linear_idx_type], src_info: TileTensor[.int32, Engine=SrcInfoEngine, address_space=src_info.address_space, linear_idx_type=src_info.linear_idx_type], send_ptrs: TileTensor[.uint64, Engine=SendPtrsEngine, address_space=send_ptrs.address_space, linear_idx_type=send_ptrs.linear_idx_type], recv_ptrs: TileTensor[.uint64, Engine=RecvPtrsEngine, address_space=recv_ptrs.address_space, linear_idx_type=recv_ptrs.linear_idx_type], recv_count_ptrs: TileTensor[.uint64, Engine=RecvCountPtrsEngine, address_space=recv_count_ptrs.address_space, linear_idx_type=recv_count_ptrs.linear_idx_type], context: DeviceContext, store_fn: StoreFnType, topk_ids_p: Optional[Pointer[Int32, ImmUntrackedOrigin]] = None)
Execute the fused Expert Parallelism combine kernel, unweighted.
Each routed expert's output is stored on its own. See the overload taking
router_weights_fn.
Arguments:
num_tokens: Number of output tokens.
atomic_counters: EP kernel synchronization counters.
input_tokens: Expert output tokens to send back.
src_info: Source routing information from dispatch phase.
send_ptrs: Send buffer pointers for each local GPU.
recv_ptrs: Receive buffer pointers for each local GPU.
recv_count_ptrs: Receive count buffer pointers for each local GPU.
context: Device context pointer.
store_fn: Stores the output tokens, called with
(token_idx, topk_idx, elem). It runs on the device, so it must
copy what it captures.
topk_ids_p: Pointer to the top-k IDs for each token, only required if
skip_a2a is True.
Parameters:
- combine_dtype (
DType): Data type for tokens during combine phase. - AtomicCountersEngine (
TensorEngine): Storage engine backing theatomic_counterstile. - InputTokensEngine (
TensorEngine): Storage engine backing theinput_tokenstile. - SrcInfoEngine (
TensorEngine): Storage engine backing thesrc_infotile. - SendPtrsEngine (
TensorEngine): Storage engine backing thesend_ptrstile. - RecvPtrsEngine (
TensorEngine): Storage engine backing therecv_ptrstile. - RecvCountPtrsEngine (
TensorEngine): Storage engine backing therecv_count_ptrstile. - StoreFnType (
EPCombineStoreFn): The type ofstore_fn(inferred). - hidden_size (
Int): Model hidden dimension size. - top_k (
Int): Number of experts each token was routed to. - n_experts (
Int): Total experts across all devices. - max_token_per_rank (
Int): Maximum tokens any device can receive. - n_gpus_per_node (
Int): GPUs per physical node. - n_nodes (
Int): Number of physical nodes. - fused_shared_expert (
Bool): Whether to add shared expert outputs to the combined routed expert outputs. - skip_a2a (
Bool): Whether to skip the A2A communication. If true, we will only send tokens within the current device. - use_shmem (
Bool): Whether to enable SHMEM communication. - allreduce_world_size (
Int): The world size of the allreduce operation. Only needed for skip_a2a. Used to calculate the workload distribution for the shared expert (if has one). - p2p_small_max_tokens (
Int): Up to this many tokens on every device, a single-node P2P combine with a fused shared expert on AMD GPUs runs the single-launchp2p_small_combine_kernelinstead, and the steps it declines take the partial-row combine whenep_p2p_prefill_onholds; both needatomic_countersto end in the small path's words. Must match the dispatch's; 0 keeps the protocol kernels. - OutputTokensEngine (
TensorEngine): Storage engine backing the output tokens tile.
def ep_fused_combine_kernel_api[combine_dtype: DType, AtomicCountersEngine: TensorEngine, InputTokensEngine: TensorEngine, SrcInfoEngine: TensorEngine, SendPtrsEngine: TensorEngine, RecvPtrsEngine: TensorEngine, RecvCountPtrsEngine: TensorEngine, StoreFnType: EPCombineStoreFn, RouterWeightsFnType: RouterWeightsFn, //, hidden_size: Int, top_k: Int, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, fused_shared_expert: Bool = False, skip_a2a: Bool = False, use_shmem: Bool = (n_nodes > Int(1)), allreduce_world_size: Int = Int(1), has_router_weights: Bool = True, p2p_small_max_tokens: Int = Int(32), *, OutputTokensEngine: TensorEngine = DefaultEngine](num_tokens: Int, atomic_counters: TileTensor[.int32, Engine=AtomicCountersEngine, address_space=atomic_counters.address_space, linear_idx_type=atomic_counters.linear_idx_type], input_tokens: TileTensor[combine_dtype, Engine=InputTokensEngine, address_space=input_tokens.address_space, linear_idx_type=input_tokens.linear_idx_type], src_info: TileTensor[.int32, Engine=SrcInfoEngine, address_space=src_info.address_space, linear_idx_type=src_info.linear_idx_type], send_ptrs: TileTensor[.uint64, Engine=SendPtrsEngine, address_space=send_ptrs.address_space, linear_idx_type=send_ptrs.linear_idx_type], recv_ptrs: TileTensor[.uint64, Engine=RecvPtrsEngine, address_space=recv_ptrs.address_space, linear_idx_type=recv_ptrs.linear_idx_type], recv_count_ptrs: TileTensor[.uint64, Engine=RecvCountPtrsEngine, address_space=recv_count_ptrs.address_space, linear_idx_type=recv_count_ptrs.linear_idx_type], context: DeviceContext, store_fn: StoreFnType, router_weights_fn: RouterWeightsFnType, topk_ids_p: Optional[Pointer[Int32, ImmUntrackedOrigin]] = None)
Execute the fused Expert Parallelism combine kernel.
This function launches the fused combine_kernel from ep_comm.mojo that combines both combine_async and combine_wait functionality in a single kernel launch. It sends expert outputs back to their original devices, then waits for all transfers to complete and computes the weighted sum of routed expert outputs for each token.
Arguments:
num_tokens: Number of output tokens.
atomic_counters: EP kernel synchronization counters.
input_tokens: Expert output tokens to send back.
src_info: Source routing information from dispatch phase.
send_ptrs: Send buffer pointers for each local GPU.
recv_ptrs: Receive buffer pointers for each local GPU.
recv_count_ptrs: Receive count buffer pointers for each local GPU.
context: Device context pointer.
store_fn: Stores the combined output tokens. Called with
(token_idx, elem) when has_router_weights, and with
(token_idx, topk_idx, elem) otherwise. It runs on the device,
so it must copy what it captures.
router_weights_fn: The router weights. It runs on the device, so it
must copy what it captures.
topk_ids_p: Pointer to the top-k IDs for each token, only required if
skip_a2a is True.
Parameters:
- combine_dtype (
DType): Data type for tokens during combine phase. - AtomicCountersEngine (
TensorEngine): Storage engine backing theatomic_counterstile. - InputTokensEngine (
TensorEngine): Storage engine backing theinput_tokenstile. - SrcInfoEngine (
TensorEngine): Storage engine backing thesrc_infotile. - SendPtrsEngine (
TensorEngine): Storage engine backing thesend_ptrstile. - RecvPtrsEngine (
TensorEngine): Storage engine backing therecv_ptrstile. - RecvCountPtrsEngine (
TensorEngine): Storage engine backing therecv_count_ptrstile. - StoreFnType (
EPCombineStoreFn): The type ofstore_fn(inferred). - RouterWeightsFnType (
RouterWeightsFn): The type ofrouter_weights_fn(inferred). - hidden_size (
Int): Model hidden dimension size. - top_k (
Int): Number of experts each token was routed to. - n_experts (
Int): Total experts across all devices. - max_token_per_rank (
Int): Maximum tokens any device can receive. - n_gpus_per_node (
Int): GPUs per physical node. - n_nodes (
Int): Number of physical nodes. - fused_shared_expert (
Bool): Whether to add shared expert outputs to the combined routed expert outputs. - skip_a2a (
Bool): Whether to skip the A2A communication. If true, we will only send tokens within the current device. - use_shmem (
Bool): Whether to enable SHMEM communication. - allreduce_world_size (
Int): The world size of the allreduce operation. Only needed for skip_a2a. Used to calculate the workload distribution for the shared expert (if has one). - has_router_weights (
Bool): Whether the routed expert outputs are weighted byrouter_weights_fnand summed. The overload withoutrouter_weights_fnpassesFalse. - p2p_small_max_tokens (
Int): Up to this many tokens on every device, a single-node P2P combine with a fused shared expert on AMD GPUs runs the single-launchp2p_small_combine_kernelinstead, and the steps it declines take the partial-row combine whenep_p2p_prefill_onholds; both needatomic_countersto end in the small path's words. Must match the dispatch's; 0 keeps the protocol kernels. - OutputTokensEngine (
TensorEngine): Storage engine backing the output tokens tile.