For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
combine_kernel
def combine_kernel[input_type: DType, num_threads: Int, input_tokens_layout: TensorLayout, src_info_layout: TensorLayout, n_sms: Int, top_k: Int, n_experts: Int, n_ranks: Int, msg_bytes: Int, max_tokens_per_rank: Int, p2p_world_size: Int, StoreFnType: EPCombineStoreFn, RouterWeightsFnType: RouterWeightsFn, has_router_weights: Bool, fused_shared_expert: Bool = False, skip_a2a: Bool = False, use_shmem: Bool = True, allreduce_world_size: Int = Int(1), p2p_small_counter_words: Int = Int(0), *, InputTokensEngine: TensorEngine = DefaultEngine, SrcInfoEngine: TensorEngine = DefaultEngine](input_tokens: TileTensor[input_type, input_tokens_layout, ImmUntrackedOrigin, Engine=InputTokensEngine], src_info: TileTensor[.int32, src_info_layout, ImmUntrackedOrigin, Engine=SrcInfoEngine], num_tokens: Int32, send_buf_p: MutPointer[UInt8, MutUntrackedOrigin], recv_buf_ptrs: Array[Pointer[UInt8, MutUntrackedOrigin], p2p_world_size], recv_count_ptrs: Array[Pointer[UInt64, MutUntrackedOrigin], p2p_world_size], ep_counters: EPLocalSyncCounters[n_experts], topk_ids_p: Optional[Pointer[Int32, ImmUntrackedOrigin]], my_rank: Int32, store_fn: StoreFnType, router_weights_fn: RouterWeightsFnType)
Fused combine kernel that combines combine_async and combine_wait functionality in a single kernel launch.
This kernel sends processed tokens back to their original ranks, then waits for all tokens to arrive and computes the weighted sum of routed expert outputs for each token.
For fused_shared_expert mode, the shared expert outputs are added to the
reduced routed expert outputs before store_fn is called. This requires
has_router_weights (output must be reduced).
Parameters:
- input_type (
DType): The type of the input/output tokens. - num_threads (
Int): The number of threads in the block. - input_tokens_layout (
TensorLayout): The layout of the input tokens. - src_info_layout (
TensorLayout): The layout of the source token info. - n_sms (
Int): The total number of SMs in the device. - top_k (
Int): The number of selected experts per token. - n_experts (
Int): The total number of experts in the model. - n_ranks (
Int): The number of all devices participating in the communication. - msg_bytes (
Int): The number of bytes per token message. - max_tokens_per_rank (
Int): The maximum number of tokens per rank. - p2p_world_size (
Int): Size of a High-speed GPU interconnect group. - StoreFnType (
EPCombineStoreFn): The type ofstore_fn. - RouterWeightsFnType (
RouterWeightsFn): The type ofrouter_weights_fn. - has_router_weights (
Bool): Whether all routed experts' outputs for a token are weighted byrouter_weights_fnand summed. REQUIRED when fused_shared_expert is True. - fused_shared_expert (
Bool): Whether to add the shared expert's output to the combined output. Requireshas_router_weights. - skip_a2a (
Bool): Whether to skip the A2A communication. If true, we will only send tokens within the current device. - use_shmem (
Bool): Whether to use the SHMEM API for the communication. - allreduce_world_size (
Int): The world size of the allreduce operation. Only needed for skip_a2a. Used to calculate the workload distribution for the shared expert (if has one). - p2p_small_counter_words (
Int): When above 0, the counter buffer's elements, which end in the P2P small path's words; the step then returns at once ifp2p_small_combine_kernelcombines it. - InputTokensEngine (
TensorEngine): Storage engine backing theinput_tokenstile. - SrcInfoEngine (
TensorEngine): Storage engine backing thesrc_infotile.
Args:
- input_tokens (
TileTensor[input_type, input_tokens_layout, ImmUntrackedOrigin, Engine=InputTokensEngine]): The tokens to be sent back to the original rank. - src_info (
TileTensor[.int32, src_info_layout, ImmUntrackedOrigin, Engine=SrcInfoEngine]): The source token info tensor. - num_tokens (
Int32): The number of output tokens. - send_buf_p (
MutPointer[UInt8, MutUntrackedOrigin]): The pointer to the send buffer. Need to be allocated usingshmem_allocifuse_shmemis True. - recv_buf_ptrs (
Array[Pointer[UInt8, MutUntrackedOrigin], p2p_world_size]): An array of pointers to the receive buffers for each device in the p2p world. The local device's buffer at index my_p2p_rank is used for both send (combine_async) and receive (combine_wait) operations. - recv_count_ptrs (
Array[Pointer[UInt64, MutUntrackedOrigin], p2p_world_size]): An array of pointers to the receive count buffers for each device in the p2p world. The local device's buffer at index my_p2p_rank is used for receive count tracking. - ep_counters (
EPLocalSyncCounters[n_experts]): EP atomic counters for kernel synchronization. - topk_ids_p (
Optional[Pointer[Int32, ImmUntrackedOrigin]]): Pointer to the top-k IDs for each token, only required if skip_a2a is True. - my_rank (
Int32): The rank of the current device. - store_fn (
StoreFnType): Stores the combined output tokens, passed by host layout. Called with(token_idx, elem)whenhas_router_weights, and with(token_idx, topk_idx, elem)otherwise. - router_weights_fn (
RouterWeightsFnType): The router weights, passed by host layout and called only whenhas_router_weights.