For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
ep_dispatch_wait_kernel_api
def ep_dispatch_wait_kernel_api[token_fmt_type: TokenFormat, RowOffsetsEngine: TensorEngine, ExpertIdsEngine: TensorEngine, SrcInfoEngine: TensorEngine, RecvPtrsEngine: TensorEngine, RecvCountPtrsEngine: TensorEngine, AtomicCountersEngine: TensorEngine, //, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, ExpertLoadT: ExpertLoadSink = NullExpertLoad](token_handler: token_fmt_type, row_offsets: TileTensor[.uint32, Engine=RowOffsetsEngine, address_space=row_offsets.address_space, linear_idx_type=row_offsets.linear_idx_type], expert_ids: TileTensor[.int32, Engine=ExpertIdsEngine, address_space=expert_ids.address_space, linear_idx_type=expert_ids.linear_idx_type], src_info: TileTensor[.int32, Engine=SrcInfoEngine, address_space=src_info.address_space, linear_idx_type=src_info.linear_idx_type], recv_ptrs: TileTensor[.uint64, Engine=RecvPtrsEngine, address_space=recv_ptrs.address_space, linear_idx_type=recv_ptrs.linear_idx_type], recv_count_ptrs: TileTensor[.uint64, Engine=RecvCountPtrsEngine, address_space=recv_count_ptrs.address_space, linear_idx_type=recv_count_ptrs.linear_idx_type], atomic_counters: TileTensor[.int32, Engine=AtomicCountersEngine, address_space=atomic_counters.address_space, linear_idx_type=atomic_counters.linear_idx_type], context: DeviceContext, num_input_tokens: Int = Int(-1), expert_load: ExpertLoadT = NullExpertLoad())
Execute the Expert Parallelism dispatch completion kernel.
This function launches the dispatch_wait_kernel from ep_comm.mojo to complete the token dispatch phase. It waits for all inter-device communication to complete, then organizes the received tokens into a format suitable for grouped matmul computation.
Arguments:
token_handler: Token handler. Wrapper for the output token tensor.
row_offsets: Cumulative token counts for grouped matmul.
expert_ids: Local expert IDs for grouped matmul.
src_info: Source routing information for combine phase.
recv_ptrs: Receive buffer pointers for each local GPU.
recv_count_ptrs: Receive count buffer pointers for each local GPU.
atomic_counters: EP kernel synchronization counters.
context: Device context pointer.
num_input_tokens: Per-rank input token count for this layer. When >= 0
enables the decode-fast-path grid sizing. Default -1 keeps the
full-sm_count grid for backwards compatibility.
expert_load: Receives each local expert's routed-token count, for
EPLB load statistics.
Parameters:
- token_fmt_type (
TokenFormat): Token format type. - RowOffsetsEngine (
TensorEngine): Storage engine backing therow_offsetstile. - ExpertIdsEngine (
TensorEngine): Storage engine backing theexpert_idstile. - SrcInfoEngine (
TensorEngine): Storage engine backing thesrc_infotile. - RecvPtrsEngine (
TensorEngine): Storage engine backing therecv_ptrstile. - RecvCountPtrsEngine (
TensorEngine): Storage engine backing therecv_count_ptrstile. - AtomicCountersEngine (
TensorEngine): Storage engine backing theatomic_counterstile. - n_experts (
Int): Total experts across all devices. - max_token_per_rank (
Int): Maximum tokens any device can send. - n_gpus_per_node (
Int): GPUs per physical node. - n_nodes (
Int): Number of physical nodes. - ExpertLoadT (
ExpertLoadSink): Type ofexpert_load; the defaultNullExpertLoadcompiles the counting away.