IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

ep_dispatch_wait_kernel_api

def ep_dispatch_wait_kernel_api[token_fmt_type: TokenFormat, RowOffsetsEngine: TensorEngine, ExpertIdsEngine: TensorEngine, SrcInfoEngine: TensorEngine, RecvPtrsEngine: TensorEngine, RecvCountPtrsEngine: TensorEngine, AtomicCountersEngine: TensorEngine, //, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, ExpertLoadT: ExpertLoadSink = NullExpertLoad](token_handler: token_fmt_type, row_offsets: TileTensor[.uint32, Engine=RowOffsetsEngine, address_space=row_offsets.address_space, linear_idx_type=row_offsets.linear_idx_type], expert_ids: TileTensor[.int32, Engine=ExpertIdsEngine, address_space=expert_ids.address_space, linear_idx_type=expert_ids.linear_idx_type], src_info: TileTensor[.int32, Engine=SrcInfoEngine, address_space=src_info.address_space, linear_idx_type=src_info.linear_idx_type], recv_ptrs: TileTensor[.uint64, Engine=RecvPtrsEngine, address_space=recv_ptrs.address_space, linear_idx_type=recv_ptrs.linear_idx_type], recv_count_ptrs: TileTensor[.uint64, Engine=RecvCountPtrsEngine, address_space=recv_count_ptrs.address_space, linear_idx_type=recv_count_ptrs.linear_idx_type], atomic_counters: TileTensor[.int32, Engine=AtomicCountersEngine, address_space=atomic_counters.address_space, linear_idx_type=atomic_counters.linear_idx_type], context: DeviceContext, num_input_tokens: Int = Int(-1), expert_load: ExpertLoadT = NullExpertLoad())

Execute the Expert Parallelism dispatch completion kernel.

This function launches the dispatch_wait_kernel from ep_comm.mojo to complete the token dispatch phase. It waits for all inter-device communication to complete, then organizes the received tokens into a format suitable for grouped matmul computation.

Arguments: token_handler: Token handler. Wrapper for the output token tensor. row_offsets: Cumulative token counts for grouped matmul. expert_ids: Local expert IDs for grouped matmul. src_info: Source routing information for combine phase. recv_ptrs: Receive buffer pointers for each local GPU. recv_count_ptrs: Receive count buffer pointers for each local GPU. atomic_counters: EP kernel synchronization counters. context: Device context pointer. num_input_tokens: Per-rank input token count for this layer. When >= 0 enables the decode-fast-path grid sizing. Default -1 keeps the full-sm_count grid for backwards compatibility. expert_load: Receives each local expert's routed-token count, for EPLB load statistics.

Parameters:

  • ​token_fmt_type (TokenFormat): Token format type.
  • ​RowOffsetsEngine (TensorEngine): Storage engine backing the row_offsets tile.
  • ​ExpertIdsEngine (TensorEngine): Storage engine backing the expert_ids tile.
  • ​SrcInfoEngine (TensorEngine): Storage engine backing the src_info tile.
  • ​RecvPtrsEngine (TensorEngine): Storage engine backing the recv_ptrs tile.
  • ​RecvCountPtrsEngine (TensorEngine): Storage engine backing the recv_count_ptrs tile.
  • ​AtomicCountersEngine (TensorEngine): Storage engine backing the atomic_counters tile.
  • ​n_experts (Int): Total experts across all devices.
  • ​max_token_per_rank (Int): Maximum tokens any device can send.
  • ​n_gpus_per_node (Int): GPUs per physical node.
  • ​n_nodes (Int): Number of physical nodes.
  • ​ExpertLoadT (ExpertLoadSink): Type of expert_load; the default NullExpertLoad compiles the counting away.

Was this page helpful?