IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

ep_fused_dispatch_kernel_api

def ep_fused_dispatch_kernel_api[token_fmt_type: TokenFormat, dispatch_dtype: DType, RowOffsetsEngine: TensorEngine, ExpertIdsEngine: TensorEngine, SrcInfoEngine: TensorEngine, AtomicCountersEngine: TensorEngine, InputTokensEngine: TensorEngine, TopkIdsEngine: TensorEngine, SendPtrsEngine: TensorEngine, RecvPtrsEngine: TensorEngine, RecvCountPtrsEngine: TensorEngine, //, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, fused_shared_expert: Bool, skip_a2a: Bool = False, use_shmem: Bool = (n_nodes > Int(1)), allreduce_world_size: Int = Int(1), dispatch_warp_min_tokens: Int = Int(-1), p2p_small_max_tokens: Int = Int(32), ExpertLoadT: ExpertLoadSink = NullExpertLoad](token_handler: token_fmt_type, row_offsets: TileTensor[.uint32, Engine=RowOffsetsEngine, address_space=row_offsets.address_space, linear_idx_type=row_offsets.linear_idx_type], expert_ids: TileTensor[.int32, Engine=ExpertIdsEngine, address_space=expert_ids.address_space, linear_idx_type=expert_ids.linear_idx_type], src_info: TileTensor[.int32, Engine=SrcInfoEngine, address_space=src_info.address_space, linear_idx_type=src_info.linear_idx_type], atomic_counters: TileTensor[.int32, Engine=AtomicCountersEngine, address_space=atomic_counters.address_space, linear_idx_type=atomic_counters.linear_idx_type], input_tokens: TileTensor[dispatch_dtype, Engine=InputTokensEngine, address_space=input_tokens.address_space, linear_idx_type=input_tokens.linear_idx_type], topk_ids: TileTensor[.int32, Engine=TopkIdsEngine, address_space=topk_ids.address_space, linear_idx_type=topk_ids.linear_idx_type], send_ptrs: TileTensor[.uint64, Engine=SendPtrsEngine, address_space=send_ptrs.address_space, linear_idx_type=send_ptrs.linear_idx_type], recv_ptrs: TileTensor[.uint64, Engine=RecvPtrsEngine, address_space=recv_ptrs.address_space, linear_idx_type=recv_ptrs.linear_idx_type], recv_count_ptrs: TileTensor[.uint64, Engine=RecvCountPtrsEngine, address_space=recv_count_ptrs.address_space, linear_idx_type=recv_count_ptrs.linear_idx_type], context: DeviceContext, expert_load: ExpertLoadT = NullExpertLoad())

Execute the fused Expert Parallelism dispatch kernel at unit input scale.

See the overload taking input_scales_fn.

Arguments: token_handler: Token handler. Wrapper for the output token tensor. row_offsets: Row offsets for grouped matmul. expert_ids: Expert IDs for grouped matmul. src_info: Source routing information for combine phase. atomic_counters: EP kernel synchronization counters. input_tokens: Tokens to dispatch to experts. topk_ids: Expert assignments from router. send_ptrs: Send buffer pointers for each local GPU. recv_ptrs: Receive buffer pointers for each local GPU. recv_count_ptrs: Receive count buffer pointers for each local GPU. context: Device context pointer. expert_load: Receives each local expert's routed-token count, for EPLB load statistics.

Parameters:

  • ​token_fmt_type (TokenFormat): Token format type.
  • ​dispatch_dtype (DType): Data type of the dispatched tokens.
  • ​RowOffsetsEngine (TensorEngine): Storage engine backing the row_offsets tile.
  • ​ExpertIdsEngine (TensorEngine): Storage engine backing the expert_ids tile.
  • ​SrcInfoEngine (TensorEngine): Storage engine backing the src_info tile.
  • ​AtomicCountersEngine (TensorEngine): Storage engine backing the atomic_counters tile.
  • ​InputTokensEngine (TensorEngine): Storage engine backing the input_tokens tile.
  • ​TopkIdsEngine (TensorEngine): Storage engine backing the topk_ids tile.
  • ​SendPtrsEngine (TensorEngine): Storage engine backing the send_ptrs tile.
  • ​RecvPtrsEngine (TensorEngine): Storage engine backing the recv_ptrs tile.
  • ​RecvCountPtrsEngine (TensorEngine): Storage engine backing the recv_count_ptrs tile.
  • ​n_experts (Int): Total experts across all devices.
  • ​max_token_per_rank (Int): Maximum tokens any device can send.
  • ​n_gpus_per_node (Int): GPUs per physical node.
  • ​n_nodes (Int): Number of physical nodes.
  • ​fused_shared_expert (Bool): Whether to pack shared expert inputs with routed experts' inputs.
  • ​skip_a2a (Bool): Whether to skip the A2A communication. If true, we will only send tokens within the current device.
  • ​use_shmem (Bool): Whether to enable SHMEM communication.
  • ​allreduce_world_size (Int): The world size of the allreduce operation. Only needed for skip_a2a. Used to calculate the workload distribution for the shared expert (if has one).
  • ​dispatch_warp_min_tokens (Int): From this many tokens a rank on, send one token per warp instead of one per CTA, and count each expert's rows with one shared-memory histogram pass at any count. Same bytes and receive layout. -1 (the default) keeps the stock send and count, as does every configuration but single-node NVFP4 or MXFP8 without a fused shared expert.
  • ​p2p_small_max_tokens (Int): Up to this many tokens on every device, a single-node P2P dispatch with a fused shared expert on AMD GPUs runs the single-launch p2p_small_dispatch_kernel instead, and the steps it declines take the P2P prefill dispatch when ep_p2p_prefill_on holds; both need atomic_counters to end in the small path's words. Must match the combine's; 0 keeps the protocol kernels.
  • ​ExpertLoadT (ExpertLoadSink): Type of expert_load; the default NullExpertLoad compiles the counting away.

def ep_fused_dispatch_kernel_api[token_fmt_type: TokenFormat, dispatch_dtype: DType, RowOffsetsEngine: TensorEngine, ExpertIdsEngine: TensorEngine, SrcInfoEngine: TensorEngine, AtomicCountersEngine: TensorEngine, InputTokensEngine: TensorEngine, TopkIdsEngine: TensorEngine, SendPtrsEngine: TensorEngine, RecvPtrsEngine: TensorEngine, RecvCountPtrsEngine: TensorEngine, InputScalesFnType: InputScalesFn, //, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, fused_shared_expert: Bool, skip_a2a: Bool = False, use_shmem: Bool = (n_nodes > Int(1)), allreduce_world_size: Int = Int(1), has_input_scales: Bool = True, dispatch_warp_min_tokens: Int = Int(-1), p2p_small_max_tokens: Int = Int(32), ExpertLoadT: ExpertLoadSink = NullExpertLoad](token_handler: token_fmt_type, row_offsets: TileTensor[.uint32, Engine=RowOffsetsEngine, address_space=row_offsets.address_space, linear_idx_type=row_offsets.linear_idx_type], expert_ids: TileTensor[.int32, Engine=ExpertIdsEngine, address_space=expert_ids.address_space, linear_idx_type=expert_ids.linear_idx_type], src_info: TileTensor[.int32, Engine=SrcInfoEngine, address_space=src_info.address_space, linear_idx_type=src_info.linear_idx_type], atomic_counters: TileTensor[.int32, Engine=AtomicCountersEngine, address_space=atomic_counters.address_space, linear_idx_type=atomic_counters.linear_idx_type], input_tokens: TileTensor[dispatch_dtype, Engine=InputTokensEngine, address_space=input_tokens.address_space, linear_idx_type=input_tokens.linear_idx_type], topk_ids: TileTensor[.int32, Engine=TopkIdsEngine, address_space=topk_ids.address_space, linear_idx_type=topk_ids.linear_idx_type], send_ptrs: TileTensor[.uint64, Engine=SendPtrsEngine, address_space=send_ptrs.address_space, linear_idx_type=send_ptrs.linear_idx_type], recv_ptrs: TileTensor[.uint64, Engine=RecvPtrsEngine, address_space=recv_ptrs.address_space, linear_idx_type=recv_ptrs.linear_idx_type], recv_count_ptrs: TileTensor[.uint64, Engine=RecvCountPtrsEngine, address_space=recv_count_ptrs.address_space, linear_idx_type=recv_count_ptrs.linear_idx_type], context: DeviceContext, input_scales_fn: InputScalesFnType, expert_load: ExpertLoadT = NullExpertLoad())

Execute the fused Expert Parallelism dispatch kernel.

This function launches the fused dispatch_kernel from ep_comm.mojo that combines both dispatch_async and dispatch_wait functionality in a single kernel launch. It distributes input tokens to expert devices based on top-k routing decisions, then waits for all tokens to arrive and aggregates them for grouped matmul computation.

Arguments: token_handler: Token handler. Wrapper for the output token tensor. row_offsets: Row offsets for grouped matmul. expert_ids: Expert IDs for grouped matmul. src_info: Source routing information for combine phase. atomic_counters: EP kernel synchronization counters. input_tokens: Tokens to dispatch to experts. topk_ids: Expert assignments from router. send_ptrs: Send buffer pointers for each local GPU. recv_ptrs: Receive buffer pointers for each local GPU. recv_count_ptrs: Receive count buffer pointers for each local GPU. context: Device context pointer. input_scales_fn: The input scales. It runs on the device, so it must copy what it captures. expert_load: Receives each local expert's routed-token count, for EPLB load statistics.

Parameters:

  • ​token_fmt_type (TokenFormat): Token format type.
  • ​dispatch_dtype (DType): Data type of the dispatched tokens.
  • ​RowOffsetsEngine (TensorEngine): Storage engine backing the row_offsets tile.
  • ​ExpertIdsEngine (TensorEngine): Storage engine backing the expert_ids tile.
  • ​SrcInfoEngine (TensorEngine): Storage engine backing the src_info tile.
  • ​AtomicCountersEngine (TensorEngine): Storage engine backing the atomic_counters tile.
  • ​InputTokensEngine (TensorEngine): Storage engine backing the input_tokens tile.
  • ​TopkIdsEngine (TensorEngine): Storage engine backing the topk_ids tile.
  • ​SendPtrsEngine (TensorEngine): Storage engine backing the send_ptrs tile.
  • ​RecvPtrsEngine (TensorEngine): Storage engine backing the recv_ptrs tile.
  • ​RecvCountPtrsEngine (TensorEngine): Storage engine backing the recv_count_ptrs tile.
  • ​InputScalesFnType (InputScalesFn): The type of input_scales_fn (inferred).
  • ​n_experts (Int): Total experts across all devices.
  • ​max_token_per_rank (Int): Maximum tokens any device can send.
  • ​n_gpus_per_node (Int): GPUs per physical node.
  • ​n_nodes (Int): Number of physical nodes.
  • ​fused_shared_expert (Bool): Whether to pack shared expert inputs with routed experts' inputs.
  • ​skip_a2a (Bool): Whether to skip the A2A communication. If true, we will only send tokens within the current device.
  • ​use_shmem (Bool): Whether to enable SHMEM communication.
  • ​allreduce_world_size (Int): The world size of the allreduce operation. Only needed for skip_a2a. Used to calculate the workload distribution for the shared expert (if has one).
  • ​has_input_scales (Bool): Whether input_scales_fn supplies the input scale. The overload without input_scales_fn passes False.
  • ​dispatch_warp_min_tokens (Int): From this many tokens a rank on, send one token per warp instead of one per CTA, and count each expert's rows with one shared-memory histogram pass at any count. Same bytes and receive layout. -1 (the default) keeps the stock send and count, as does every configuration but single-node NVFP4 or MXFP8 without a fused shared expert.
  • ​p2p_small_max_tokens (Int): Up to this many tokens on every device, a single-node P2P dispatch with a fused shared expert on AMD GPUs runs the single-launch p2p_small_dispatch_kernel instead, and the steps it declines take the P2P prefill dispatch when ep_p2p_prefill_on holds; both need atomic_counters to end in the small path's words. Must match the combine's; 0 keeps the protocol kernels.
  • ​ExpertLoadT (ExpertLoadSink): Type of expert_load; the default NullExpertLoad compiles the counting away.

Was this page helpful?