For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
ep_fused_dispatch_kernel_api
def ep_fused_dispatch_kernel_api[token_fmt_type: TokenFormat, dispatch_dtype: DType, RowOffsetsEngine: TensorEngine, ExpertIdsEngine: TensorEngine, SrcInfoEngine: TensorEngine, AtomicCountersEngine: TensorEngine, InputTokensEngine: TensorEngine, TopkIdsEngine: TensorEngine, SendPtrsEngine: TensorEngine, RecvPtrsEngine: TensorEngine, RecvCountPtrsEngine: TensorEngine, //, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, fused_shared_expert: Bool, skip_a2a: Bool = False, use_shmem: Bool = (n_nodes > Int(1)), allreduce_world_size: Int = Int(1), dispatch_warp_min_tokens: Int = Int(-1), p2p_small_max_tokens: Int = Int(32), ExpertLoadT: ExpertLoadSink = NullExpertLoad](token_handler: token_fmt_type, row_offsets: TileTensor[.uint32, Engine=RowOffsetsEngine, address_space=row_offsets.address_space, linear_idx_type=row_offsets.linear_idx_type], expert_ids: TileTensor[.int32, Engine=ExpertIdsEngine, address_space=expert_ids.address_space, linear_idx_type=expert_ids.linear_idx_type], src_info: TileTensor[.int32, Engine=SrcInfoEngine, address_space=src_info.address_space, linear_idx_type=src_info.linear_idx_type], atomic_counters: TileTensor[.int32, Engine=AtomicCountersEngine, address_space=atomic_counters.address_space, linear_idx_type=atomic_counters.linear_idx_type], input_tokens: TileTensor[dispatch_dtype, Engine=InputTokensEngine, address_space=input_tokens.address_space, linear_idx_type=input_tokens.linear_idx_type], topk_ids: TileTensor[.int32, Engine=TopkIdsEngine, address_space=topk_ids.address_space, linear_idx_type=topk_ids.linear_idx_type], send_ptrs: TileTensor[.uint64, Engine=SendPtrsEngine, address_space=send_ptrs.address_space, linear_idx_type=send_ptrs.linear_idx_type], recv_ptrs: TileTensor[.uint64, Engine=RecvPtrsEngine, address_space=recv_ptrs.address_space, linear_idx_type=recv_ptrs.linear_idx_type], recv_count_ptrs: TileTensor[.uint64, Engine=RecvCountPtrsEngine, address_space=recv_count_ptrs.address_space, linear_idx_type=recv_count_ptrs.linear_idx_type], context: DeviceContext, expert_load: ExpertLoadT = NullExpertLoad())
Execute the fused Expert Parallelism dispatch kernel at unit input scale.
See the overload taking input_scales_fn.
Arguments: token_handler: Token handler. Wrapper for the output token tensor. row_offsets: Row offsets for grouped matmul. expert_ids: Expert IDs for grouped matmul. src_info: Source routing information for combine phase. atomic_counters: EP kernel synchronization counters. input_tokens: Tokens to dispatch to experts. topk_ids: Expert assignments from router. send_ptrs: Send buffer pointers for each local GPU. recv_ptrs: Receive buffer pointers for each local GPU. recv_count_ptrs: Receive count buffer pointers for each local GPU. context: Device context pointer. expert_load: Receives each local expert's routed-token count, for EPLB load statistics.
Parameters:
- token_fmt_type (
TokenFormat): Token format type. - dispatch_dtype (
DType): Data type of the dispatched tokens. - RowOffsetsEngine (
TensorEngine): Storage engine backing therow_offsetstile. - ExpertIdsEngine (
TensorEngine): Storage engine backing theexpert_idstile. - SrcInfoEngine (
TensorEngine): Storage engine backing thesrc_infotile. - AtomicCountersEngine (
TensorEngine): Storage engine backing theatomic_counterstile. - InputTokensEngine (
TensorEngine): Storage engine backing theinput_tokenstile. - TopkIdsEngine (
TensorEngine): Storage engine backing thetopk_idstile. - SendPtrsEngine (
TensorEngine): Storage engine backing thesend_ptrstile. - RecvPtrsEngine (
TensorEngine): Storage engine backing therecv_ptrstile. - RecvCountPtrsEngine (
TensorEngine): Storage engine backing therecv_count_ptrstile. - n_experts (
Int): Total experts across all devices. - max_token_per_rank (
Int): Maximum tokens any device can send. - n_gpus_per_node (
Int): GPUs per physical node. - n_nodes (
Int): Number of physical nodes. - fused_shared_expert (
Bool): Whether to pack shared expert inputs with routed experts' inputs. - skip_a2a (
Bool): Whether to skip the A2A communication. If true, we will only send tokens within the current device. - use_shmem (
Bool): Whether to enable SHMEM communication. - allreduce_world_size (
Int): The world size of the allreduce operation. Only needed for skip_a2a. Used to calculate the workload distribution for the shared expert (if has one). - dispatch_warp_min_tokens (
Int): From this many tokens a rank on, send one token per warp instead of one per CTA, and count each expert's rows with one shared-memory histogram pass at any count. Same bytes and receive layout. -1 (the default) keeps the stock send and count, as does every configuration but single-node NVFP4 or MXFP8 without a fused shared expert. - p2p_small_max_tokens (
Int): Up to this many tokens on every device, a single-node P2P dispatch with a fused shared expert on AMD GPUs runs the single-launchp2p_small_dispatch_kernelinstead, and the steps it declines take the P2P prefill dispatch whenep_p2p_prefill_onholds; both needatomic_countersto end in the small path's words. Must match the combine's; 0 keeps the protocol kernels. - ExpertLoadT (
ExpertLoadSink): Type ofexpert_load; the defaultNullExpertLoadcompiles the counting away.
def ep_fused_dispatch_kernel_api[token_fmt_type: TokenFormat, dispatch_dtype: DType, RowOffsetsEngine: TensorEngine, ExpertIdsEngine: TensorEngine, SrcInfoEngine: TensorEngine, AtomicCountersEngine: TensorEngine, InputTokensEngine: TensorEngine, TopkIdsEngine: TensorEngine, SendPtrsEngine: TensorEngine, RecvPtrsEngine: TensorEngine, RecvCountPtrsEngine: TensorEngine, InputScalesFnType: InputScalesFn, //, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, fused_shared_expert: Bool, skip_a2a: Bool = False, use_shmem: Bool = (n_nodes > Int(1)), allreduce_world_size: Int = Int(1), has_input_scales: Bool = True, dispatch_warp_min_tokens: Int = Int(-1), p2p_small_max_tokens: Int = Int(32), ExpertLoadT: ExpertLoadSink = NullExpertLoad](token_handler: token_fmt_type, row_offsets: TileTensor[.uint32, Engine=RowOffsetsEngine, address_space=row_offsets.address_space, linear_idx_type=row_offsets.linear_idx_type], expert_ids: TileTensor[.int32, Engine=ExpertIdsEngine, address_space=expert_ids.address_space, linear_idx_type=expert_ids.linear_idx_type], src_info: TileTensor[.int32, Engine=SrcInfoEngine, address_space=src_info.address_space, linear_idx_type=src_info.linear_idx_type], atomic_counters: TileTensor[.int32, Engine=AtomicCountersEngine, address_space=atomic_counters.address_space, linear_idx_type=atomic_counters.linear_idx_type], input_tokens: TileTensor[dispatch_dtype, Engine=InputTokensEngine, address_space=input_tokens.address_space, linear_idx_type=input_tokens.linear_idx_type], topk_ids: TileTensor[.int32, Engine=TopkIdsEngine, address_space=topk_ids.address_space, linear_idx_type=topk_ids.linear_idx_type], send_ptrs: TileTensor[.uint64, Engine=SendPtrsEngine, address_space=send_ptrs.address_space, linear_idx_type=send_ptrs.linear_idx_type], recv_ptrs: TileTensor[.uint64, Engine=RecvPtrsEngine, address_space=recv_ptrs.address_space, linear_idx_type=recv_ptrs.linear_idx_type], recv_count_ptrs: TileTensor[.uint64, Engine=RecvCountPtrsEngine, address_space=recv_count_ptrs.address_space, linear_idx_type=recv_count_ptrs.linear_idx_type], context: DeviceContext, input_scales_fn: InputScalesFnType, expert_load: ExpertLoadT = NullExpertLoad())
Execute the fused Expert Parallelism dispatch kernel.
This function launches the fused dispatch_kernel from ep_comm.mojo that combines both dispatch_async and dispatch_wait functionality in a single kernel launch. It distributes input tokens to expert devices based on top-k routing decisions, then waits for all tokens to arrive and aggregates them for grouped matmul computation.
Arguments: token_handler: Token handler. Wrapper for the output token tensor. row_offsets: Row offsets for grouped matmul. expert_ids: Expert IDs for grouped matmul. src_info: Source routing information for combine phase. atomic_counters: EP kernel synchronization counters. input_tokens: Tokens to dispatch to experts. topk_ids: Expert assignments from router. send_ptrs: Send buffer pointers for each local GPU. recv_ptrs: Receive buffer pointers for each local GPU. recv_count_ptrs: Receive count buffer pointers for each local GPU. context: Device context pointer. input_scales_fn: The input scales. It runs on the device, so it must copy what it captures. expert_load: Receives each local expert's routed-token count, for EPLB load statistics.
Parameters:
- token_fmt_type (
TokenFormat): Token format type. - dispatch_dtype (
DType): Data type of the dispatched tokens. - RowOffsetsEngine (
TensorEngine): Storage engine backing therow_offsetstile. - ExpertIdsEngine (
TensorEngine): Storage engine backing theexpert_idstile. - SrcInfoEngine (
TensorEngine): Storage engine backing thesrc_infotile. - AtomicCountersEngine (
TensorEngine): Storage engine backing theatomic_counterstile. - InputTokensEngine (
TensorEngine): Storage engine backing theinput_tokenstile. - TopkIdsEngine (
TensorEngine): Storage engine backing thetopk_idstile. - SendPtrsEngine (
TensorEngine): Storage engine backing thesend_ptrstile. - RecvPtrsEngine (
TensorEngine): Storage engine backing therecv_ptrstile. - RecvCountPtrsEngine (
TensorEngine): Storage engine backing therecv_count_ptrstile. - InputScalesFnType (
InputScalesFn): The type ofinput_scales_fn(inferred). - n_experts (
Int): Total experts across all devices. - max_token_per_rank (
Int): Maximum tokens any device can send. - n_gpus_per_node (
Int): GPUs per physical node. - n_nodes (
Int): Number of physical nodes. - fused_shared_expert (
Bool): Whether to pack shared expert inputs with routed experts' inputs. - skip_a2a (
Bool): Whether to skip the A2A communication. If true, we will only send tokens within the current device. - use_shmem (
Bool): Whether to enable SHMEM communication. - allreduce_world_size (
Int): The world size of the allreduce operation. Only needed for skip_a2a. Used to calculate the workload distribution for the shared expert (if has one). - has_input_scales (
Bool): Whetherinput_scales_fnsupplies the input scale. The overload withoutinput_scales_fnpassesFalse. - dispatch_warp_min_tokens (
Int): From this many tokens a rank on, send one token per warp instead of one per CTA, and count each expert's rows with one shared-memory histogram pass at any count. Same bytes and receive layout. -1 (the default) keeps the stock send and count, as does every configuration but single-node NVFP4 or MXFP8 without a fused shared expert. - p2p_small_max_tokens (
Int): Up to this many tokens on every device, a single-node P2P dispatch with a fused shared expert on AMD GPUs runs the single-launchp2p_small_dispatch_kernelinstead, and the steps it declines take the P2P prefill dispatch whenep_p2p_prefill_onholds; both needatomic_countersto end in the small path's words. Must match the combine's; 0 keeps the protocol kernels. - ExpertLoadT (
ExpertLoadSink): Type ofexpert_load; the defaultNullExpertLoadcompiles the counting away.