For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
ep_dispatch_async_kernel_api
def ep_dispatch_async_kernel_api[AtomicCountersEngine: TensorEngine, InputTokensEngine: TensorEngine, TopkIdsEngine: TensorEngine, SendPtrsEngine: TensorEngine, RecvPtrsEngine: TensorEngine, RecvCountPtrsEngine: TensorEngine, //, token_fmt_type: TokenFormat, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, use_shmem: Bool = (n_nodes > Int(1))](atomic_counters: TileTensor[.int32, Engine=AtomicCountersEngine, address_space=atomic_counters.address_space, linear_idx_type=atomic_counters.linear_idx_type], input_tokens: TileTensor[Engine=InputTokensEngine, address_space=input_tokens.address_space, linear_idx_type=input_tokens.linear_idx_type], topk_ids: TileTensor[.int32, Engine=TopkIdsEngine, address_space=topk_ids.address_space, linear_idx_type=topk_ids.linear_idx_type], send_ptrs: TileTensor[.uint64, Engine=SendPtrsEngine, address_space=send_ptrs.address_space, linear_idx_type=send_ptrs.linear_idx_type], recv_ptrs: TileTensor[.uint64, Engine=RecvPtrsEngine, address_space=recv_ptrs.address_space, linear_idx_type=recv_ptrs.linear_idx_type], recv_count_ptrs: TileTensor[.uint64, Engine=RecvCountPtrsEngine, address_space=recv_count_ptrs.address_space, linear_idx_type=recv_count_ptrs.linear_idx_type], context: DeviceContext)
Execute the Expert Parallelism async dispatch kernel at unit input scale.
See the overload taking input_scales_fn.
Arguments: atomic_counters: EP kernel synchronization counters. input_tokens: Tokens to dispatch to experts. topk_ids: Expert assignments from router. send_ptrs: Send buffer pointers for each local GPU. recv_ptrs: Receive buffer pointers for each local GPU. recv_count_ptrs: Receive count buffer pointers for each local GPU. context: Device context pointer
Parameters:
- AtomicCountersEngine (
TensorEngine): Storage engine backing theatomic_counterstile. - InputTokensEngine (
TensorEngine): Storage engine backing theinput_tokenstile. - TopkIdsEngine (
TensorEngine): Storage engine backing thetopk_idstile. - SendPtrsEngine (
TensorEngine): Storage engine backing thesend_ptrstile. - RecvPtrsEngine (
TensorEngine): Storage engine backing therecv_ptrstile. - RecvCountPtrsEngine (
TensorEngine): Storage engine backing therecv_count_ptrstile. - token_fmt_type (
TokenFormat): Token format type. - n_experts (
Int): Total experts across all devices. - max_token_per_rank (
Int): Maximum tokens any device can send. - n_gpus_per_node (
Int): GPUs per physical node. - n_nodes (
Int): Number of physical nodes. - use_shmem (
Bool): Whether to enable SHMEM communication.
def ep_dispatch_async_kernel_api[AtomicCountersEngine: TensorEngine, InputTokensEngine: TensorEngine, TopkIdsEngine: TensorEngine, SendPtrsEngine: TensorEngine, RecvPtrsEngine: TensorEngine, RecvCountPtrsEngine: TensorEngine, InputScalesFnType: InputScalesFn, //, token_fmt_type: TokenFormat, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, use_shmem: Bool = (n_nodes > Int(1)), has_input_scales: Bool = True](atomic_counters: TileTensor[.int32, Engine=AtomicCountersEngine, address_space=atomic_counters.address_space, linear_idx_type=atomic_counters.linear_idx_type], input_tokens: TileTensor[Engine=InputTokensEngine, address_space=input_tokens.address_space, linear_idx_type=input_tokens.linear_idx_type], topk_ids: TileTensor[.int32, Engine=TopkIdsEngine, address_space=topk_ids.address_space, linear_idx_type=topk_ids.linear_idx_type], send_ptrs: TileTensor[.uint64, Engine=SendPtrsEngine, address_space=send_ptrs.address_space, linear_idx_type=send_ptrs.linear_idx_type], recv_ptrs: TileTensor[.uint64, Engine=RecvPtrsEngine, address_space=recv_ptrs.address_space, linear_idx_type=recv_ptrs.linear_idx_type], recv_count_ptrs: TileTensor[.uint64, Engine=RecvCountPtrsEngine, address_space=recv_count_ptrs.address_space, linear_idx_type=recv_count_ptrs.linear_idx_type], context: DeviceContext, input_scales_fn: InputScalesFnType)
Execute the Expert Parallelism async dispatch kernel.
This function launches the dispatch_async_kernel from ep_comm.mojo to
initiate token distribution across expert devices. In multi-node
scenarios, all the communication buffers need to be allocated using
shmem_malloc.
Arguments: atomic_counters: EP kernel synchronization counters. input_tokens: Tokens to dispatch to experts. topk_ids: Expert assignments from router. send_ptrs: Send buffer pointers for each local GPU. recv_ptrs: Receive buffer pointers for each local GPU. recv_count_ptrs: Receive count buffer pointers for each local GPU. context: Device context pointer input_scales_fn: The input scales. It runs on the device, so it must copy what it captures.
Parameters:
- AtomicCountersEngine (
TensorEngine): Storage engine backing theatomic_counterstile. - InputTokensEngine (
TensorEngine): Storage engine backing theinput_tokenstile. - TopkIdsEngine (
TensorEngine): Storage engine backing thetopk_idstile. - SendPtrsEngine (
TensorEngine): Storage engine backing thesend_ptrstile. - RecvPtrsEngine (
TensorEngine): Storage engine backing therecv_ptrstile. - RecvCountPtrsEngine (
TensorEngine): Storage engine backing therecv_count_ptrstile. - InputScalesFnType (
InputScalesFn): The type ofinput_scales_fn(inferred). - token_fmt_type (
TokenFormat): Token format type. - n_experts (
Int): Total experts across all devices. - max_token_per_rank (
Int): Maximum tokens any device can send. - n_gpus_per_node (
Int): GPUs per physical node. - n_nodes (
Int): Number of physical nodes. - use_shmem (
Bool): Whether to enable SHMEM communication. - has_input_scales (
Bool): Whetherinput_scales_fnsupplies the input scale. The overload withoutinput_scales_fnpassesFalse.