IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

ep_dispatch_async_kernel_api

def ep_dispatch_async_kernel_api[AtomicCountersEngine: TensorEngine, InputTokensEngine: TensorEngine, TopkIdsEngine: TensorEngine, SendPtrsEngine: TensorEngine, RecvPtrsEngine: TensorEngine, RecvCountPtrsEngine: TensorEngine, //, token_fmt_type: TokenFormat, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, use_shmem: Bool = (n_nodes > Int(1))](atomic_counters: TileTensor[.int32, Engine=AtomicCountersEngine, address_space=atomic_counters.address_space, linear_idx_type=atomic_counters.linear_idx_type], input_tokens: TileTensor[Engine=InputTokensEngine, address_space=input_tokens.address_space, linear_idx_type=input_tokens.linear_idx_type], topk_ids: TileTensor[.int32, Engine=TopkIdsEngine, address_space=topk_ids.address_space, linear_idx_type=topk_ids.linear_idx_type], send_ptrs: TileTensor[.uint64, Engine=SendPtrsEngine, address_space=send_ptrs.address_space, linear_idx_type=send_ptrs.linear_idx_type], recv_ptrs: TileTensor[.uint64, Engine=RecvPtrsEngine, address_space=recv_ptrs.address_space, linear_idx_type=recv_ptrs.linear_idx_type], recv_count_ptrs: TileTensor[.uint64, Engine=RecvCountPtrsEngine, address_space=recv_count_ptrs.address_space, linear_idx_type=recv_count_ptrs.linear_idx_type], context: DeviceContext)

Execute the Expert Parallelism async dispatch kernel at unit input scale.

See the overload taking input_scales_fn.

Arguments: atomic_counters: EP kernel synchronization counters. input_tokens: Tokens to dispatch to experts. topk_ids: Expert assignments from router. send_ptrs: Send buffer pointers for each local GPU. recv_ptrs: Receive buffer pointers for each local GPU. recv_count_ptrs: Receive count buffer pointers for each local GPU. context: Device context pointer

Parameters:

  • ​AtomicCountersEngine (TensorEngine): Storage engine backing the atomic_counters tile.
  • ​InputTokensEngine (TensorEngine): Storage engine backing the input_tokens tile.
  • ​TopkIdsEngine (TensorEngine): Storage engine backing the topk_ids tile.
  • ​SendPtrsEngine (TensorEngine): Storage engine backing the send_ptrs tile.
  • ​RecvPtrsEngine (TensorEngine): Storage engine backing the recv_ptrs tile.
  • ​RecvCountPtrsEngine (TensorEngine): Storage engine backing the recv_count_ptrs tile.
  • ​token_fmt_type (TokenFormat): Token format type.
  • ​n_experts (Int): Total experts across all devices.
  • ​max_token_per_rank (Int): Maximum tokens any device can send.
  • ​n_gpus_per_node (Int): GPUs per physical node.
  • ​n_nodes (Int): Number of physical nodes.
  • ​use_shmem (Bool): Whether to enable SHMEM communication.

def ep_dispatch_async_kernel_api[AtomicCountersEngine: TensorEngine, InputTokensEngine: TensorEngine, TopkIdsEngine: TensorEngine, SendPtrsEngine: TensorEngine, RecvPtrsEngine: TensorEngine, RecvCountPtrsEngine: TensorEngine, InputScalesFnType: InputScalesFn, //, token_fmt_type: TokenFormat, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, use_shmem: Bool = (n_nodes > Int(1)), has_input_scales: Bool = True](atomic_counters: TileTensor[.int32, Engine=AtomicCountersEngine, address_space=atomic_counters.address_space, linear_idx_type=atomic_counters.linear_idx_type], input_tokens: TileTensor[Engine=InputTokensEngine, address_space=input_tokens.address_space, linear_idx_type=input_tokens.linear_idx_type], topk_ids: TileTensor[.int32, Engine=TopkIdsEngine, address_space=topk_ids.address_space, linear_idx_type=topk_ids.linear_idx_type], send_ptrs: TileTensor[.uint64, Engine=SendPtrsEngine, address_space=send_ptrs.address_space, linear_idx_type=send_ptrs.linear_idx_type], recv_ptrs: TileTensor[.uint64, Engine=RecvPtrsEngine, address_space=recv_ptrs.address_space, linear_idx_type=recv_ptrs.linear_idx_type], recv_count_ptrs: TileTensor[.uint64, Engine=RecvCountPtrsEngine, address_space=recv_count_ptrs.address_space, linear_idx_type=recv_count_ptrs.linear_idx_type], context: DeviceContext, input_scales_fn: InputScalesFnType)

Execute the Expert Parallelism async dispatch kernel.

This function launches the dispatch_async_kernel from ep_comm.mojo to initiate token distribution across expert devices. In multi-node scenarios, all the communication buffers need to be allocated using shmem_malloc.

Arguments: atomic_counters: EP kernel synchronization counters. input_tokens: Tokens to dispatch to experts. topk_ids: Expert assignments from router. send_ptrs: Send buffer pointers for each local GPU. recv_ptrs: Receive buffer pointers for each local GPU. recv_count_ptrs: Receive count buffer pointers for each local GPU. context: Device context pointer input_scales_fn: The input scales. It runs on the device, so it must copy what it captures.

Parameters:

  • ​AtomicCountersEngine (TensorEngine): Storage engine backing the atomic_counters tile.
  • ​InputTokensEngine (TensorEngine): Storage engine backing the input_tokens tile.
  • ​TopkIdsEngine (TensorEngine): Storage engine backing the topk_ids tile.
  • ​SendPtrsEngine (TensorEngine): Storage engine backing the send_ptrs tile.
  • ​RecvPtrsEngine (TensorEngine): Storage engine backing the recv_ptrs tile.
  • ​RecvCountPtrsEngine (TensorEngine): Storage engine backing the recv_count_ptrs tile.
  • ​InputScalesFnType (InputScalesFn): The type of input_scales_fn (inferred).
  • ​token_fmt_type (TokenFormat): Token format type.
  • ​n_experts (Int): Total experts across all devices.
  • ​max_token_per_rank (Int): Maximum tokens any device can send.
  • ​n_gpus_per_node (Int): GPUs per physical node.
  • ​n_nodes (Int): Number of physical nodes.
  • ​use_shmem (Bool): Whether to enable SHMEM communication.
  • ​has_input_scales (Bool): Whether input_scales_fn supplies the input scale. The overload without input_scales_fn passes False.

Was this page helpful?