IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

dispatch_kernel

def dispatch_kernel[input_type: DType, num_threads: Int, input_tokens_layout: TensorLayout, topk_ids_layout: TensorLayout, row_offsets_layout: TensorLayout, expert_ids_layout: TensorLayout, src_info_layout: TensorLayout, n_sms: Int, n_experts: Int, n_ranks: Int, max_tokens_per_rank: Int, p2p_world_size: Int, token_fmt_type: TokenFormat, InputScalesFnType: InputScalesFn, fused_shared_expert: Bool = False, has_input_scales: Bool = False, skip_a2a: Bool = False, use_shmem: Bool = True, allreduce_world_size: Int = Int(1), warp_min_tokens: Int = Int(-1), token_recv: Bool = False, p2p_small_counter_words: Int = Int(0), *, InputTokensEngine: TensorEngine = DefaultEngine, TopkIdsEngine: TensorEngine = DefaultEngine, RowOffsetsEngine: TensorEngine = DefaultEngine, ExpertIdsEngine: TensorEngine = DefaultEngine, SrcInfoEngine: TensorEngine = DefaultEngine, ExpertLoadT: ExpertLoadSink = NullExpertLoad](input_tokens: TileTensor[input_type, input_tokens_layout, ImmUntrackedOrigin, Engine=InputTokensEngine], topk_ids: TileTensor[.int32, topk_ids_layout, ImmUntrackedOrigin, Engine=TopkIdsEngine], format_handler: token_fmt_type, row_offsets: TileTensor[.uint32, row_offsets_layout, MutUntrackedOrigin, Engine=RowOffsetsEngine], expert_ids: TileTensor[.int32, expert_ids_layout, MutUntrackedOrigin, Engine=ExpertIdsEngine], src_info: TileTensor[.int32, src_info_layout, MutUntrackedOrigin, Engine=SrcInfoEngine], send_buf_p: MutPointer[UInt8, MutUntrackedOrigin], recv_buf_ptrs: Array[Pointer[UInt8, MutUntrackedOrigin], p2p_world_size], recv_count_ptrs: Array[Pointer[UInt64, MutUntrackedOrigin], p2p_world_size], ep_counters: EPLocalSyncCounters[n_experts], my_rank: Int32, expert_load: ExpertLoadT, input_scales_fn: InputScalesFnType)

Fused dispatch kernel that combines dispatch_async and dispatch_wait functionality in a single kernel launch.

This kernel dispatches tokens to experts on remote ranks based on the top-k expert IDs, then waits for all tokens to arrive and aggregates them for grouped matmul computation.

Parameters:

  • ​input_type (DType): The type of the input tokens.
  • ​num_threads (Int): The number of threads in the block.
  • ​input_tokens_layout (TensorLayout): The layout of the input tokens.
  • ​topk_ids_layout (TensorLayout): The layout of the top-k expert IDs.
  • ​row_offsets_layout (TensorLayout): The layout of the row offsets.
  • ​expert_ids_layout (TensorLayout): The layout of the expert IDs.
  • ​src_info_layout (TensorLayout): The layout of the source token info.
  • ​n_sms (Int): The total number of SMs in the device.
  • ​n_experts (Int): The total number of experts in the model.
  • ​n_ranks (Int): The number of all devices participating in the communication.
  • ​max_tokens_per_rank (Int): The maximum number of tokens per rank.
  • ​p2p_world_size (Int): Size of a High-speed GPU interconnect group.
  • ​token_fmt_type (TokenFormat): Type conforming to TokenFormat trait that defines the token encoding scheme.
  • ​InputScalesFnType (InputScalesFn): The type of input_scales_fn.
  • ​fused_shared_expert (Bool): Whether to pack the shared expert inputs with the routed experts' inputs. When enabled, input_tokens is used as the shared expert inputs.
  • ​has_input_scales (Bool): Whether input_scales_fn supplies the input scale.
  • ​skip_a2a (Bool): Whether to skip the A2A communication. If true, we will only send tokens within the current device.
  • ​use_shmem (Bool): Whether to use the SHMEM API for the communication.
  • ​allreduce_world_size (Int): The world size of the allreduce operation. Only needed for skip_a2a. Used to calculate the workload distribution for the shared expert (if has one).
  • ​warp_min_tokens (Int): From this many tokens a rank on, send one token per warp, and count experts with the histogram monitor at any count (see EPDispatchKernel). -1 (the default) keeps the stock monitor and send, as does every configuration but single-node NVFP4 without a fused shared expert.
  • ​token_recv (Bool): Whether each destination receives a token once (see EPDispatchKernel).
  • ​p2p_small_counter_words (Int): When above 0, the counter buffer's elements, which end in the P2P small path's words; the step then returns at once if p2p_small_dispatch_kernel dispatched it.
  • ​InputTokensEngine (TensorEngine): Storage engine backing the input_tokens tile.
  • ​TopkIdsEngine (TensorEngine): Storage engine backing the topk_ids tile.
  • ​RowOffsetsEngine (TensorEngine): Storage engine backing the row_offsets tile.
  • ​ExpertIdsEngine (TensorEngine): Storage engine backing the expert_ids tile.
  • ​SrcInfoEngine (TensorEngine): Storage engine backing the src_info tile.
  • ​ExpertLoadT (ExpertLoadSink): Type of expert_load; the default NullExpertLoad compiles the counting away.

Args:

Was this page helpful?