For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
dispatch_kernel
def dispatch_kernel[input_type: DType, num_threads: Int, input_tokens_layout: TensorLayout, topk_ids_layout: TensorLayout, row_offsets_layout: TensorLayout, expert_ids_layout: TensorLayout, src_info_layout: TensorLayout, n_sms: Int, n_experts: Int, n_ranks: Int, max_tokens_per_rank: Int, p2p_world_size: Int, token_fmt_type: TokenFormat, InputScalesFnType: InputScalesFn, fused_shared_expert: Bool = False, has_input_scales: Bool = False, skip_a2a: Bool = False, use_shmem: Bool = True, allreduce_world_size: Int = Int(1), warp_min_tokens: Int = Int(-1), token_recv: Bool = False, p2p_small_counter_words: Int = Int(0), *, InputTokensEngine: TensorEngine = DefaultEngine, TopkIdsEngine: TensorEngine = DefaultEngine, RowOffsetsEngine: TensorEngine = DefaultEngine, ExpertIdsEngine: TensorEngine = DefaultEngine, SrcInfoEngine: TensorEngine = DefaultEngine, ExpertLoadT: ExpertLoadSink = NullExpertLoad](input_tokens: TileTensor[input_type, input_tokens_layout, ImmUntrackedOrigin, Engine=InputTokensEngine], topk_ids: TileTensor[.int32, topk_ids_layout, ImmUntrackedOrigin, Engine=TopkIdsEngine], format_handler: token_fmt_type, row_offsets: TileTensor[.uint32, row_offsets_layout, MutUntrackedOrigin, Engine=RowOffsetsEngine], expert_ids: TileTensor[.int32, expert_ids_layout, MutUntrackedOrigin, Engine=ExpertIdsEngine], src_info: TileTensor[.int32, src_info_layout, MutUntrackedOrigin, Engine=SrcInfoEngine], send_buf_p: MutPointer[UInt8, MutUntrackedOrigin], recv_buf_ptrs: Array[Pointer[UInt8, MutUntrackedOrigin], p2p_world_size], recv_count_ptrs: Array[Pointer[UInt64, MutUntrackedOrigin], p2p_world_size], ep_counters: EPLocalSyncCounters[n_experts], my_rank: Int32, expert_load: ExpertLoadT, input_scales_fn: InputScalesFnType)
Fused dispatch kernel that combines dispatch_async and dispatch_wait functionality in a single kernel launch.
This kernel dispatches tokens to experts on remote ranks based on the top-k expert IDs, then waits for all tokens to arrive and aggregates them for grouped matmul computation.
Parameters:
- input_type (
DType): The type of the input tokens. - num_threads (
Int): The number of threads in the block. - input_tokens_layout (
TensorLayout): The layout of the input tokens. - topk_ids_layout (
TensorLayout): The layout of the top-k expert IDs. - row_offsets_layout (
TensorLayout): The layout of the row offsets. - expert_ids_layout (
TensorLayout): The layout of the expert IDs. - src_info_layout (
TensorLayout): The layout of the source token info. - n_sms (
Int): The total number of SMs in the device. - n_experts (
Int): The total number of experts in the model. - n_ranks (
Int): The number of all devices participating in the communication. - max_tokens_per_rank (
Int): The maximum number of tokens per rank. - p2p_world_size (
Int): Size of a High-speed GPU interconnect group. - token_fmt_type (
TokenFormat): Type conforming to TokenFormat trait that defines the token encoding scheme. - InputScalesFnType (
InputScalesFn): The type ofinput_scales_fn. - fused_shared_expert (
Bool): Whether to pack the shared expert inputs with the routed experts' inputs. When enabled, input_tokens is used as the shared expert inputs. - has_input_scales (
Bool): Whetherinput_scales_fnsupplies the input scale. - skip_a2a (
Bool): Whether to skip the A2A communication. If true, we will only send tokens within the current device. - use_shmem (
Bool): Whether to use the SHMEM API for the communication. - allreduce_world_size (
Int): The world size of the allreduce operation. Only needed for skip_a2a. Used to calculate the workload distribution for the shared expert (if has one). - warp_min_tokens (
Int): From this many tokens a rank on, send one token per warp, and count experts with the histogram monitor at any count (seeEPDispatchKernel). -1 (the default) keeps the stock monitor and send, as does every configuration but single-node NVFP4 without a fused shared expert. - token_recv (
Bool): Whether each destination receives a token once (seeEPDispatchKernel). - p2p_small_counter_words (
Int): When above 0, the counter buffer's elements, which end in the P2P small path's words; the step then returns at once ifp2p_small_dispatch_kerneldispatched it. - InputTokensEngine (
TensorEngine): Storage engine backing theinput_tokenstile. - TopkIdsEngine (
TensorEngine): Storage engine backing thetopk_idstile. - RowOffsetsEngine (
TensorEngine): Storage engine backing therow_offsetstile. - ExpertIdsEngine (
TensorEngine): Storage engine backing theexpert_idstile. - SrcInfoEngine (
TensorEngine): Storage engine backing thesrc_infotile. - ExpertLoadT (
ExpertLoadSink): Type ofexpert_load; the defaultNullExpertLoadcompiles the counting away.
Args:
- input_tokens (
TileTensor[input_type, input_tokens_layout, ImmUntrackedOrigin, Engine=InputTokensEngine]): The input tokens to be dispatched. Also used as shared expert inputs when fused_shared_expert is True. - topk_ids (
TileTensor[.int32, topk_ids_layout, ImmUntrackedOrigin, Engine=TopkIdsEngine]): The top-k expert IDs for each token. - format_handler (
token_fmt_type): Instance of token_fmt_type that performs token decoding and manages output tensor writes. - row_offsets (
TileTensor[.uint32, row_offsets_layout, MutUntrackedOrigin, Engine=RowOffsetsEngine]): The row offsets to be updated. Will be consumed by thegrouped_matmulkernel. - expert_ids (
TileTensor[.int32, expert_ids_layout, MutUntrackedOrigin, Engine=ExpertIdsEngine]): The expert IDs to be updated. Will be consumed by thegrouped_matmulkernel. - src_info (
TileTensor[.int32, src_info_layout, MutUntrackedOrigin, Engine=SrcInfoEngine]): The source token info to be updated. Once the expert computation is complete, tokens will be sent back to the original rank using information in this tensor. - send_buf_p (
MutPointer[UInt8, MutUntrackedOrigin]): The pointer to the send buffer. Need to be allocated usingshmem_allocifuse_shmemis True. - recv_buf_ptrs (
Array[Pointer[UInt8, MutUntrackedOrigin], p2p_world_size]): An array of pointers to the receive buffers for each device in the p2p world. - recv_count_ptrs (
Array[Pointer[UInt64, MutUntrackedOrigin], p2p_world_size]): An array of pointers to the receive count buffers for each device in the p2p world. - ep_counters (
EPLocalSyncCounters[n_experts]): EP atomic counters for kernel synchronization. - my_rank (
Int32): The rank of the current device. - expert_load (
ExpertLoadT): Receives each local expert's routed-token count. - input_scales_fn (
InputScalesFnType): Input scale supplier, passed by host layout.