For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
p2p_small_combine_kernel
def p2p_small_combine_kernel[input_type: DType, num_threads: Int, input_tokens_layout: TensorLayout, src_info_layout: TensorLayout, top_k: Int, n_ranks: Int, max_tokens_per_rank: Int, StoreFnType: EPCombineStoreFn, RouterWeightsFnType: RouterWeightsFn, has_router_weights: Bool, fused_shared_expert: Bool, small_max_tokens: Int, *, InputTokensEngine: TensorEngine = DefaultEngine, SrcInfoEngine: TensorEngine = DefaultEngine](input_tokens: TileTensor[input_type, input_tokens_layout, ImmUntrackedOrigin, Engine=InputTokensEngine], src_info: TileTensor[.int32, src_info_layout, ImmUntrackedOrigin, Engine=SrcInfoEngine], num_tokens: Int32, recv_buf_ptrs: Array[Pointer[UInt8, MutUntrackedOrigin], n_ranks], small_words: MutPointer[UInt32, MutUntrackedOrigin], my_rank: Int32, store_fn: StoreFnType, router_weights_fn: RouterWeightsFnType)
The combine paired with p2p_small_dispatch_kernel, in one launch.
Does nothing unless that dispatch took the small path. Each routed row
goes back to its source device's combine receive buffer at the
(token, top-k) slot, followed by a flag tagged with the dispatch's
epoch. Then one warp per (token, chunk) of this device's tokens waits
for its token's top_k flags and reduces the slots as combine_kernel
does, adding the shared expert's row when fused.
Parameters:
- input_type (
DType): The type of the expert outputs. - num_threads (
Int): The number of threads in the block. - input_tokens_layout (
TensorLayout): The layout of the expert outputs. - src_info_layout (
TensorLayout): The layout of the row sources. - top_k (
Int): Number of experts each token is routed to. - n_ranks (
Int): The devices of the node. - max_tokens_per_rank (
Int): The combine receive buffer's tokens. - StoreFnType (
EPCombineStoreFn): The type ofstore_fn. - RouterWeightsFnType (
RouterWeightsFn): The type ofrouter_weights_fn. - has_router_weights (
Bool): Whether the routed outputs are weighted and summed; otherwise each slot's output is stored on its own. - fused_shared_expert (
Bool): Whether to add the shared expert's output. - small_max_tokens (
Int): Must match the dispatch's. - InputTokensEngine (
TensorEngine): Storage engine backing theinput_tokenstile. - SrcInfoEngine (
TensorEngine): Storage engine backing thesrc_infotile.
Args:
- input_tokens (
TileTensor[input_type, input_tokens_layout, ImmUntrackedOrigin, Engine=InputTokensEngine]): The expert outputs, one row per dispatched row. - src_info (
TileTensor[.int32, src_info_layout, ImmUntrackedOrigin, Engine=SrcInfoEngine]): The row sources, fromp2p_small_dispatch_kernel. - num_tokens (
Int32): The tokens on this device. - recv_buf_ptrs (
Array[Pointer[UInt8, MutUntrackedOrigin], n_ranks]): Every device's combine receive buffer. - small_words (
MutPointer[UInt32, MutUntrackedOrigin]): This device'sEP_P2P_SMALL_WORDS. - my_rank (
Int32): The rank of this device. - store_fn (
StoreFnType): Stores one output vector, as incombine_kernel. - router_weights_fn (
RouterWeightsFnType): The router weights.