For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
p2p_prefill_combine_kernel
def p2p_prefill_combine_kernel[input_type: DType, num_threads: Int, input_tokens_layout: TensorLayout, top_k: Int, n_ranks: Int, max_tokens_per_rank: Int, StoreFnType: EPCombineStoreFn, fused_shared_expert: Bool, phase: Int, *, InputTokensEngine: TensorEngine = DefaultEngine](input_tokens: TileTensor[input_type, input_tokens_layout, ImmUntrackedOrigin, Engine=InputTokensEngine], num_tokens: Int32, recv_buf_ptrs: Array[Pointer[UInt8, MutUntrackedOrigin], n_ranks], small_words: MutPointer[UInt32, MutUntrackedOrigin], my_rank: Int32, store_fn: StoreFnType)
Second half of the partial-row combine of a prefill-dispatched step.
Does nothing on a step the P2P small path took. Each device sums, per
(source, token), the weighted outputs of its own experts for that
token in fp32 and sends the sum to the source as one BF16 row, instead
of one row per top-k slot as the protocol combine does: a token's 8
experts sit on at most 4 devices. A flag per (sender, token) says
whether a partial came. Then one warp per (token, chunk) of this
device's tokens sums the partials in sender order and adds the shared
expert's row when fused.
Parameters:
- input_type (
DType): The type of the expert outputs. - num_threads (
Int): The number of threads in the block. - input_tokens_layout (
TensorLayout): The layout of the expert outputs. - top_k (
Int): Number of experts each token is routed to. - n_ranks (
Int): The devices of the node. - max_tokens_per_rank (
Int): The receive buffers' tokens per source device. - StoreFnType (
EPCombineStoreFn): The type ofstore_fn. - fused_shared_expert (
Bool): Whether to add the shared expert's output. - phase (
Int): 0 sends this device's partials, 1 reduces its tokens. They are two launches so a device does not poll while it sends. - InputTokensEngine (
TensorEngine): Storage engine backing theinput_tokenstile.
Args:
- input_tokens (
TileTensor[input_type, input_tokens_layout, ImmUntrackedOrigin, Engine=InputTokensEngine]): The expert outputs, one row per dispatched row. - num_tokens (
Int32): The tokens on this device. - recv_buf_ptrs (
Array[Pointer[UInt8, MutUntrackedOrigin], n_ranks]): Every device's combine receive buffer. - small_words (
MutPointer[UInt32, MutUntrackedOrigin]): This device'sEP_P2P_SMALL_WORDS. - my_rank (
Int32): The rank of this device. - store_fn (
StoreFnType): Stores one output vector, as incombine_kernel.