IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

p2p_prefill_combine_kernel

def p2p_prefill_combine_kernel[input_type: DType, num_threads: Int, input_tokens_layout: TensorLayout, top_k: Int, n_ranks: Int, max_tokens_per_rank: Int, StoreFnType: EPCombineStoreFn, fused_shared_expert: Bool, phase: Int, *, InputTokensEngine: TensorEngine = DefaultEngine](input_tokens: TileTensor[input_type, input_tokens_layout, ImmUntrackedOrigin, Engine=InputTokensEngine], num_tokens: Int32, recv_buf_ptrs: Array[Pointer[UInt8, MutUntrackedOrigin], n_ranks], small_words: MutPointer[UInt32, MutUntrackedOrigin], my_rank: Int32, store_fn: StoreFnType)

Second half of the partial-row combine of a prefill-dispatched step.

Does nothing on a step the P2P small path took. Each device sums, per (source, token), the weighted outputs of its own experts for that token in fp32 and sends the sum to the source as one BF16 row, instead of one row per top-k slot as the protocol combine does: a token's 8 experts sit on at most 4 devices. A flag per (sender, token) says whether a partial came. Then one warp per (token, chunk) of this device's tokens sums the partials in sender order and adds the shared expert's row when fused.

Parameters:

  • ​input_type (DType): The type of the expert outputs.
  • ​num_threads (Int): The number of threads in the block.
  • ​input_tokens_layout (TensorLayout): The layout of the expert outputs.
  • ​top_k (Int): Number of experts each token is routed to.
  • ​n_ranks (Int): The devices of the node.
  • ​max_tokens_per_rank (Int): The receive buffers' tokens per source device.
  • ​StoreFnType (EPCombineStoreFn): The type of store_fn.
  • ​fused_shared_expert (Bool): Whether to add the shared expert's output.
  • ​phase (Int): 0 sends this device's partials, 1 reduces its tokens. They are two launches so a device does not poll while it sends.
  • ​InputTokensEngine (TensorEngine): Storage engine backing the input_tokens tile.

Args:

Was this page helpful?