IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

p2p_prefill_dispatch_send_kernel

def p2p_prefill_dispatch_send_kernel[input_type: DType, input_tokens_layout: TensorLayout, topk_ids_layout: TensorLayout, n_experts: Int, n_ranks: Int, max_tokens_per_rank: Int, token_fmt_type: TokenFormat, InputScalesFnType: InputScalesFn, fused_shared_expert: Bool, has_input_scales: Bool, *, InputTokensEngine: TensorEngine = DefaultEngine, TopkIdsEngine: TensorEngine = DefaultEngine](input_tokens: TileTensor[input_type, input_tokens_layout, ImmUntrackedOrigin, Engine=InputTokensEngine], topk_ids: TileTensor[.int32, topk_ids_layout, ImmUntrackedOrigin, Engine=TopkIdsEngine], format_handler: token_fmt_type, recv_buf_ptrs: Array[Pointer[UInt8, MutUntrackedOrigin], n_ranks], small_words: MutPointer[UInt32, MutUntrackedOrigin], my_rank: Int32, input_scales_fn: InputScalesFnType)

First of the three launches of the P2P prefill dispatch.

Does nothing on a step the P2P small path took. Each one-warp CTA quantizes its tokens once each and writes each message, with its top-k ids, to the protocol's message slot on every device that owns one of the token's experts (the shared expert's row, this device's token, goes straight to the output). Once its stores land it flags each token at every device, with a bit for whether a message came. It also clears this device's histogram for p2p_prefill_dispatch_count_kernel.

The protocol dispatch reserves its rows and signals its peers through per-expert counters and handshakes; this one needs a flag per token and two local launches to place the rows.

Parameters:

  • ​input_type (DType): The type of the input tokens.
  • ​input_tokens_layout (TensorLayout): The layout of the input tokens.
  • ​topk_ids_layout (TensorLayout): The layout of the top-k expert IDs.
  • ​n_experts (Int): The number of experts over all devices.
  • ​n_ranks (Int): The devices of the node.
  • ​max_tokens_per_rank (Int): The receive buffer's tokens per source device.
  • ​token_fmt_type (TokenFormat): The token format.
  • ​InputScalesFnType (InputScalesFn): The type of input_scales_fn.
  • ​fused_shared_expert (Bool): Whether the shared expert's rows come first.
  • ​has_input_scales (Bool): Whether input_scales_fn supplies the input scale.
  • ​InputTokensEngine (TensorEngine): Storage engine backing the input_tokens tile.
  • ​TopkIdsEngine (TensorEngine): Storage engine backing the topk_ids tile.

Args:

Was this page helpful?