For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
p2p_prefill_dispatch_send_kernel
def p2p_prefill_dispatch_send_kernel[input_type: DType, input_tokens_layout: TensorLayout, topk_ids_layout: TensorLayout, n_experts: Int, n_ranks: Int, max_tokens_per_rank: Int, token_fmt_type: TokenFormat, InputScalesFnType: InputScalesFn, fused_shared_expert: Bool, has_input_scales: Bool, *, InputTokensEngine: TensorEngine = DefaultEngine, TopkIdsEngine: TensorEngine = DefaultEngine](input_tokens: TileTensor[input_type, input_tokens_layout, ImmUntrackedOrigin, Engine=InputTokensEngine], topk_ids: TileTensor[.int32, topk_ids_layout, ImmUntrackedOrigin, Engine=TopkIdsEngine], format_handler: token_fmt_type, recv_buf_ptrs: Array[Pointer[UInt8, MutUntrackedOrigin], n_ranks], small_words: MutPointer[UInt32, MutUntrackedOrigin], my_rank: Int32, input_scales_fn: InputScalesFnType)
First of the three launches of the P2P prefill dispatch.
Does nothing on a step the P2P small path took. Each one-warp CTA
quantizes its tokens once each and writes each message, with its top-k
ids, to the protocol's message slot on every device that owns one of the
token's experts (the shared expert's row, this device's token, goes
straight to the output). Once its stores land it flags each token at
every device, with a bit for whether a message came. It also clears this
device's histogram for p2p_prefill_dispatch_count_kernel.
The protocol dispatch reserves its rows and signals its peers through per-expert counters and handshakes; this one needs a flag per token and two local launches to place the rows.
Parameters:
- input_type (
DType): The type of the input tokens. - input_tokens_layout (
TensorLayout): The layout of the input tokens. - topk_ids_layout (
TensorLayout): The layout of the top-k expert IDs. - n_experts (
Int): The number of experts over all devices. - n_ranks (
Int): The devices of the node. - max_tokens_per_rank (
Int): The receive buffer's tokens per source device. - token_fmt_type (
TokenFormat): The token format. - InputScalesFnType (
InputScalesFn): The type ofinput_scales_fn. - fused_shared_expert (
Bool): Whether the shared expert's rows come first. - has_input_scales (
Bool): Whetherinput_scales_fnsupplies the input scale. - InputTokensEngine (
TensorEngine): Storage engine backing theinput_tokenstile. - TopkIdsEngine (
TensorEngine): Storage engine backing thetopk_idstile.
Args:
- input_tokens (
TileTensor[input_type, input_tokens_layout, ImmUntrackedOrigin, Engine=InputTokensEngine]): The tokens on this device. - topk_ids (
TileTensor[.int32, topk_ids_layout, ImmUntrackedOrigin, Engine=TopkIdsEngine]): The global top-k expert IDs of each token. - format_handler (
token_fmt_type): Writes the shared expert's rows. - recv_buf_ptrs (
Array[Pointer[UInt8, MutUntrackedOrigin], n_ranks]): Every device's dispatch receive buffer. - small_words (
MutPointer[UInt32, MutUntrackedOrigin]): This device'sEP_P2P_SMALL_WORDS. - my_rank (
Int32): The rank of this device. - input_scales_fn (
InputScalesFnType): The input scale, read whenhas_input_scales.