For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
p2p_prefill_dispatch_place_kernel
def p2p_prefill_dispatch_place_kernel[num_threads: Int, row_offsets_layout: TensorLayout, expert_ids_layout: TensorLayout, src_info_layout: TensorLayout, n_experts: Int, n_ranks: Int, max_tokens_per_rank: Int, token_fmt_type: TokenFormat, fused_shared_expert: Bool, *, RowOffsetsEngine: TensorEngine = DefaultEngine, ExpertIdsEngine: TensorEngine = DefaultEngine, SrcInfoEngine: TensorEngine = DefaultEngine, ExpertLoadT: ExpertLoadSink = NullExpertLoad](format_handler: token_fmt_type, row_offsets: TileTensor[.uint32, row_offsets_layout, MutUntrackedOrigin, Engine=RowOffsetsEngine], expert_ids: TileTensor[.int32, expert_ids_layout, MutUntrackedOrigin, Engine=ExpertIdsEngine], src_info: TileTensor[.int32, src_info_layout, MutUntrackedOrigin, Engine=SrcInfoEngine], num_tokens: Int32, recv_buf_ptrs: Array[Pointer[UInt8, MutUntrackedOrigin], n_ranks], small_words: MutPointer[UInt32, MutUntrackedOrigin], my_rank: Int32, expert_load: ExpertLoadT)
Last launch of the P2P prefill dispatch: lays out the rows.
Every CTA turns the histogram into the protocol's row offsets (the shared
expert's group first when fused, each group padded by the token format);
CTA 0 writes them and marks padding rows. Then one warp per received
token stages its message in shared memory and copies it to a row of each
local expert it is routed to, taken from that expert's cursor, so rows
within an expert follow arrival order. src_info[row] is
(source * max_tokens_per_rank + token, k), which
p2p_prefill_combine_prep_kernel reads.
Parameters:
- num_threads (
Int): The number of threads in the block. - row_offsets_layout (
TensorLayout): The layout of the row offsets. - expert_ids_layout (
TensorLayout): The layout of the expert IDs. - src_info_layout (
TensorLayout): The layout of the row sources. - n_experts (
Int): The number of experts over all devices. - n_ranks (
Int): The devices of the node. - max_tokens_per_rank (
Int): The receive buffer's tokens per source device. - token_fmt_type (
TokenFormat): The token format. - fused_shared_expert (
Bool): Whether the shared expert's rows come first. - RowOffsetsEngine (
TensorEngine): Storage engine backing therow_offsetstile. - ExpertIdsEngine (
TensorEngine): Storage engine backing theexpert_idstile. - SrcInfoEngine (
TensorEngine): Storage engine backing thesrc_infotile. - ExpertLoadT (
ExpertLoadSink): Type ofexpert_load; the defaultNullExpertLoadcompiles the counting away.
Args:
- format_handler (
token_fmt_type): Writes a token's message to its rows. - row_offsets (
TileTensor[.uint32, row_offsets_layout, MutUntrackedOrigin, Engine=RowOffsetsEngine]): The first row of each expert group, then the end. - expert_ids (
TileTensor[.int32, expert_ids_layout, MutUntrackedOrigin, Engine=ExpertIdsEngine]): The expert of each group. - src_info (
TileTensor[.int32, src_info_layout, MutUntrackedOrigin, Engine=SrcInfoEngine]): The source of each row (see above). - num_tokens (
Int32): The tokens on this device. - recv_buf_ptrs (
Array[Pointer[UInt8, MutUntrackedOrigin], n_ranks]): Every device's dispatch receive buffer. - small_words (
MutPointer[UInt32, MutUntrackedOrigin]): This device'sEP_P2P_SMALL_WORDS. - my_rank (
Int32): The rank of this device. - expert_load (
ExpertLoadT): Receives each local expert's routed-token count, for EPLB load statistics.