IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

p2p_prefill_dispatch_place_kernel

def p2p_prefill_dispatch_place_kernel[num_threads: Int, row_offsets_layout: TensorLayout, expert_ids_layout: TensorLayout, src_info_layout: TensorLayout, n_experts: Int, n_ranks: Int, max_tokens_per_rank: Int, token_fmt_type: TokenFormat, fused_shared_expert: Bool, *, RowOffsetsEngine: TensorEngine = DefaultEngine, ExpertIdsEngine: TensorEngine = DefaultEngine, SrcInfoEngine: TensorEngine = DefaultEngine, ExpertLoadT: ExpertLoadSink = NullExpertLoad](format_handler: token_fmt_type, row_offsets: TileTensor[.uint32, row_offsets_layout, MutUntrackedOrigin, Engine=RowOffsetsEngine], expert_ids: TileTensor[.int32, expert_ids_layout, MutUntrackedOrigin, Engine=ExpertIdsEngine], src_info: TileTensor[.int32, src_info_layout, MutUntrackedOrigin, Engine=SrcInfoEngine], num_tokens: Int32, recv_buf_ptrs: Array[Pointer[UInt8, MutUntrackedOrigin], n_ranks], small_words: MutPointer[UInt32, MutUntrackedOrigin], my_rank: Int32, expert_load: ExpertLoadT)

Last launch of the P2P prefill dispatch: lays out the rows.

Every CTA turns the histogram into the protocol's row offsets (the shared expert's group first when fused, each group padded by the token format); CTA 0 writes them and marks padding rows. Then one warp per received token stages its message in shared memory and copies it to a row of each local expert it is routed to, taken from that expert's cursor, so rows within an expert follow arrival order. src_info[row] is (source * max_tokens_per_rank + token, k), which p2p_prefill_combine_prep_kernel reads.

Parameters:

  • ​num_threads (Int): The number of threads in the block.
  • ​row_offsets_layout (TensorLayout): The layout of the row offsets.
  • ​expert_ids_layout (TensorLayout): The layout of the expert IDs.
  • ​src_info_layout (TensorLayout): The layout of the row sources.
  • ​n_experts (Int): The number of experts over all devices.
  • ​n_ranks (Int): The devices of the node.
  • ​max_tokens_per_rank (Int): The receive buffer's tokens per source device.
  • ​token_fmt_type (TokenFormat): The token format.
  • ​fused_shared_expert (Bool): Whether the shared expert's rows come first.
  • ​RowOffsetsEngine (TensorEngine): Storage engine backing the row_offsets tile.
  • ​ExpertIdsEngine (TensorEngine): Storage engine backing the expert_ids tile.
  • ​SrcInfoEngine (TensorEngine): Storage engine backing the src_info tile.
  • ​ExpertLoadT (ExpertLoadSink): Type of expert_load; the default NullExpertLoad compiles the counting away.

Args:

Was this page helpful?