IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

p2p_small_dispatch_kernel

def p2p_small_dispatch_kernel[input_type: DType, num_threads: Int, input_tokens_layout: TensorLayout, topk_ids_layout: TensorLayout, row_offsets_layout: TensorLayout, expert_ids_layout: TensorLayout, src_info_layout: TensorLayout, n_experts: Int, n_ranks: Int, max_tokens_per_rank: Int, token_fmt_type: TokenFormat, InputScalesFnType: InputScalesFn, fused_shared_expert: Bool, has_input_scales: Bool, small_max_tokens: Int, *, InputTokensEngine: TensorEngine = DefaultEngine, TopkIdsEngine: TensorEngine = DefaultEngine, RowOffsetsEngine: TensorEngine = DefaultEngine, ExpertIdsEngine: TensorEngine = DefaultEngine, SrcInfoEngine: TensorEngine = DefaultEngine, ExpertLoadT: ExpertLoadSink = NullExpertLoad](input_tokens: TileTensor[input_type, input_tokens_layout, ImmUntrackedOrigin, Engine=InputTokensEngine], topk_ids: TileTensor[.int32, topk_ids_layout, ImmUntrackedOrigin, Engine=TopkIdsEngine], format_handler: token_fmt_type, row_offsets: TileTensor[.uint32, row_offsets_layout, MutUntrackedOrigin, Engine=RowOffsetsEngine], expert_ids: TileTensor[.int32, expert_ids_layout, MutUntrackedOrigin, Engine=ExpertIdsEngine], src_info: TileTensor[.int32, src_info_layout, MutUntrackedOrigin, Engine=SrcInfoEngine], recv_buf_ptrs: Array[Pointer[UInt8, MutUntrackedOrigin], n_ranks], small_words: MutPointer[UInt32, MutUntrackedOrigin], my_rank: Int32, expert_load: ExpertLoadT, input_scales_fn: InputScalesFnType)

Dispatches a decode step's tokens between the devices of one node in one launch, or decides that the step is too large for it.

At decode sizes the protocol dispatch_kernel spends its time on cross-CTA and cross-device handshakes, not on data. This kernel replaces them with one flag per token. Each device publishes its token count to every peer; the step takes this path only when no device holds more than small_max_tokens tokens, a decision every device reaches from the same counts. Otherwise the kernel records that and the protocol kernel runs the step. Either way the devices advance one shared epoch, which tags every flag, so flags never need resetting.

On the small path CTA t quantizes token t once, writes the message (with its top-k ids) to each device that owns one of its experts, at the message slot the protocol uses, and flags the token at every device. Every CTA then waits for all the devices' flags, rebuilds the routing of every token in shared memory, and copies its share of the received messages to their rows. Rows follow the protocol's output layout: the shared expert's rows (this device's tokens, in order) first when fused, then each local expert's rows, padded by the token format. Within an expert, rows follow (source rank, token, top-k) order. src_info[row] holds (source rank * small_max_tokens + token, top-k), or -1 for a padding row, which only p2p_small_combine_kernel reads.

Flags live at the end of the receive buffer and come in two epoch parities: a device can run one dispatch ahead of a peer that has not yet read its flags, but not two, since each dispatch waits for every device's count. A device writes a message only to a peer that will copy it, and it then waits for that peer's expert outputs in the combine, so a message is never overwritten before it is read.

Parameters:

  • ​input_type (DType): The type of the input tokens.
  • ​num_threads (Int): The number of threads in the block.
  • ​input_tokens_layout (TensorLayout): The layout of the input tokens.
  • ​topk_ids_layout (TensorLayout): The layout of the top-k expert IDs.
  • ​row_offsets_layout (TensorLayout): The layout of the row offsets.
  • ​expert_ids_layout (TensorLayout): The layout of the expert IDs.
  • ​src_info_layout (TensorLayout): The layout of the source token info.
  • ​n_experts (Int): The number of experts over all devices.
  • ​n_ranks (Int): The devices of the node.
  • ​max_tokens_per_rank (Int): The receive buffer's tokens per source device.
  • ​token_fmt_type (TokenFormat): The token format.
  • ​InputScalesFnType (InputScalesFn): The type of input_scales_fn.
  • ​fused_shared_expert (Bool): Whether the shared expert's rows come first.
  • ​has_input_scales (Bool): Whether input_scales_fn supplies the input scale.
  • ​small_max_tokens (Int): Most tokens per device this path moves.
  • ​InputTokensEngine (TensorEngine): Storage engine backing the input_tokens tile.
  • ​TopkIdsEngine (TensorEngine): Storage engine backing the topk_ids tile.
  • ​RowOffsetsEngine (TensorEngine): Storage engine backing the row_offsets tile.
  • ​ExpertIdsEngine (TensorEngine): Storage engine backing the expert_ids tile.
  • ​SrcInfoEngine (TensorEngine): Storage engine backing the src_info tile.
  • ​ExpertLoadT (ExpertLoadSink): Type of expert_load; the default NullExpertLoad compiles the counting away.

Args:

Was this page helpful?