For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
p2p_small_dispatch_kernel
def p2p_small_dispatch_kernel[input_type: DType, num_threads: Int, input_tokens_layout: TensorLayout, topk_ids_layout: TensorLayout, row_offsets_layout: TensorLayout, expert_ids_layout: TensorLayout, src_info_layout: TensorLayout, n_experts: Int, n_ranks: Int, max_tokens_per_rank: Int, token_fmt_type: TokenFormat, InputScalesFnType: InputScalesFn, fused_shared_expert: Bool, has_input_scales: Bool, small_max_tokens: Int, *, InputTokensEngine: TensorEngine = DefaultEngine, TopkIdsEngine: TensorEngine = DefaultEngine, RowOffsetsEngine: TensorEngine = DefaultEngine, ExpertIdsEngine: TensorEngine = DefaultEngine, SrcInfoEngine: TensorEngine = DefaultEngine, ExpertLoadT: ExpertLoadSink = NullExpertLoad](input_tokens: TileTensor[input_type, input_tokens_layout, ImmUntrackedOrigin, Engine=InputTokensEngine], topk_ids: TileTensor[.int32, topk_ids_layout, ImmUntrackedOrigin, Engine=TopkIdsEngine], format_handler: token_fmt_type, row_offsets: TileTensor[.uint32, row_offsets_layout, MutUntrackedOrigin, Engine=RowOffsetsEngine], expert_ids: TileTensor[.int32, expert_ids_layout, MutUntrackedOrigin, Engine=ExpertIdsEngine], src_info: TileTensor[.int32, src_info_layout, MutUntrackedOrigin, Engine=SrcInfoEngine], recv_buf_ptrs: Array[Pointer[UInt8, MutUntrackedOrigin], n_ranks], small_words: MutPointer[UInt32, MutUntrackedOrigin], my_rank: Int32, expert_load: ExpertLoadT, input_scales_fn: InputScalesFnType)
Dispatches a decode step's tokens between the devices of one node in one launch, or decides that the step is too large for it.
At decode sizes the protocol dispatch_kernel spends its time on
cross-CTA and cross-device handshakes, not on data. This kernel replaces
them with one flag per token. Each device publishes its token count to
every peer; the step takes this path only when no device holds more than
small_max_tokens tokens, a decision every device reaches from the same
counts. Otherwise the kernel records that and the protocol kernel runs
the step. Either way the devices advance one shared epoch, which tags
every flag, so flags never need resetting.
On the small path CTA t quantizes token t once, writes the message
(with its top-k ids) to each device that owns one of its experts, at the
message slot the protocol uses, and flags the token at every device.
Every CTA then waits for all the devices' flags, rebuilds the routing of
every token in shared memory, and copies its share of the received
messages to their rows. Rows follow the protocol's output layout: the
shared expert's rows (this device's tokens, in order) first when fused,
then each local expert's rows, padded by the token format. Within an
expert, rows follow (source rank, token, top-k) order. src_info[row]
holds (source rank * small_max_tokens + token, top-k), or -1 for a
padding row, which only p2p_small_combine_kernel reads.
Flags live at the end of the receive buffer and come in two epoch parities: a device can run one dispatch ahead of a peer that has not yet read its flags, but not two, since each dispatch waits for every device's count. A device writes a message only to a peer that will copy it, and it then waits for that peer's expert outputs in the combine, so a message is never overwritten before it is read.
Parameters:
- input_type (
DType): The type of the input tokens. - num_threads (
Int): The number of threads in the block. - input_tokens_layout (
TensorLayout): The layout of the input tokens. - topk_ids_layout (
TensorLayout): The layout of the top-k expert IDs. - row_offsets_layout (
TensorLayout): The layout of the row offsets. - expert_ids_layout (
TensorLayout): The layout of the expert IDs. - src_info_layout (
TensorLayout): The layout of the source token info. - n_experts (
Int): The number of experts over all devices. - n_ranks (
Int): The devices of the node. - max_tokens_per_rank (
Int): The receive buffer's tokens per source device. - token_fmt_type (
TokenFormat): The token format. - InputScalesFnType (
InputScalesFn): The type ofinput_scales_fn. - fused_shared_expert (
Bool): Whether the shared expert's rows come first. - has_input_scales (
Bool): Whetherinput_scales_fnsupplies the input scale. - small_max_tokens (
Int): Most tokens per device this path moves. - InputTokensEngine (
TensorEngine): Storage engine backing theinput_tokenstile. - TopkIdsEngine (
TensorEngine): Storage engine backing thetopk_idstile. - RowOffsetsEngine (
TensorEngine): Storage engine backing therow_offsetstile. - ExpertIdsEngine (
TensorEngine): Storage engine backing theexpert_idstile. - SrcInfoEngine (
TensorEngine): Storage engine backing thesrc_infotile. - ExpertLoadT (
ExpertLoadSink): Type ofexpert_load; the defaultNullExpertLoadcompiles the counting away.
Args:
- input_tokens (
TileTensor[input_type, input_tokens_layout, ImmUntrackedOrigin, Engine=InputTokensEngine]): The tokens on this device. - topk_ids (
TileTensor[.int32, topk_ids_layout, ImmUntrackedOrigin, Engine=TopkIdsEngine]): The global top-k expert IDs of each token. - format_handler (
token_fmt_type): Writes a token's message to its output rows. - row_offsets (
TileTensor[.uint32, row_offsets_layout, MutUntrackedOrigin, Engine=RowOffsetsEngine]): The first row of each expert group, then the end. - expert_ids (
TileTensor[.int32, expert_ids_layout, MutUntrackedOrigin, Engine=ExpertIdsEngine]): The expert of each group. - src_info (
TileTensor[.int32, src_info_layout, MutUntrackedOrigin, Engine=SrcInfoEngine]): The source of each row (see above). - recv_buf_ptrs (
Array[Pointer[UInt8, MutUntrackedOrigin], n_ranks]): Every device's dispatch receive buffer. - small_words (
MutPointer[UInt32, MutUntrackedOrigin]): This device'sEP_P2P_SMALL_WORDS. - my_rank (
Int32): The rank of this device. - expert_load (
ExpertLoadT): Receives each local expert's routed-token count, for EPLB load statistics. - input_scales_fn (
InputScalesFnType): The input scale, read whenhas_input_scales.