IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

ep_comm

Expert-parallelism (EP) communication kernels for MoE token dispatch and combine.

Implements the token dispatch (scatter) and combine (gather) collectives used in Mixture-of-Experts inference across multiple GPUs, including FP8 and FP4 quantization paths for bandwidth-limited transfers.

comptime values​

BLOCK_SCOPE​

comptime BLOCK_SCOPE = _BLOCK_SCOPE()

DEVICE_SCOPE​

comptime DEVICE_SCOPE = _DEVICE_SCOPE()

DISPATCH_COPY_BEGIN​

comptime DISPATCH_COPY_BEGIN = 0

Comm warp index starts copying its first owned token.

DISPATCH_CTA_INFO​

comptime DISPATCH_CTA_INFO = 12

The CTA's send share as tokens | first_token << 16 | n_signal_sms << 40 | n_send_sms << 48 (index 0).

DISPATCH_PACK_DONE​

comptime DISPATCH_PACK_DONE = 1

Comm warp index finishes packing an owned token; reported once per token.

DISPATCH_PAIR_COPY_DONE​

comptime DISPATCH_PAIR_COPY_DONE = 9

Publisher index completes its first pair's copy and scale placement.

DISPATCH_PAIR_COPY_ISSUE​

comptime DISPATCH_PAIR_COPY_ISSUE = 8

Publisher index issues its first pair's peer copy.

DISPATCH_PAIR_FINISHED​

comptime DISPATCH_PAIR_FINISHED = 10

Publisher index issues its first pair's finished-count increment.

DISPATCH_PAIR_INFO​

comptime DISPATCH_PAIR_INFO = 11

Publisher index's first pair as token | dst_rank << 16 | expert << 24 | slot << 32, with its pair count in bits 48 and up.

DISPATCH_PAIR_KEY​

comptime DISPATCH_PAIR_KEY = 7

Publisher index's first pair as final_row | counter_offset << 32.

DISPATCH_PAIR_RESERVE_ISSUE​

comptime DISPATCH_PAIR_RESERVE_ISSUE = 4

Publisher index issues its first pair's slot reservation.

DISPATCH_PAIR_RESERVED​

comptime DISPATCH_PAIR_RESERVED = 5

Publisher index's first slot reservation returns.

DISPATCH_PAIR_ROW​

comptime DISPATCH_PAIR_ROW = 6

Publisher index resolves its first pair's final row.

DISPATCH_SEND_JOIN_ARRIVE​

comptime DISPATCH_SEND_JOIN_ARRIVE = 2

Comm warp index arrives at the send join.

DISPATCH_SEND_JOIN_PASS​

comptime DISPATCH_SEND_JOIN_PASS = 3

Comm warp index leaves the send join.

EP_COPY_SRC_WIDTH​

comptime EP_COPY_SRC_WIDTH = 8

Source elements a thread handles per copy item in the quantizing token formats. Hardcoded there rather than derived from simd_width_of, so the role split has to agree with it explicitly.

EP_DATA_READY_FLAG​

comptime EP_DATA_READY_FLAG = 1024

EP_LOCAL_SYNC_RESERVED_WORDS​

comptime EP_LOCAL_SYNC_RESERVED_WORDS = 262144

Int32 words (1 MiB) reserved past the regions of an EPLocalSyncCounters buffer for one fused EP kernel's own counters: the fused MegaMoE path keeps a typed view there. Only a buffer of allocation_size() has them, and ep.init zeroes them with the rest of the buffer.

EP_MX_MSG_GROUP​

comptime EP_MX_MSG_GROUP = 32

Elements sharing one E8M0 scale in an MXFP8 combine message.

EP_NO_INPUT_SCALES​

comptime EP_NO_INPUT_SCALES = _as_input_scales_fn[inflated$def[comptime[dtype: DType] #kgen.func.symbol<@shmem::@ep_comm::@"_unit_input_scale[::DType](::SIMD[DType.int, 1])"<:!lit.struct<@std::@builtin::@dtype::@DType> dtype>>]](.__init__())

Stands in for input_scales_fn when a dispatch has no input scales.

EP_NO_ROUTER_WEIGHTS​

comptime EP_NO_ROUTER_WEIGHTS = _as_router_weights_fn[inflated$def[comptime[width: Int] #kgen.func.symbol<@shmem::@ep_comm::@"_unit_router_weight[::SIMD[DType.int, 1]](::SIMD[DType.int, 1],::SIMD[DType.int, 1])"<:!lit.struct<@std::@simd::@SIMD<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype index}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}>> width>>]](.__init__())

Stands in for router_weights_fn when a combine has no router weights.

EP_P2P_EPOCH_MAX​

comptime EP_P2P_EPOCH_MAX = 2147483647

Largest epoch the P2P paths use, so a flag's (epoch << 1) | bit fits 32 bits. Past it the epoch wraps to 2: parity keeps alternating, and the zeroed flags' epoch 0 never recurs.

EP_P2P_PREFILL_PLACE_THREADS​

comptime EP_P2P_PREFILL_PLACE_THREADS = 128

Threads of p2p_prefill_dispatch_place_kernel, one per expert group.

EP_P2P_SMALL_COMBINE_BYTES​

comptime EP_P2P_SMALL_COMBINE_BYTES = 4096

Bytes just before the published start of a single-node combine receive buffer for the P2P small path's UInt32 flag per (token, top-k slot). ep.init allocates and zeroes them only for counter buffers that hold EP_P2P_SMALL_WORDS.

EP_P2P_SMALL_DISPATCH_BYTES​

comptime EP_P2P_SMALL_DISPATCH_BYTES = 4096

Bytes just before the published start of a single-node dispatch receive buffer for the P2P small path's flags, which peers write: a UInt64 token count per (epoch parity, source rank) at offset 0, then a UInt32 flag per (epoch parity, source rank, token) at EP_P2P_SMALL_TOKEN_FLAGS_OFFSET. Their place depends on no exchange's message size, so exchanges that share the buffers (an MTP draft and its target) find the same flags. ep.init allocates and zeroes them only for counter buffers that hold EP_P2P_SMALL_WORDS.

EP_P2P_SMALL_HIDDEN_MULTIPLE​

comptime EP_P2P_SMALL_HIDDEN_MULTIPLE = 1024

The hidden size the P2P small combine needs to divide: a 64-lane wavefront times the 16 one-byte elements of a vector. p2p_small_combine_kernel asserts the dtype-specific multiple, which this one implies for every combine dtype.

EP_P2P_SMALL_MAX_TOKENS​

comptime EP_P2P_SMALL_MAX_TOKENS = 32

Most tokens per device the single-node P2P dispatch and combine move with the single-launch small path on AMD GPUs (see p2p_small_dispatch_kernel).

EP_P2P_SMALL_TOKEN_FLAGS_OFFSET​

comptime EP_P2P_SMALL_TOKEN_FLAGS_OFFSET = 256

EP_P2P_SMALL_WORDS​

comptime EP_P2P_SMALL_WORDS = 8

UInt32 words the P2P small path owns at the end of an EPLocalSyncCounters buffer this many words longer than total_size() or allocation_size(): [0] the last epoch it used, [1] its finished-CTA count, [2] 1 when the last dispatch took the small path, [3] the end of its rows, [4] the first routed row (past the shared expert's), [5] the prefill combine's finished-CTA count. The graph opts a configuration in by allocating them; a buffer without them keeps the protocol kernels.

EP_PDL_MAX_RANKS​

comptime EP_PDL_MAX_RANKS = 2

Largest world size at which the split EP kernels use PDL.

PDL on dispatch_async, combine_async and combine_wait is verified on two GPUs only. Wider launches compile the wait and the release out of the kernels and are launched without the PDL attribute.

EP_TOKEN_INDEX_BYTES​

comptime EP_TOKEN_INDEX_BYTES = 32

Bytes of one token index in a token_recv receive buffer. A peer writes each index alone, so it fills a whole 32-byte sector: a partial-sector write costs the owning GPU a read-modify-write.

EP_WARP_TOKENS_MIN_TOKENS​

comptime EP_WARP_TOKENS_MIN_TOKENS = 1152

EPCombineStoreFn​

comptime EPCombineStoreFn = def[dtype: DType, width: SIMDLength, *, alignment: Int = Int(1)](Coord[**?], SIMD[dtype, width]) -> None & RegisterPassable & ImplicitlyCopyable

Combine output store: (coords, value) -> None.

Kernels receive it as a trailing host-layout argument (host_args), so it runs on the device and must copy ({var}) what it captures.

InputScalesFn​

comptime InputScalesFn = def[dtype: DType](Int) -> Scalar[dtype] & RegisterPassable & ImplicitlyCopyable

Dispatch input scale of an expert: (expert_id) -> scale.

Kernels receive it as a trailing host-layout argument (host_args), so it runs on the device and must copy ({var}) what it captures.

MAX_GPUS_PER_NODE​

comptime MAX_GPUS_PER_NODE = 8

MAX_SMS_PER_DEVICE​

comptime MAX_SMS_PER_DEVICE = 304

RouterWeightsFn​

comptime RouterWeightsFn = def[width: Int](token_idx: Int, topk_id: Int) -> SIMD[.float32, width] & RegisterPassable & ImplicitlyCopyable

Router weight of a (token, top-k slot) pair for the combine reduction.

Kernels receive it as a trailing host-layout argument (host_args), so it runs on the device and must copy ({var}) what it captures.

SF_ATOM_ELEMS_LOCAL​

comptime SF_ATOM_ELEMS_LOCAL = ((load_from_mem SF_ATOM_M.__getitem_param__[Int(0)]()) * SF_ROW_STRIDE_LOCAL)

SF_ROW_STRIDE_LOCAL​

comptime SF_ROW_STRIDE_LOCAL = ((load_from_mem SF_ATOM_M.__getitem_param__[Int(1)]()) * Int(4))

Structs​

Traits​

Functions​

  • ​block_memcpy: Copies a memory area from source to destination. This function will use the vectorized store and load instructions to copy the memory area. User should make sure pointers are aligned to the simd width.
  • ​block_prefix_sum: Performs a prefix sum (scan) operation across all threads in a block.
  • ​combine_async_kernel: Send tokens to the original rank based on the src_info tensor. This kernel utilizes the non-blocking SHMEM API, and would return immediately after initiating the communication. The communication is considered complete after calling the combine_wait_kernel.
  • ​combine_kernel: Fused combine kernel that combines combine_async and combine_wait functionality in a single kernel launch.
  • ​combine_wait_kernel: This kernel is called after the combine_kernel to complete the communication. It will keep polling the receive count buffer, and once the count is no longer MAX_FINITE, it can confirm that the communication is complete from a remote rank.
  • ​dispatch_async_kernel: Dispatch tokens to experts on remote ranks based on the top-k expert IDs. This kernel utilizes the non-blocking SHMEM API if use_shmem is True, and would return immediately after initiating the communication. The communication is considered complete after calling the dispatch_wait_kernel.
  • ​dispatch_kernel: Fused dispatch kernel that combines dispatch_async and dispatch_wait functionality in a single kernel launch.
  • ​dispatch_wait_kernel: This kernel is called after the dispatch_kernel to complete the communication. It will keep polling the receive count buffer, and once the count is no longer MAX_FINITE, it can confirm that the communication is complete from a remote rank.
  • ​ep_dispatch_recv_size: Returns the bytes of one rank's EP dispatch receive buffer.
  • ​ep_mx_msg_bytes: Bytes of an MXFP8 combine message: E4M3 values, then E8M0 scales.
  • ​ep_p2p_combine_header_bytes: Returns the bytes ep.init puts before each single-node combine receive buffer that takes the P2P paths: the small path's flags, and the partial-row combine's scratch when it is on.
  • ​ep_p2p_dispatch_header_bytes: Returns the bytes ep.init puts before each single-node dispatch receive buffer that takes the P2P paths: the small path's flags, and the prefill dispatch's scratch when it is on. None of it depends on the message size, so exchanges that share the buffers agree on it.
  • ​ep_p2p_prefill_on: Whether a step the P2P small path declines takes the P2P prefill dispatch and the partial-row combine, which the dispatch and combine APIs decide alike: a token's partial rows (one per device) take the slots of its top-k rows, the kernels hold top-k-wide vectors, and the place kernel has a thread per expert group.
  • ​ep_signal_completion: Signals the completion of the communication by writing to the receive count buffer. Will use direct memory access if the target device is on the same node, and use the SHMEM API if the target device is on a different node.
  • ​ep_token_index_offset: Returns where a token_recv receive buffer's token indices start.
  • ​fused_silu_kernel: This kernel performs the gated activation for all the MLPs in the EP MoE module. We need to manually implement the kernel here is because after the EP dispatch phase, the actual number of received tokens is not known to the host. This kernel will read the row offsets to determine the actual number of received tokens in the input tensor.
  • ​fused_silu_mx_kernel: This kernel performs the gated activation for all the MLPs in the EP MoE module. We need to manually implement the kernel here is because after the EP dispatch phase, the actual number of received tokens is not known to the host. This kernel will read the row offsets to determine the actual number of received tokens in the input tensor.
  • ​fused_silu_mxfp6_kernel: Gated activation + MXFP6 quantization for the EP MoE down-projection input. The activation is the comptime activation parameter (defaults to swiglu).
  • ​fused_silu_mxfp8_interleaved_kernel: SwiGLU + MXFP8 quantization for interleaved gate/up layout.
  • ​fused_silu_nvfp4_interleaved_kernel: SwiGLU + NVFP4 quantization for interleaved gate/up layout.
  • ​fused_silu_nvfp4_kernel: This kernel performs the gated activation for all the MLPs in the EP MoE module. We need to manually implement the kernel here is because after the EP dispatch phase, the actual number of received tokens is not known to the host. This kernel will read the row offsets to determine the actual number of received tokens in the input tensor.
  • ​get_device_alignment: Returns the natural SIMD alignment in bytes for the current GPU target.
  • ​p2p_prefill_combine_kernel: Second half of the partial-row combine of a prefill-dispatched step.
  • ​p2p_prefill_combine_prep_kernel: First half of the partial-row combine of a prefill-dispatched step.
  • ​p2p_prefill_dispatch_count_kernel: Second launch of the P2P prefill dispatch: waits for every source's token flags and counts this device's rows per local expert.
  • ​p2p_prefill_dispatch_place_kernel: Last launch of the P2P prefill dispatch: lays out the rows.
  • ​p2p_prefill_dispatch_send_kernel: First of the three launches of the P2P prefill dispatch.
  • ​p2p_small_combine_kernel: The combine paired with p2p_small_dispatch_kernel, in one launch.
  • ​p2p_small_dispatch_kernel: Dispatches a decode step's tokens between the devices of one node in one launch, or decides that the step is too large for it.

Was this page helpful?