For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
ep_comm
Expert-parallelism (EP) communication kernels for MoE token dispatch and combine.
Implements the token dispatch (scatter) and combine (gather) collectives used in Mixture-of-Experts inference across multiple GPUs, including FP8 and FP4 quantization paths for bandwidth-limited transfers.
comptime values
BLOCK_SCOPE
comptime BLOCK_SCOPE = _BLOCK_SCOPE()
DEVICE_SCOPE
comptime DEVICE_SCOPE = _DEVICE_SCOPE()
DISPATCH_COPY_BEGIN
comptime DISPATCH_COPY_BEGIN = 0
Comm warp index starts copying its first owned token.
DISPATCH_CTA_INFO
comptime DISPATCH_CTA_INFO = 12
The CTA's send share as tokens | first_token << 16 | n_signal_sms << 40 | n_send_sms << 48 (index 0).
DISPATCH_PACK_DONE
comptime DISPATCH_PACK_DONE = 1
Comm warp index finishes packing an owned token; reported once per token.
DISPATCH_PAIR_COPY_DONE
comptime DISPATCH_PAIR_COPY_DONE = 9
Publisher index completes its first pair's copy and scale placement.
DISPATCH_PAIR_COPY_ISSUE
comptime DISPATCH_PAIR_COPY_ISSUE = 8
Publisher index issues its first pair's peer copy.
DISPATCH_PAIR_FINISHED
comptime DISPATCH_PAIR_FINISHED = 10
Publisher index issues its first pair's finished-count increment.
DISPATCH_PAIR_INFO
comptime DISPATCH_PAIR_INFO = 11
Publisher index's first pair as token | dst_rank << 16 | expert << 24 | slot << 32, with its pair count in bits 48 and up.
DISPATCH_PAIR_KEY
comptime DISPATCH_PAIR_KEY = 7
Publisher index's first pair as final_row | counter_offset << 32.
DISPATCH_PAIR_RESERVE_ISSUE
comptime DISPATCH_PAIR_RESERVE_ISSUE = 4
Publisher index issues its first pair's slot reservation.
DISPATCH_PAIR_RESERVED
comptime DISPATCH_PAIR_RESERVED = 5
Publisher index's first slot reservation returns.
DISPATCH_PAIR_ROW
comptime DISPATCH_PAIR_ROW = 6
Publisher index resolves its first pair's final row.
DISPATCH_SEND_JOIN_ARRIVE
comptime DISPATCH_SEND_JOIN_ARRIVE = 2
Comm warp index arrives at the send join.
DISPATCH_SEND_JOIN_PASS
comptime DISPATCH_SEND_JOIN_PASS = 3
Comm warp index leaves the send join.
EP_COPY_SRC_WIDTH
comptime EP_COPY_SRC_WIDTH = 8
Source elements a thread handles per copy item in the quantizing token formats. Hardcoded there rather than derived from simd_width_of, so the role split has to agree with it explicitly.
EP_DATA_READY_FLAG
comptime EP_DATA_READY_FLAG = 1024
EP_LOCAL_SYNC_RESERVED_WORDS
comptime EP_LOCAL_SYNC_RESERVED_WORDS = 262144
Int32 words (1 MiB) reserved past the regions of an EPLocalSyncCounters buffer for one fused EP kernel's own counters: the fused MegaMoE path keeps a typed view there. Only a buffer of allocation_size() has them, and ep.init zeroes them with the rest of the buffer.
EP_MX_MSG_GROUP
comptime EP_MX_MSG_GROUP = 32
Elements sharing one E8M0 scale in an MXFP8 combine message.
EP_NO_INPUT_SCALES
comptime EP_NO_INPUT_SCALES = _as_input_scales_fn[inflated$def[comptime[dtype: DType] #kgen.func.symbol<@shmem::@ep_comm::@"_unit_input_scale[::DType](::SIMD[DType.int, 1])"<:!lit.struct<@std::@builtin::@dtype::@DType> dtype>>]](.__init__())
Stands in for input_scales_fn when a dispatch has no input scales.
EP_NO_ROUTER_WEIGHTS
comptime EP_NO_ROUTER_WEIGHTS = _as_router_weights_fn[inflated$def[comptime[width: Int] #kgen.func.symbol<@shmem::@ep_comm::@"_unit_router_weight[::SIMD[DType.int, 1]](::SIMD[DType.int, 1],::SIMD[DType.int, 1])"<:!lit.struct<@std::@simd::@SIMD<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype index}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}>> width>>]](.__init__())
Stands in for router_weights_fn when a combine has no router weights.
EP_P2P_EPOCH_MAX
comptime EP_P2P_EPOCH_MAX = 2147483647
Largest epoch the P2P paths use, so a flag's (epoch << 1) | bit fits 32 bits. Past it the epoch wraps to 2: parity keeps alternating, and the zeroed flags' epoch 0 never recurs.
EP_P2P_PREFILL_PLACE_THREADS
comptime EP_P2P_PREFILL_PLACE_THREADS = 128
Threads of p2p_prefill_dispatch_place_kernel, one per expert group.
EP_P2P_SMALL_COMBINE_BYTES
comptime EP_P2P_SMALL_COMBINE_BYTES = 4096
Bytes just before the published start of a single-node combine receive buffer for the P2P small path's UInt32 flag per (token, top-k slot). ep.init allocates and zeroes them only for counter buffers that hold EP_P2P_SMALL_WORDS.
EP_P2P_SMALL_DISPATCH_BYTES
comptime EP_P2P_SMALL_DISPATCH_BYTES = 4096
Bytes just before the published start of a single-node dispatch receive buffer for the P2P small path's flags, which peers write: a UInt64 token count per (epoch parity, source rank) at offset 0, then a UInt32 flag per (epoch parity, source rank, token) at EP_P2P_SMALL_TOKEN_FLAGS_OFFSET. Their place depends on no exchange's message size, so exchanges that share the buffers (an MTP draft and its target) find the same flags. ep.init allocates and zeroes them only for counter buffers that hold EP_P2P_SMALL_WORDS.
EP_P2P_SMALL_HIDDEN_MULTIPLE
comptime EP_P2P_SMALL_HIDDEN_MULTIPLE = 1024
The hidden size the P2P small combine needs to divide: a 64-lane wavefront times the 16 one-byte elements of a vector. p2p_small_combine_kernel asserts the dtype-specific multiple, which this one implies for every combine dtype.
EP_P2P_SMALL_MAX_TOKENS
comptime EP_P2P_SMALL_MAX_TOKENS = 32
Most tokens per device the single-node P2P dispatch and combine move with the single-launch small path on AMD GPUs (see p2p_small_dispatch_kernel).
EP_P2P_SMALL_TOKEN_FLAGS_OFFSET
comptime EP_P2P_SMALL_TOKEN_FLAGS_OFFSET = 256
EP_P2P_SMALL_WORDS
comptime EP_P2P_SMALL_WORDS = 8
UInt32 words the P2P small path owns at the end of an EPLocalSyncCounters buffer this many words longer than total_size() or allocation_size(): [0] the last epoch it used, [1] its finished-CTA count, [2] 1 when the last dispatch took the small path, [3] the end of its rows, [4] the first routed row (past the shared expert's), [5] the prefill combine's finished-CTA count. The graph opts a configuration in by allocating them; a buffer without them keeps the protocol kernels.
EP_PDL_MAX_RANKS
comptime EP_PDL_MAX_RANKS = 2
Largest world size at which the split EP kernels use PDL.
PDL on dispatch_async, combine_async and combine_wait is verified on two
GPUs only. Wider launches compile the wait and the release out of the kernels
and are launched without the PDL attribute.
EP_TOKEN_INDEX_BYTES
comptime EP_TOKEN_INDEX_BYTES = 32
Bytes of one token index in a token_recv receive buffer. A peer writes each index alone, so it fills a whole 32-byte sector: a partial-sector write costs the owning GPU a read-modify-write.
EP_WARP_TOKENS_MIN_TOKENS
comptime EP_WARP_TOKENS_MIN_TOKENS = 1152
EPCombineStoreFn
comptime EPCombineStoreFn = def[dtype: DType, width: SIMDLength, *, alignment: Int = Int(1)](Coord[**?], SIMD[dtype, width]) -> None & RegisterPassable & ImplicitlyCopyable
Combine output store: (coords, value) -> None.
Kernels receive it as a trailing host-layout argument (host_args), so it
runs on the device and must copy ({var}) what it captures.
InputScalesFn
comptime InputScalesFn = def[dtype: DType](Int) -> Scalar[dtype] & RegisterPassable & ImplicitlyCopyable
Dispatch input scale of an expert: (expert_id) -> scale.
Kernels receive it as a trailing host-layout argument (host_args), so it
runs on the device and must copy ({var}) what it captures.
MAX_GPUS_PER_NODE
comptime MAX_GPUS_PER_NODE = 8
MAX_SMS_PER_DEVICE
comptime MAX_SMS_PER_DEVICE = 304
RouterWeightsFn
comptime RouterWeightsFn = def[width: Int](token_idx: Int, topk_id: Int) -> SIMD[.float32, width] & RegisterPassable & ImplicitlyCopyable
Router weight of a (token, top-k slot) pair for the combine reduction.
Kernels receive it as a trailing host-layout argument (host_args), so it
runs on the device and must copy ({var}) what it captures.
SF_ATOM_ELEMS_LOCAL
comptime SF_ATOM_ELEMS_LOCAL = ((load_from_mem SF_ATOM_M.__getitem_param__[Int(0)]()) * SF_ROW_STRIDE_LOCAL)
SF_ROW_STRIDE_LOCAL
comptime SF_ROW_STRIDE_LOCAL = ((load_from_mem SF_ATOM_M.__getitem_param__[Int(1)]()) * Int(4))
Structs
-
BF16TokenFormat: Token format that transmits the full hidden state in BFloat16. -
BlockwiseFP8TokenFormat: Token format that quantizes the hidden state to FP8 with block-wise scales. -
EPCombineKernel: Implements combine_async and combine_wait kernel logic for Expert Parallelism. -
EPDispatchKernel: Implements dispatch_async and dispatch_wait kernel logic for Expert Parallelism. -
EPLocalSyncCounters: Manages atomic counters for EP kernel synchronization within a device. -
EPRoleSplit: Splits one dispatch CTA into a copy role and a publisher role. -
ExpertLoadTally: Running per-layer, per-local-expert token totals in device memory. -
MXTokenFormat: Token format for MX (microscaling) FP4 quantization. -
NullDispatchTrace: Discards every dispatch-phase event. -
NullExpertLoad: Zero-sized sink whoseaddcompiles away. -
NVBlockScaledTokenFormat: Token format for NVIDIA block-scaled FP4/FP8 quantization.
Traits
-
DispatchTraceSink: Receives the dispatch-phase events (DISPATCH_*). -
ExpertLoadSink: Receives the routed-token count of each local expert at dispatch wait. -
TokenFormat: Specifies the wire format for a single MoE token in EP dispatch/combine.
Functions
-
block_memcpy: Copies a memory area from source to destination. This function will use the vectorized store and load instructions to copy the memory area. User should make sure pointers are aligned to the simd width. -
block_prefix_sum: Performs a prefix sum (scan) operation across all threads in a block. -
combine_async_kernel: Send tokens to the original rank based on the src_info tensor. This kernel utilizes the non-blocking SHMEM API, and would return immediately after initiating the communication. The communication is considered complete after calling thecombine_wait_kernel. -
combine_kernel: Fused combine kernel that combines combine_async and combine_wait functionality in a single kernel launch. -
combine_wait_kernel: This kernel is called after thecombine_kernelto complete the communication. It will keep polling the receive count buffer, and once the count is no longer MAX_FINITE, it can confirm that the communication is complete from a remote rank. -
dispatch_async_kernel: Dispatch tokens to experts on remote ranks based on the top-k expert IDs. This kernel utilizes the non-blocking SHMEM API ifuse_shmemis True, and would return immediately after initiating the communication. The communication is considered complete after calling thedispatch_wait_kernel. -
dispatch_kernel: Fused dispatch kernel that combines dispatch_async and dispatch_wait functionality in a single kernel launch. -
dispatch_wait_kernel: This kernel is called after thedispatch_kernelto complete the communication. It will keep polling the receive count buffer, and once the count is no longer MAX_FINITE, it can confirm that the communication is complete from a remote rank. -
ep_dispatch_recv_size: Returns the bytes of one rank's EP dispatch receive buffer. -
ep_mx_msg_bytes: Bytes of an MXFP8 combine message: E4M3 values, then E8M0 scales. -
ep_p2p_combine_header_bytes: Returns the bytesep.initputs before each single-node combine receive buffer that takes the P2P paths: the small path's flags, and the partial-row combine's scratch when it is on. -
ep_p2p_dispatch_header_bytes: Returns the bytesep.initputs before each single-node dispatch receive buffer that takes the P2P paths: the small path's flags, and the prefill dispatch's scratch when it is on. None of it depends on the message size, so exchanges that share the buffers agree on it. -
ep_p2p_prefill_on: Whether a step the P2P small path declines takes the P2P prefill dispatch and the partial-row combine, which the dispatch and combine APIs decide alike: a token's partial rows (one per device) take the slots of its top-k rows, the kernels hold top-k-wide vectors, and the place kernel has a thread per expert group. -
ep_signal_completion: Signals the completion of the communication by writing to the receive count buffer. Will use direct memory access if the target device is on the same node, and use the SHMEM API if the target device is on a different node. -
ep_token_index_offset: Returns where atoken_recvreceive buffer's token indices start. -
fused_silu_kernel: This kernel performs the gated activation for all the MLPs in the EP MoE module. We need to manually implement the kernel here is because after the EP dispatch phase, the actual number of received tokens is not known to the host. This kernel will read the row offsets to determine the actual number of received tokens in the input tensor. -
fused_silu_mx_kernel: This kernel performs the gated activation for all the MLPs in the EP MoE module. We need to manually implement the kernel here is because after the EP dispatch phase, the actual number of received tokens is not known to the host. This kernel will read the row offsets to determine the actual number of received tokens in the input tensor. -
fused_silu_mxfp6_kernel: Gated activation + MXFP6 quantization for the EP MoE down-projection input. The activation is the comptimeactivationparameter (defaults toswiglu). -
fused_silu_mxfp8_interleaved_kernel: SwiGLU + MXFP8 quantization for interleaved gate/up layout. -
fused_silu_nvfp4_interleaved_kernel: SwiGLU + NVFP4 quantization for interleaved gate/up layout. -
fused_silu_nvfp4_kernel: This kernel performs the gated activation for all the MLPs in the EP MoE module. We need to manually implement the kernel here is because after the EP dispatch phase, the actual number of received tokens is not known to the host. This kernel will read the row offsets to determine the actual number of received tokens in the input tensor. -
get_device_alignment: Returns the natural SIMD alignment in bytes for the current GPU target. -
p2p_prefill_combine_kernel: Second half of the partial-row combine of a prefill-dispatched step. -
p2p_prefill_combine_prep_kernel: First half of the partial-row combine of a prefill-dispatched step. -
p2p_prefill_dispatch_count_kernel: Second launch of the P2P prefill dispatch: waits for every source's token flags and counts this device's rows per local expert. -
p2p_prefill_dispatch_place_kernel: Last launch of the P2P prefill dispatch: lays out the rows. -
p2p_prefill_dispatch_send_kernel: First of the three launches of the P2P prefill dispatch. -
p2p_small_combine_kernel: The combine paired withp2p_small_dispatch_kernel, in one launch. -
p2p_small_dispatch_kernel: Dispatches a decode step's tokens between the devices of one node in one launch, or decides that the step is too large for it.