For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
DistributedEPDispatchBlockScaledNVDynGlobalScales
struct DistributedEPDispatchBlockScaledNVDynGlobalScales
Registers the mo.distributed.ep.dispatch.block.scaled.nv.dyn_global_scales graph op with the graph compiler.
Implemented traits
Methods
execute
static def execute[OutTokens: TypeList[OutTokens.values], OutScales: TypeList[OutScales.values], OutRowwiseScales: TypeList[OutRowwiseScales.values], RowOffsets: TypeList[RowOffsets.values], ScalesOffsets: TypeList[ScalesOffsets.values], ExpertIds: TypeList[ExpertIds.values], SrcInfos: TypeList[SrcInfos.values], InTokens: TypeList[InTokens.values], TopkIds: TypeList[TopkIds.values], SendPtrs: TypeList[SendPtrs.values], RecvPtrs: TypeList[RecvPtrs.values], RecvCountPtrs: TypeList[RecvCountPtrs.values], Counters: TypeList[Counters.values], hidden_size: Int, top_k: Int, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, fused_shared_expert: Bool, //, _trace_name: StaticString](output_tokens: TensorTuple[*OutTokens.values], output_scales: TensorTuple[*OutScales.values], output_rowwise_scales: TensorTuple[*OutRowwiseScales.values], row_offsets: TensorTuple[*RowOffsets.values], scales_offsets: TensorTuple[*ScalesOffsets.values], expert_ids: TensorTuple[*ExpertIds.values], src_info: TensorTuple[*SrcInfos.values], input_tokens: TensorTuple[*InTokens.values], topk_ids: TensorTuple[*TopkIds.values], send_ptrs: TensorTuple[*SendPtrs.values], recv_ptrs: TensorTuple[*RecvPtrs.values], recv_count_ptrs: TensorTuple[*RecvCountPtrs.values], atomic_counters: TensorTuple[*Counters.values], dev_ctxs: DeviceContextArray)
Multi-device fused Expert Parallelism NVFP4 dispatch with dynamic global scales.
Launches the EP dispatch kernel on all devices simultaneously via
_ep_launch_device_collective. Each device routes its tokens to experts
based on top-k IDs, quantizes each token to NVFP4 against its own
global scale, 2688 / rowmax, and sends it to the appropriate peer
devices. The receivers store the scales' BF16 inverses in
output_rowwise_scales.
Parameters:
- OutTokens (
TypeList[values]): The element types ofoutput_tokens(inferred). - OutScales (
TypeList[values]): The element types ofoutput_scales(inferred). - OutRowwiseScales (
TypeList[values]): The element types ofoutput_rowwise_scales(inferred). - RowOffsets (
TypeList[values]): The element types ofrow_offsets(inferred). - ScalesOffsets (
TypeList[values]): The element types ofscales_offsets(inferred). - ExpertIds (
TypeList[values]): The element types ofexpert_ids(inferred). - SrcInfos (
TypeList[values]): The element types ofsrc_info(inferred). - InTokens (
TypeList[values]): The element types ofinput_tokens(inferred). - TopkIds (
TypeList[values]): The element types oftopk_ids(inferred). - SendPtrs (
TypeList[values]): The element types ofsend_ptrs(inferred). - RecvPtrs (
TypeList[values]): The element types ofrecv_ptrs(inferred). - RecvCountPtrs (
TypeList[values]): The element types ofrecv_count_ptrs(inferred). - Counters (
TypeList[values]): The element types ofatomic_counters(inferred). - hidden_size (
Int): Size of the model's hidden dimension (inferred). - top_k (
Int): Number of experts each token is routed to (inferred). - n_experts (
Int): Total number of experts across all GPUs (inferred). - max_token_per_rank (
Int): Maximum number of tokens per GPU (inferred). - n_gpus_per_node (
Int): Number of GPUs per node (inferred). - n_nodes (
Int): Number of physical nodes (inferred). - fused_shared_expert (
Bool): Whether a shared expert is fused into the dispatch kernel (inferred). - _trace_name (
StaticString): Trace label for this op.
Args:
- output_tokens (
TensorTuple[*OutTokens.values]): Output variadic tensors storing the dispatched NVFP4-quantized tokens, one per device. - output_scales (
TensorTuple[*OutScales.values]): Output variadic tensors storing the NVFP4 block scales, one per device. - output_rowwise_scales (
TensorTuple[*OutRowwiseScales.values]): Output variadic tensors storing each received token's BF16 inverse global scale, one per device. - row_offsets (
TensorTuple[*RowOffsets.values]): Output variadic tensors storing the row offsets for the received tokens, one per device. - scales_offsets (
TensorTuple[*ScalesOffsets.values]): Output variadic tensors storing the offsets into the scales buffer, one per device. - expert_ids (
TensorTuple[*ExpertIds.values]): Output variadic tensors storing the expert ID for each received token, one per device. - src_info (
TensorTuple[*SrcInfos.values]): Output variadic tensors recording the originating rank and token index for each received token, one per device. - input_tokens (
TensorTuple[*InTokens.values]): Input variadic tensors of tokens to dispatch, one per device. - topk_ids (
TensorTuple[*TopkIds.values]): Input variadic tensors of top-k expert IDs per token, one per device. - send_ptrs (
TensorTuple[*SendPtrs.values]): Send buffer pointers for the dispatch phase, one per device. - recv_ptrs (
TensorTuple[*RecvPtrs.values]): Receive buffer pointers for the dispatch phase, one per device. - recv_count_ptrs (
TensorTuple[*RecvCountPtrs.values]): Receive count buffer pointers tracking tokens received per expert, one per device. - atomic_counters (
TensorTuple[*Counters.values]): Atomic counters coordinating work across thread blocks, one per device. - dev_ctxs (
DeviceContextArray): List of GPU device contexts, one per device.