IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

DistributedEPDispatchBlockScaledNVDynGlobalScales

struct DistributedEPDispatchBlockScaledNVDynGlobalScales

Registers the mo.distributed.ep.dispatch.block.scaled.nv.dyn_global_scales graph op with the graph compiler.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[OutTokens: TypeList[OutTokens.values], OutScales: TypeList[OutScales.values], OutRowwiseScales: TypeList[OutRowwiseScales.values], RowOffsets: TypeList[RowOffsets.values], ScalesOffsets: TypeList[ScalesOffsets.values], ExpertIds: TypeList[ExpertIds.values], SrcInfos: TypeList[SrcInfos.values], InTokens: TypeList[InTokens.values], TopkIds: TypeList[TopkIds.values], SendPtrs: TypeList[SendPtrs.values], RecvPtrs: TypeList[RecvPtrs.values], RecvCountPtrs: TypeList[RecvCountPtrs.values], Counters: TypeList[Counters.values], hidden_size: Int, top_k: Int, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, fused_shared_expert: Bool, //, _trace_name: StaticString](output_tokens: TensorTuple[*OutTokens.values], output_scales: TensorTuple[*OutScales.values], output_rowwise_scales: TensorTuple[*OutRowwiseScales.values], row_offsets: TensorTuple[*RowOffsets.values], scales_offsets: TensorTuple[*ScalesOffsets.values], expert_ids: TensorTuple[*ExpertIds.values], src_info: TensorTuple[*SrcInfos.values], input_tokens: TensorTuple[*InTokens.values], topk_ids: TensorTuple[*TopkIds.values], send_ptrs: TensorTuple[*SendPtrs.values], recv_ptrs: TensorTuple[*RecvPtrs.values], recv_count_ptrs: TensorTuple[*RecvCountPtrs.values], atomic_counters: TensorTuple[*Counters.values], dev_ctxs: DeviceContextArray)

Multi-device fused Expert Parallelism NVFP4 dispatch with dynamic global scales.

Launches the EP dispatch kernel on all devices simultaneously via _ep_launch_device_collective. Each device routes its tokens to experts based on top-k IDs, quantizes each token to NVFP4 against its own global scale, 2688 / rowmax, and sends it to the appropriate peer devices. The receivers store the scales' BF16 inverses in output_rowwise_scales.

Parameters:

  • ​OutTokens (TypeList[values]): The element types of output_tokens (inferred).
  • ​OutScales (TypeList[values]): The element types of output_scales (inferred).
  • ​OutRowwiseScales (TypeList[values]): The element types of output_rowwise_scales (inferred).
  • ​RowOffsets (TypeList[values]): The element types of row_offsets (inferred).
  • ​ScalesOffsets (TypeList[values]): The element types of scales_offsets (inferred).
  • ​ExpertIds (TypeList[values]): The element types of expert_ids (inferred).
  • ​SrcInfos (TypeList[values]): The element types of src_info (inferred).
  • ​InTokens (TypeList[values]): The element types of input_tokens (inferred).
  • ​TopkIds (TypeList[values]): The element types of topk_ids (inferred).
  • ​SendPtrs (TypeList[values]): The element types of send_ptrs (inferred).
  • ​RecvPtrs (TypeList[values]): The element types of recv_ptrs (inferred).
  • ​RecvCountPtrs (TypeList[values]): The element types of recv_count_ptrs (inferred).
  • ​Counters (TypeList[values]): The element types of atomic_counters (inferred).
  • ​hidden_size (Int): Size of the model's hidden dimension (inferred).
  • ​top_k (Int): Number of experts each token is routed to (inferred).
  • ​n_experts (Int): Total number of experts across all GPUs (inferred).
  • ​max_token_per_rank (Int): Maximum number of tokens per GPU (inferred).
  • ​n_gpus_per_node (Int): Number of GPUs per node (inferred).
  • ​n_nodes (Int): Number of physical nodes (inferred).
  • ​fused_shared_expert (Bool): Whether a shared expert is fused into the dispatch kernel (inferred).
  • ​_trace_name (StaticString): Trace label for this op.

Args:

Was this page helpful?