IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

DistributedEPDispatchBlockScaledNV

struct DistributedEPDispatchBlockScaledNV

Registers the mo.distributed.ep.dispatch.block.scaled.nv graph op with the graph compiler.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[OutTokens: TypeList[OutTokens.values], OutScales: TypeList[OutScales.values], RowOffsets: TypeList[RowOffsets.values], ScalesOffsets: TypeList[ScalesOffsets.values], ExpertIds: TypeList[ExpertIds.values], SrcInfos: TypeList[SrcInfos.values], InTokens: TypeList[InTokens.values], TopkIds: TypeList[TopkIds.values], SendPtrs: TypeList[SendPtrs.values], RecvPtrs: TypeList[RecvPtrs.values], RecvCountPtrs: TypeList[RecvCountPtrs.values], InScales: TypeList[InScales.values], Counters: TypeList[Counters.values], hidden_size: Int, top_k: Int, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, fused_shared_expert: Bool, //, _trace_name: StaticString](output_tokens: TensorTuple[*OutTokens.values], output_scales: TensorTuple[*OutScales.values], row_offsets: TensorTuple[*RowOffsets.values], scales_offsets: TensorTuple[*ScalesOffsets.values], expert_ids: TensorTuple[*ExpertIds.values], src_info: TensorTuple[*SrcInfos.values], input_tokens: TensorTuple[*InTokens.values], topk_ids: TensorTuple[*TopkIds.values], send_ptrs: TensorTuple[*SendPtrs.values], recv_ptrs: TensorTuple[*RecvPtrs.values], recv_count_ptrs: TensorTuple[*RecvCountPtrs.values], input_scales: TensorTuple[*InScales.values], atomic_counters: TensorTuple[*Counters.values], dev_ctxs: DeviceContextArray)

Multi-device fused Expert Parallelism NVFP4 dispatch.

Launches the EP dispatch kernel on all devices simultaneously via _ep_launch_device_collective. Each device routes its tokens to experts based on top-k IDs, quantizes them to NVFP4 format, and sends them to the appropriate peer devices.

Parameters:

  • ​OutTokens (TypeList[values]): The element types of output_tokens (inferred).
  • ​OutScales (TypeList[values]): The element types of output_scales (inferred).
  • ​RowOffsets (TypeList[values]): The element types of row_offsets (inferred).
  • ​ScalesOffsets (TypeList[values]): The element types of scales_offsets (inferred).
  • ​ExpertIds (TypeList[values]): The element types of expert_ids (inferred).
  • ​SrcInfos (TypeList[values]): The element types of src_info (inferred).
  • ​InTokens (TypeList[values]): The element types of input_tokens (inferred).
  • ​TopkIds (TypeList[values]): The element types of topk_ids (inferred).
  • ​SendPtrs (TypeList[values]): The element types of send_ptrs (inferred).
  • ​RecvPtrs (TypeList[values]): The element types of recv_ptrs (inferred).
  • ​RecvCountPtrs (TypeList[values]): The element types of recv_count_ptrs (inferred).
  • ​InScales (TypeList[values]): The element types of input_scales (inferred).
  • ​Counters (TypeList[values]): The element types of atomic_counters (inferred).
  • ​hidden_size (Int): Size of the model's hidden dimension (inferred).
  • ​top_k (Int): Number of experts each token is routed to (inferred).
  • ​n_experts (Int): Total number of experts across all GPUs (inferred).
  • ​max_token_per_rank (Int): Maximum number of tokens per GPU (inferred).
  • ​n_gpus_per_node (Int): Number of GPUs per node (inferred).
  • ​n_nodes (Int): Number of physical nodes (inferred).
  • ​fused_shared_expert (Bool): Whether a shared expert is fused into the dispatch kernel (inferred).
  • ​_trace_name (StaticString): Trace label for this op.

Args:

Was this page helpful?