IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

DistributedEPDispatchFP8

struct DistributedEPDispatchFP8

Registers the mo.distributed.ep.dispatch.fp8 graph op with the graph compiler.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[OutTokens: TypeList[OutTokens.values], OutScales: TypeList[OutScales.values], RowOffsets: TypeList[RowOffsets.values], ExpertIds: TypeList[ExpertIds.values], SrcInfos: TypeList[SrcInfos.values], InTokens: TypeList[InTokens.values], TopkIds: TypeList[TopkIds.values], SendPtrs: TypeList[SendPtrs.values], RecvPtrs: TypeList[RecvPtrs.values], RecvCountPtrs: TypeList[RecvCountPtrs.values], Counters: TypeList[Counters.values], hidden_size: Int, top_k: Int, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, dispatch_scale_granularity: StaticString, fused_shared_expert: Bool, //, _trace_name: StaticString](output_tokens: TensorTuple[*OutTokens.values], output_scales: TensorTuple[*OutScales.values], row_offsets: TensorTuple[*RowOffsets.values], expert_ids: TensorTuple[*ExpertIds.values], src_info: TensorTuple[*SrcInfos.values], input_tokens: TensorTuple[*InTokens.values], topk_ids: TensorTuple[*TopkIds.values], send_ptrs: TensorTuple[*SendPtrs.values], recv_ptrs: TensorTuple[*RecvPtrs.values], recv_count_ptrs: TensorTuple[*RecvCountPtrs.values], atomic_counters: TensorTuple[*Counters.values], dev_ctxs: DeviceContextArray)

Multi-device fused Expert Parallelism FP8 dispatch.

Launches the EP dispatch kernel on all devices simultaneously via _launch_device_collective. Each device routes its tokens to experts based on top-k IDs, quantizes them to Blockwise FP8 format, and sends them to peer devices.

Parameters:

  • ​OutTokens (TypeList[values]): The element types of output_tokens (inferred).
  • ​OutScales (TypeList[values]): The element types of output_scales (inferred).
  • ​RowOffsets (TypeList[values]): The element types of row_offsets (inferred).
  • ​ExpertIds (TypeList[values]): The element types of expert_ids (inferred).
  • ​SrcInfos (TypeList[values]): The element types of src_info (inferred).
  • ​InTokens (TypeList[values]): The element types of input_tokens (inferred).
  • ​TopkIds (TypeList[values]): The element types of topk_ids (inferred).
  • ​SendPtrs (TypeList[values]): The element types of send_ptrs (inferred).
  • ​RecvPtrs (TypeList[values]): The element types of recv_ptrs (inferred).
  • ​RecvCountPtrs (TypeList[values]): The element types of recv_count_ptrs (inferred).
  • ​Counters (TypeList[values]): The element types of atomic_counters (inferred).
  • ​hidden_size (Int): Size of the model's hidden dimension (inferred).
  • ​top_k (Int): Number of experts each token is routed to (inferred).
  • ​n_experts (Int): Total number of experts across all GPUs (inferred).
  • ​max_token_per_rank (Int): Maximum number of tokens per GPU (inferred).
  • ​n_gpus_per_node (Int): Number of GPUs per node (inferred).
  • ​n_nodes (Int): Number of physical nodes (inferred).
  • ​dispatch_scale_granularity (StaticString): Block size used for the blockwise FP8 quantization scales (inferred).
  • ​fused_shared_expert (Bool): Whether a shared expert is fused into the dispatch kernel (inferred).
  • ​_trace_name (StaticString): Trace label for this op.

Args:

Was this page helpful?