IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

Struct_ep_combine_async_mx

struct Struct_ep_combine_async_mx

Registers the ep.combine_async.mx graph op with the graph compiler.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[combine_dtype: DType, hidden_size: Int, top_k: Int, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, //, *, mx_min_tokens: Int](atomic_counters: T, input_tokens: T, src_info: T, send_ptrs: T, recv_ptrs: T, recv_count_ptrs: T, total_tokens: UInt32, context: DeviceContext)

ep.combine_async that sends MXFP8 messages for large steps.

When total_tokens reaches mx_min_tokens, each expert output goes back to its rank as MXFP8 (E4M3 values with an E8M0 scale per 32 elements, about half the bytes), which is lossy; smaller steps send it exactly. total_tokens must be the same on every rank, and the matching ep.combine_wait.mx must get the same value and threshold.

Parameters:

  • ​combine_dtype (DType): Expert output dtype (inferred).
  • ​hidden_size (Int): Model hidden size (inferred).
  • ​top_k (Int): Experts each token is routed to (inferred).
  • ​n_experts (Int): Experts across all GPUs (inferred).
  • ​max_token_per_rank (Int): Most tokens per GPU (inferred).
  • ​n_gpus_per_node (Int): GPUs per node (inferred).
  • ​n_nodes (Int): Physical nodes (inferred).
  • ​mx_min_tokens (Int): Smallest total_tokens sent as MXFP8.

Args:

  • ​atomic_counters (T): Atomic counters coordinating the combine.
  • ​input_tokens (T): Expert outputs. Shape [num_tokens, hidden_size].
  • ​src_info (T): Each row's originating rank and token from the dispatch.
  • ​send_ptrs (T): Send buffer pointers for the combine phase.
  • ​recv_ptrs (T): Receive buffer pointers for the combine phase.
  • ​recv_count_ptrs (T): Receive count buffer pointers.
  • ​total_tokens (UInt32): Tokens of the step across all ranks.
  • ​context (DeviceContext): GPU device context for the current device.

Was this page helpful?