IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

Struct_ep_dispatch_mxfp4

struct Struct_ep_dispatch_mxfp4

Registers the ep.dispatch.mxfp4 graph op with the graph compiler.

Implemented traitsโ€‹

AnyType, Deinitable, Movable

Methodsโ€‹

executeโ€‹

static def execute[input_dtype: DType, dispatch_dtype: DType, dispatch_scale_dtype: DType, hidden_size: Int, top_k: Int, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, fused_shared_expert: Bool, skip_a2a: Bool, allreduce_world_size: Int, //, target: StringSpan[ImmStaticOrigin], *, MX_FORMAT: StringSpan[ImmStaticOrigin] = StringSpan("auto")](output_tokens: ManagedTensorSlice[IOSpec[_, _].Output, static_spec=output_tokens.static_spec], output_scales: ManagedTensorSlice[IOSpec[_, _].Output, static_spec=output_scales.static_spec], row_offsets: ManagedTensorSlice[IOSpec[_, _].Output, static_spec=row_offsets.static_spec], expert_ids: ManagedTensorSlice[IOSpec[_, _].Output, static_spec=expert_ids.static_spec], src_info: ManagedTensorSlice[IOSpec[_, _].Output, static_spec=src_info.static_spec], atomic_counters: ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=atomic_counters.static_spec], input_tokens: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=input_tokens.static_spec], topk_ids: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=topk_ids.static_spec], send_ptrs: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=send_ptrs.static_spec], recv_ptrs: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=recv_ptrs.static_spec], recv_count_ptrs: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=recv_count_ptrs.static_spec], context: DeviceContextBase)

Execute the fused Expert Parallelism MXFP4 dispatch kernel. Tokens are dispatched in MXFP4 format.

The wait-side copy writes the E8M0 activation scale directly into the up-proj grouped matmul's per-expert fixed-stride scale_4d slot layout (KS224 up-proj fusion). A slot holds max_padded_M rows, the most tokens one expert can receive rounded up to 32, so the scales output tensor has shape [n_local_experts * max_padded_M, K_SCALES].

Was this page helpful?