IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

Struct_ep_dispatch_wait_mxfp4

struct Struct_ep_dispatch_wait_mxfp4

Registers the ep.dispatch_wait.mxfp4 graph op with the graph compiler.

Implemented traitsโ€‹

AnyType, Deinitable, Movable

Methodsโ€‹

executeโ€‹

static def execute[dispatch_dtype: DType, dispatch_scale_dtype: DType, hidden_size: Int, top_k: Int, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, //, target: StringSpan[ImmStaticOrigin], num_input_tokens: Int = Int(-1), *, MX_FORMAT: StringSpan[ImmStaticOrigin] = StringSpan("auto")](output_tokens: ManagedTensorSlice[IOSpec[_, _].Output, static_spec=output_tokens.static_spec], output_scales: ManagedTensorSlice[IOSpec[_, _].Output, static_spec=output_scales.static_spec], row_offsets: ManagedTensorSlice[IOSpec[_, _].Output, static_spec=row_offsets.static_spec], expert_ids: ManagedTensorSlice[IOSpec[_, _].Output, static_spec=expert_ids.static_spec], src_info: ManagedTensorSlice[IOSpec[_, _].Output, static_spec=src_info.static_spec], atomic_counters: ManagedTensorSlice[IOSpec[_, _].MutableInput, static_spec=atomic_counters.static_spec], recv_ptrs: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=recv_ptrs.static_spec], recv_count_ptrs: ManagedTensorSlice[IOSpec[_, _].Input, static_spec=recv_count_ptrs.static_spec], context: DeviceContextBase)

Execute the Expert Parallelism dispatch completion kernel. Received tokens are in MXFP4 format: two FP4 elements packed per uint8 in output_tokens with per-token even-mode scales in output_scales.

The kernel writes the E8M0 activation scale directly into the up-proj grouped matmul's per-expert fixed-stride scale_4d slot layout (KS224 up-proj fusion). A slot holds max_padded_M rows, the most tokens one expert can receive rounded up to 32, so the scales output tensor has shape [n_local_experts * max_padded_M, K_SCALES].

Was this page helpful?