IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

Struct_ep_dispatch_fp8

struct Struct_ep_dispatch_fp8

Registers the ep.dispatch.fp8 graph op with the graph compiler.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[hidden_size: Int, top_k: Int, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, dispatch_scale_granularity: StaticString, fused_shared_expert: Bool, skip_a2a: Bool, allreduce_world_size: Int, //](output_tokens: T, output_scales: T, row_offsets: T, expert_ids: T, src_info: T, atomic_counters: T, input_tokens: T, topk_ids: T, send_ptrs: T, recv_ptrs: T, recv_count_ptrs: T, context: DeviceContext)

Execute the fused Expert Parallelism FP8 dispatch kernel. Tokens are dispatched in Blockwise FP8 format.

run​

static def run[hidden_size: Int, top_k: Int, n_experts: Int, max_token_per_rank: Int, n_gpus_per_node: Int, n_nodes: Int, dispatch_scale_granularity: StaticString, fused_shared_expert: Bool, skip_a2a: Bool, allreduce_world_size: Int, ExpertLoadT: ExpertLoadSink = NullExpertLoad](output_tokens: T, output_scales: T, row_offsets: T, expert_ids: T, src_info: T, atomic_counters: T, input_tokens: T, topk_ids: T, send_ptrs: T, recv_ptrs: T, recv_count_ptrs: T, context: DeviceContext, expert_load: ExpertLoadT)

Runs ep.dispatch.fp8, reporting each local expert's routed-token count to expert_load.

Was this page helpful?