IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

moe_gather_quantize_mx_amd

def moe_gather_quantize_mx_amd[out_dtype: DType = .float8_e4m3fn, scales_dtype: DType = .float8_e8m0fnu, in_dtype: DType = .bfloat16, //, OutputEngine: TensorEngine, ScalesEngine: TensorEngine, InputEngine: TensorEngine, OrderEngine: TensorEngine, *, num_max_threads: Int = Int(512)](ctx: DeviceContext, output_tile: TileTensor[out_dtype, Engine=OutputEngine, linear_idx_type=output_tile.linear_idx_type], scales_tile: TileTensor[scales_dtype, Engine=ScalesEngine, linear_idx_type=scales_tile.linear_idx_type], input_tile: TileTensor[in_dtype, Engine=InputEngine, linear_idx_type=input_tile.linear_idx_type], order_tile: TileTensor[.uint32, Engine=OrderEngine, linear_idx_type=order_tile.linear_idx_type], topk: Int)

Gathers each sorted MoE row and quantizes it to MXFP8 in one launch.

The wide-row fallback inside the sort-fused MoE op -- rows past the source-scatter kernel's one-CTA-per-token width -- and the byte-identity reference the kernel tests compare against: sorted row p reads source token row token_expert_order[p] // topk and quantizes it exactly as quantize_mx_amd would the gathered copy, so the two produce byte-identical payloads and scales.

Parameters:

  • ​out_dtype (DType): Element type of the quantized output tensor (float8_e4m3fn, inferred).
  • ​scales_dtype (DType): Element type of the block scale-factor tensor (float8_e8m0fnu, inferred).
  • ​in_dtype (DType): Element type of the input activation tensor (bfloat16, inferred). A float32 tensor -- the split-K reduce's output, read without the separate bf16 cast -- rounds through bfloat16 before the quantize, so it produces the same payload and scales.
  • ​OutputEngine (TensorEngine): Engine policy of the output_tile tensor.
  • ​ScalesEngine (TensorEngine): Engine policy of the scales_tile tensor.
  • ​InputEngine (TensorEngine): Engine policy of the input_tile tensor.
  • ​OrderEngine (TensorEngine): Engine policy of the order_tile tensor.
  • ​num_max_threads (Int): Maximum threads per block for the launch grid (defaults to 512), rounded down to whole scale-block lane groups.

Args:

Was this page helpful?