IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

moe_source_scatter_quantize_mx_amd

def moe_source_scatter_quantize_mx_amd[num_cols: Int, out_dtype: DType = .float8_e4m3fn, scales_dtype: DType = .float8_e8m0fnu, in_dtype: DType = .bfloat16, //, OutputEngine: TensorEngine, ScalesEngine: TensorEngine, InputEngine: TensorEngine, RestoreEngine: TensorEngine](ctx: DeviceContext, output_tile: TileTensor[out_dtype, Engine=OutputEngine, linear_idx_type=output_tile.linear_idx_type], scales_tile: TileTensor[scales_dtype, Engine=ScalesEngine, linear_idx_type=scales_tile.linear_idx_type], input_tile: TileTensor[in_dtype, Engine=InputEngine, linear_idx_type=input_tile.linear_idx_type], restore_tile: TileTensor[.uint32, Engine=RestoreEngine, linear_idx_type=restore_tile.linear_idx_type], topk: Int)

Scatters each token's MXFP8-quantized row to its sorted slots.

The source-major replacement for the moe_gather_quantize_mx_amd pass the K3 amd_tp_w4a8 path runs after moe_create_indices at large fan-outs: one CTA per source token row quantizes the row once and writes it at every sorted position restore_token_order gives its pairs, so the payload and scale bytes per sorted row are identical to the sorted-row kernel's -- the same source row, MX block partition and E8M0 rounding -- while the bf16 source is read once per token instead of once per sorted row.

Parameters:

  • ​num_cols (Int): Activation row width K; one CTA is K // 8 threads covering a whole row.
  • ​out_dtype (DType): Element type of the quantized output tensor (float8_e4m3fn, inferred).
  • ​scales_dtype (DType): Element type of the block scale-factor tensor (float8_e8m0fnu, inferred).
  • ​in_dtype (DType): Element type of the input activation tensor (bfloat16, inferred). A float32 tensor -- the split-K reduce's output, read without the separate bf16 cast -- rounds through bfloat16 before the quantize, so it produces the same payload and scales.
  • ​OutputEngine (TensorEngine): Engine policy of the output_tile tensor.
  • ​ScalesEngine (TensorEngine): Engine policy of the scales_tile tensor.
  • ​InputEngine (TensorEngine): Engine policy of the input_tile tensor.
  • ​RestoreEngine (TensorEngine): Engine policy of the restore_tile tensor.

Args:

Raises:

Error: If any tensor's shape or stride disagrees with the contract, or the launch falls outside moe_source_scatter_quantize_supported's envelope.

Was this page helpful?