For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
moe_source_scatter_quantize_mx_amd
def moe_source_scatter_quantize_mx_amd[num_cols: Int, out_dtype: DType = .float8_e4m3fn, scales_dtype: DType = .float8_e8m0fnu, in_dtype: DType = .bfloat16, //, OutputEngine: TensorEngine, ScalesEngine: TensorEngine, InputEngine: TensorEngine, RestoreEngine: TensorEngine](ctx: DeviceContext, output_tile: TileTensor[out_dtype, Engine=OutputEngine, linear_idx_type=output_tile.linear_idx_type], scales_tile: TileTensor[scales_dtype, Engine=ScalesEngine, linear_idx_type=scales_tile.linear_idx_type], input_tile: TileTensor[in_dtype, Engine=InputEngine, linear_idx_type=input_tile.linear_idx_type], restore_tile: TileTensor[.uint32, Engine=RestoreEngine, linear_idx_type=restore_tile.linear_idx_type], topk: Int)
Scatters each token's MXFP8-quantized row to its sorted slots.
The source-major replacement for the moe_gather_quantize_mx_amd pass
the K3 amd_tp_w4a8 path runs after moe_create_indices at large
fan-outs: one CTA per source token row quantizes the row once and
writes it at every sorted position restore_token_order gives its
pairs, so the payload and scale bytes per sorted row are identical to
the sorted-row kernel's -- the same source row, MX block partition and
E8M0 rounding -- while the bf16 source is read once per token instead
of once per sorted row.
Parameters:
- num_cols (
Int): Activation row widthK; one CTA isK // 8threads covering a whole row. - out_dtype (
DType): Element type of the quantized output tensor (float8_e4m3fn, inferred). - scales_dtype (
DType): Element type of the block scale-factor tensor (float8_e8m0fnu, inferred). - in_dtype (
DType): Element type of the input activation tensor (bfloat16, inferred). Afloat32tensor -- the split-K reduce's output, read without the separate bf16 cast -- rounds through bfloat16 before the quantize, so it produces the same payload and scales. - OutputEngine (
TensorEngine): Engine policy of theoutput_tiletensor. - ScalesEngine (
TensorEngine): Engine policy of thescales_tiletensor. - InputEngine (
TensorEngine): Engine policy of theinput_tiletensor. - RestoreEngine (
TensorEngine): Engine policy of therestore_tiletensor.
Args:
- ctx (
DeviceContext): Device context. - output_tile (
TileTensor[out_dtype, Engine=OutputEngine, linear_idx_type=output_tile.linear_idx_type]): Output[P, K]float8_e4m3fn, one row per sorted slot. - scales_tile (
TileTensor[scales_dtype, Engine=ScalesEngine, linear_idx_type=scales_tile.linear_idx_type]): Output[P, K // 32]float8_e8m0fnu block scales, row-major (thepreshuffled_b=Falselayout the CDNA4 grouped GEMM reads). - input_tile (
TileTensor[in_dtype, Engine=InputEngine, linear_idx_type=input_tile.linear_idx_type]): Input[M, K]bfloat16 activations (or thefloat32they were cast from), one row per token. - restore_tile (
TileTensor[.uint32, Engine=RestoreEngine, linear_idx_type=restore_tile.linear_idx_type]):moe_create_indices'srestore_token_order,[P]uint32 withP == M * topk. A pair whose expert id was out of range holds thePsentinel and is skipped. - topk (
Int): Experts per token; at least 1.
Raises:
Error: If any tensor's shape or stride disagrees with the
contract, or the launch falls outside
moe_source_scatter_quantize_supported's envelope.