For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
moe_gather_quantize_mx_amd
def moe_gather_quantize_mx_amd[out_dtype: DType = .float8_e4m3fn, scales_dtype: DType = .float8_e8m0fnu, in_dtype: DType = .bfloat16, //, OutputEngine: TensorEngine, ScalesEngine: TensorEngine, InputEngine: TensorEngine, OrderEngine: TensorEngine, *, num_max_threads: Int = Int(512)](ctx: DeviceContext, output_tile: TileTensor[out_dtype, Engine=OutputEngine, linear_idx_type=output_tile.linear_idx_type], scales_tile: TileTensor[scales_dtype, Engine=ScalesEngine, linear_idx_type=scales_tile.linear_idx_type], input_tile: TileTensor[in_dtype, Engine=InputEngine, linear_idx_type=input_tile.linear_idx_type], order_tile: TileTensor[.uint32, Engine=OrderEngine, linear_idx_type=order_tile.linear_idx_type], topk: Int)
Gathers each sorted MoE row and quantizes it to MXFP8 in one launch.
The wide-row fallback inside the sort-fused MoE op -- rows past the
source-scatter kernel's one-CTA-per-token width -- and the
byte-identity reference the kernel tests compare against: sorted row
p reads source token row
token_expert_order[p] // topk and quantizes it exactly as
quantize_mx_amd would the gathered copy, so the two produce
byte-identical payloads and scales.
Parameters:
- out_dtype (
DType): Element type of the quantized output tensor (float8_e4m3fn, inferred). - scales_dtype (
DType): Element type of the block scale-factor tensor (float8_e8m0fnu, inferred). - in_dtype (
DType): Element type of the input activation tensor (bfloat16, inferred). Afloat32tensor -- the split-K reduce's output, read without the separate bf16 cast -- rounds through bfloat16 before the quantize, so it produces the same payload and scales. - OutputEngine (
TensorEngine): Engine policy of theoutput_tiletensor. - ScalesEngine (
TensorEngine): Engine policy of thescales_tiletensor. - InputEngine (
TensorEngine): Engine policy of theinput_tiletensor. - OrderEngine (
TensorEngine): Engine policy of theorder_tiletensor. - num_max_threads (
Int): Maximum threads per block for the launch grid (defaults to 512), rounded down to whole scale-block lane groups.
Args:
- ctx (
DeviceContext): Device context. - output_tile (
TileTensor[out_dtype, Engine=OutputEngine, linear_idx_type=output_tile.linear_idx_type]): Output[P, K]float8_e4m3fn, one row per sorted slot. - scales_tile (
TileTensor[scales_dtype, Engine=ScalesEngine, linear_idx_type=scales_tile.linear_idx_type]): Output[P, K // 32]float8_e8m0fnu block scales, row-major (thepreshuffled_b=Falselayout the CDNA4 grouped GEMM reads). - input_tile (
TileTensor[in_dtype, Engine=InputEngine, linear_idx_type=input_tile.linear_idx_type]): Input[M, K]bfloat16 activations (or thefloat32they were cast from), one row per token. - order_tile (
TileTensor[.uint32, Engine=OrderEngine, linear_idx_type=order_tile.linear_idx_type]):moe_create_indices'stoken_expert_order,[P]uint32 withP == M * topk. Every entry must name a slot belowP:moe_create_indicesfiles each valid pair exactly once, and a slot it files nowhere keeps its unwritten value, which the kernel skips rather than reading aninputrow past the last. - topk (
Int): Experts per token; the source row is the order entry divided by it.