For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
swiglu_quantize_mx_amd
def swiglu_quantize_mx_amd[out_dtype: DType = .float8_e4m3fn, scales_dtype: DType = .float8_e8m0fnu, in_dtype: DType = .bfloat16, //, OutputEngine: TensorEngine, ScalesEngine: TensorEngine, GateUpEngine: TensorEngine, *, num_max_threads: Int = Int(512), activation: def(Float32, Float32) thin -> Float32](ctx: DeviceContext, output_tile: TileTensor[out_dtype, Engine=OutputEngine, linear_idx_type=output_tile.linear_idx_type], scales_tile: TileTensor[scales_dtype, Engine=ScalesEngine, linear_idx_type=scales_tile.linear_idx_type], gate_up_tile: TileTensor[in_dtype, Engine=GateUpEngine, linear_idx_type=gate_up_tile.linear_idx_type])
Quantizes the gated activation of a BF16 [gate | up] tensor to MXFP4 or MXFP8 on AMD CDNA4 (MI355X).
One kernel computes what applying activation to each (gate, up)
pair, rounding to bfloat16, and then quantize_mx_amd would,
without writing the activation out. Produces the same 2D E8M0 block
scales as the ungated kernel: [M, N // 32] for a [M, 2 * N]
input, whose launch geometry is computed on N.
Parameters:
- out_dtype (
DType): Element type of the quantized output (inferred).uint8for packed MXFP4 orfloat8_e4m3fnfor MXFP8. - scales_dtype (
DType): Element type of the block scale-factor tensor (inferred). Must befloat8_e8m0fnu. - in_dtype (
DType): Element type of the input (inferred). Must bebfloat16. - OutputEngine (
TensorEngine): Engine policy of theoutput_tiletensor. - ScalesEngine (
TensorEngine): Engine policy of thescales_tiletensor. - GateUpEngine (
TensorEngine): Engine policy of thegate_up_tiletensor. - num_max_threads (
Int): Maximum number of threads per block for the launch grid (defaults to 512). Must be at least 4, and is rounded down to a whole number of scale-block lane groups so every block stays inside one group. - activation (
def(Float32, Float32) thin -> Float32): The gated activation, applied to each(gate, up)pair.
Args:
- ctx (
DeviceContext): Device context. - output_tile (
TileTensor[out_dtype, Engine=OutputEngine, linear_idx_type=output_tile.linear_idx_type]): Output [M, N] float8_e4m3fn, or [M, N // 2] uint8 for packed MXFP4. - scales_tile (
TileTensor[scales_dtype, Engine=ScalesEngine, linear_idx_type=scales_tile.linear_idx_type]): Output [M, N // 32] float8_e8m0fnu (block scales). - gate_up_tile (
TileTensor[in_dtype, Engine=GateUpEngine, linear_idx_type=gate_up_tile.linear_idx_type]): Input [M, 2 * N] bfloat16,gatethenup.
Raises:
Error: If output_tile is not [M, N] (MXFP8) or [M, N // 2]
(MXFP4) or scales_tile is not [M, N // 32] for a
[M, 2 * N] gate_up_tile.