For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /max/get-started.md).

Mojo function

quantize_dynamic_scaled_fp4fp8_kernel

def quantize_dynamic_scaled_fp4fp8_kernel[out_dtype: DType, scales_dtype: DType, in_dtype: DType, output_layout: Layout, scales_layout: Layout, input_layout: Layout, *, SF_VECTOR_SIZE: Int = Int(16), ELEMENTS_PER_THREAD: Int = Int(8), num_max_threads: Int = Int(512)](output: LayoutTensor[out_dtype, output_layout, MutAnyOrigin], scales: LayoutTensor[scales_dtype, scales_layout, MutAnyOrigin], input: LayoutTensor[in_dtype, input_layout, ImmutAnyOrigin], num_cols: Int, num_cols_padded: Int, tensor_sf: Float32)

GPU kernel that quantizes BF16 elements to NVFP4/MXFP4 or MXFP8 with per-block scale factors.

Each thread loads 8 elements, cooperatively reduces the per-group maximum across warp lanes via shuffle, derives the scale factor, and writes the quantized output and scales. Padded rows and columns are zeroed out to satisfy tensor-core scale-factor expectations.

Parameters:

out_dtype (DType): Element type of the quantized output tensor. Must be uint8 for NVFP4 or MXFP4, or float8_e4m3fn for MXFP8.
scales_dtype (DType): Element type of the block scale-factor tensor. Must be float8_e4m3fn for NVFP4 or float8_e8m0fnu for MXFP4 and MXFP8.
in_dtype (DType): Element type of the input activation tensor. Must be bfloat16.
output_layout (Layout): Memory layout of the quantized output tensor.
scales_layout (Layout): Memory layout of the block scale-factor tensor.
input_layout (Layout): Memory layout of the input activation tensor.
SF_VECTOR_SIZE (Int): Number of input elements covered by each block scale factor: 16 for NVFP4 or MXFP4, 32 for MXFP8 (defaults to 16).
ELEMENTS_PER_THREAD (Int): Number of input elements each thread loads and quantizes per iteration (defaults to 8).
num_max_threads (Int): Maximum number of threads per block for the launch grid (defaults to 512).