For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
grouped_quantize_nvfp4_row_amax_kernel
def grouped_quantize_nvfp4_row_amax_kernel[output_dtype: DType, scales_dtype: DType, input_dtype: DType, output_layout: TensorLayout, scales_layout: TensorLayout, row_scales_layout: TensorLayout, input_layout: TensorLayout, row_offsets_layout: TensorLayout, scales_offsets_layout: TensorLayout, OutputEngine: TensorEngine, ScalesEngine: TensorEngine, RowScalesEngine: TensorEngine, InputEngine: TensorEngine, RowOffsetsEngine: TensorEngine, ScalesOffsetsEngine: TensorEngine, input_fn: Optional[def[dtype: DType, width: SIMDLength](SIMD[dtype, width]) capturing thin -> SIMD[dtype, width]], row_indices_layout: TensorLayout = Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], RowIndicesEngine: TensorEngine = DefaultEngine, row_indices_dtype: DType = .uint32, row_index_divisor: Int = Int(1), rows_per_block: Int = Int(4)](output_tensor: TileTensor[output_dtype, output_layout, MutAnyOrigin, Engine=OutputEngine], scales_tensor: TileTensor[scales_dtype, scales_layout, MutAnyOrigin, Engine=ScalesEngine], row_scales: TileTensor[.bfloat16, row_scales_layout, MutAnyOrigin, Engine=RowScalesEngine], input_tensor: TileTensor[input_dtype, input_layout, ImmutAnyOrigin, Engine=InputEngine], row_offsets: TileTensor[.uint32, row_offsets_layout, ImmutAnyOrigin, Engine=RowOffsetsEngine], scales_offsets: TileTensor[.uint32, scales_offsets_layout, ImmutAnyOrigin, Engine=ScalesOffsetsEngine], row_indices: OptionalReg[TileTensor[row_indices_dtype, row_indices_layout, ImmutAnyOrigin, Engine=RowIndicesEngine]] = None)
Quantizes BF16 rows to NVFP4, each with its own global scale.
This is the fusion of the graph-op chain
amax = max(abs(f32(x)), axis=-1)
row_scale = bf16(max(amax / 2688, 1e-30))
scaled = bf16(f32(x) * (1 / f32(row_scale)))
grouped_quantize_dynamic_scaled_fp4(scaled, sf_tensor=1)with each rounding to BF16 kept, so that the payload and the block scales
match the chain bit for bit. A warp owns a row: it loads the row into
registers with 16-byte loads, reduces the row's amax, derives
row_scale, and quantizes the row in 16-element blocks from those
registers, so the row is read from memory once. Rows wider than the
register cache are read a second time for the quantize.
The grid scales with the rows rather than the scale tiles: the first
ceildiv(rows, rows_per_block) blocks give a warp to each output row,
which finds its expert, and so its scale tile, from the expert offsets
in one round of loads. One more block per allocated scale tile clears
the scale slots of that tile's padding rows, which must be zero. No two
warps write the same byte, and the scale stores go through
scales_layout.
Parameters:
- output_dtype (
DType): Element type of the packed output (uint8). - scales_dtype (
DType): Element type of the block scales (float8_e4m3fn). - input_dtype (
DType): Element type of the input (bfloat16). - output_layout (
TensorLayout): Layout ofoutput_tensor. - scales_layout (
TensorLayout): Layout ofscales_tensor. - row_scales_layout (
TensorLayout): Layout ofrow_scales. - input_layout (
TensorLayout): Layout ofinput_tensor. - row_offsets_layout (
TensorLayout): Layout ofrow_offsets. - scales_offsets_layout (
TensorLayout): Layout ofscales_offsets. - OutputEngine (
TensorEngine): Engine policy of theoutput_tensor. - ScalesEngine (
TensorEngine): Engine policy of thescales_tensor. - RowScalesEngine (
TensorEngine): Engine policy ofrow_scales. - InputEngine (
TensorEngine): Engine policy of theinput_tensor. - RowOffsetsEngine (
TensorEngine): Engine policy of therow_offsetstensor. - ScalesOffsetsEngine (
TensorEngine): Engine policy of thescales_offsetstensor. - input_fn (
Optional[def[dtype: DType, width: SIMDLength](SIMD[dtype, width]) capturing thin -> SIMD[dtype, width]]): Optional transform quantized in place of each input vector, such as relu2. - row_indices_layout (
TensorLayout): Layout of the optionalrow_indices. - RowIndicesEngine (
TensorEngine): Engine policy of therow_indicestensor. - row_indices_dtype (
DType): Integer element type ofrow_indices. - row_index_divisor (
Int): Divisor applied to eachrow_indicesentry to get the source row. - rows_per_block (
Int): Rows (one per warp) each block quantizes.
Args:
- output_tensor (
TileTensor[output_dtype, output_layout, MutAnyOrigin, Engine=OutputEngine]): Packed FP4 output,[rows, K / 2]. - scales_tensor (
TileTensor[scales_dtype, scales_layout, MutAnyOrigin, Engine=ScalesEngine]): Block scales in the 5D tcgen interleaved layout. - row_scales (
TileTensor[.bfloat16, row_scales_layout, MutAnyOrigin, Engine=RowScalesEngine]): Each output row's BF16 global scale,[rows]. - input_tensor (
TileTensor[input_dtype, input_layout, ImmutAnyOrigin, Engine=InputEngine]): BF16 input,[source rows, K]. - row_offsets (
TileTensor[.uint32, row_offsets_layout, ImmutAnyOrigin, Engine=RowOffsetsEngine]): Row boundaries per expert,[num_experts + 1]. - scales_offsets (
TileTensor[.uint32, scales_offsets_layout, ImmutAnyOrigin, Engine=ScalesOffsetsEngine]): Per-expert offset into the scale tiles. - row_indices (
OptionalReg[TileTensor[row_indices_dtype, row_indices_layout, ImmutAnyOrigin, Engine=RowIndicesEngine]]): Optional source row of each output row.