IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

grouped_quantize_nvfp4_row_amax_kernel

def grouped_quantize_nvfp4_row_amax_kernel[output_dtype: DType, scales_dtype: DType, input_dtype: DType, output_layout: TensorLayout, scales_layout: TensorLayout, row_scales_layout: TensorLayout, input_layout: TensorLayout, row_offsets_layout: TensorLayout, scales_offsets_layout: TensorLayout, OutputEngine: TensorEngine, ScalesEngine: TensorEngine, RowScalesEngine: TensorEngine, InputEngine: TensorEngine, RowOffsetsEngine: TensorEngine, ScalesOffsetsEngine: TensorEngine, input_fn: Optional[def[dtype: DType, width: SIMDLength](SIMD[dtype, width]) capturing thin -> SIMD[dtype, width]], row_indices_layout: TensorLayout = Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], RowIndicesEngine: TensorEngine = DefaultEngine, row_indices_dtype: DType = .uint32, row_index_divisor: Int = Int(1), rows_per_block: Int = Int(4)](output_tensor: TileTensor[output_dtype, output_layout, MutAnyOrigin, Engine=OutputEngine], scales_tensor: TileTensor[scales_dtype, scales_layout, MutAnyOrigin, Engine=ScalesEngine], row_scales: TileTensor[.bfloat16, row_scales_layout, MutAnyOrigin, Engine=RowScalesEngine], input_tensor: TileTensor[input_dtype, input_layout, ImmutAnyOrigin, Engine=InputEngine], row_offsets: TileTensor[.uint32, row_offsets_layout, ImmutAnyOrigin, Engine=RowOffsetsEngine], scales_offsets: TileTensor[.uint32, scales_offsets_layout, ImmutAnyOrigin, Engine=ScalesOffsetsEngine], row_indices: OptionalReg[TileTensor[row_indices_dtype, row_indices_layout, ImmutAnyOrigin, Engine=RowIndicesEngine]] = None)

Quantizes BF16 rows to NVFP4, each with its own global scale.

This is the fusion of the graph-op chain

amax = max(abs(f32(x)), axis=-1)
row_scale = bf16(max(amax / 2688, 1e-30))
scaled = bf16(f32(x) * (1 / f32(row_scale)))
grouped_quantize_dynamic_scaled_fp4(scaled, sf_tensor=1)

with each rounding to BF16 kept, so that the payload and the block scales match the chain bit for bit. A warp owns a row: it loads the row into registers with 16-byte loads, reduces the row's amax, derives row_scale, and quantizes the row in 16-element blocks from those registers, so the row is read from memory once. Rows wider than the register cache are read a second time for the quantize.

The grid scales with the rows rather than the scale tiles: the first ceildiv(rows, rows_per_block) blocks give a warp to each output row, which finds its expert, and so its scale tile, from the expert offsets in one round of loads. One more block per allocated scale tile clears the scale slots of that tile's padding rows, which must be zero. No two warps write the same byte, and the scale stores go through scales_layout.

Parameters:

  • ​output_dtype (DType): Element type of the packed output (uint8).
  • ​scales_dtype (DType): Element type of the block scales (float8_e4m3fn).
  • ​input_dtype (DType): Element type of the input (bfloat16).
  • ​output_layout (TensorLayout): Layout of output_tensor.
  • ​scales_layout (TensorLayout): Layout of scales_tensor.
  • ​row_scales_layout (TensorLayout): Layout of row_scales.
  • ​input_layout (TensorLayout): Layout of input_tensor.
  • ​row_offsets_layout (TensorLayout): Layout of row_offsets.
  • ​scales_offsets_layout (TensorLayout): Layout of scales_offsets.
  • ​OutputEngine (TensorEngine): Engine policy of the output_tensor.
  • ​ScalesEngine (TensorEngine): Engine policy of the scales_tensor.
  • ​RowScalesEngine (TensorEngine): Engine policy of row_scales.
  • ​InputEngine (TensorEngine): Engine policy of the input_tensor.
  • ​RowOffsetsEngine (TensorEngine): Engine policy of the row_offsets tensor.
  • ​ScalesOffsetsEngine (TensorEngine): Engine policy of the scales_offsets tensor.
  • ​input_fn (Optional[def[dtype: DType, width: SIMDLength](SIMD[dtype, width]) capturing thin -> SIMD[dtype, width]]): Optional transform quantized in place of each input vector, such as relu2.
  • ​row_indices_layout (TensorLayout): Layout of the optional row_indices.
  • ​RowIndicesEngine (TensorEngine): Engine policy of the row_indices tensor.
  • ​row_indices_dtype (DType): Integer element type of row_indices.
  • ​row_index_divisor (Int): Divisor applied to each row_indices entry to get the source row.
  • ​rows_per_block (Int): Rows (one per warp) each block quantizes.

Args:

Was this page helpful?