IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

grouped_quantize_nvfp4_row_amax

def grouped_quantize_nvfp4_row_amax[input_dtype: DType, output_dtype: DType, scales_dtype: DType, //, OutputEngine: TensorEngine, ScalesEngine: TensorEngine, RowScalesEngine: TensorEngine, InputEngine: TensorEngine, RowOffsetsEngine: TensorEngine, ScalesOffsetsEngine: TensorEngine, *, input_fn: Optional[def[dtype: DType, width: SIMDLength](SIMD[dtype, width]) thin -> SIMD[dtype, width]] = None, RowIndicesLayoutType: TensorLayout = Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], RowIndicesEngine: TensorEngine = DefaultEngine, row_indices_dtype: DType = .uint32, row_index_divisor: Int = Int(1), rows_per_block: Int = Int(4)](output_tensor: TileTensor[output_dtype, Engine=OutputEngine, linear_idx_type=output_tensor.linear_idx_type], scales_tensor: TileTensor[scales_dtype, Engine=ScalesEngine, linear_idx_type=scales_tensor.linear_idx_type], row_scales: TileTensor[.bfloat16, Engine=RowScalesEngine, linear_idx_type=row_scales.linear_idx_type], input_tensor: TileTensor[input_dtype, Engine=InputEngine, linear_idx_type=input_tensor.linear_idx_type], row_offsets: TileTensor[.uint32, Engine=RowOffsetsEngine, linear_idx_type=row_offsets.linear_idx_type], scales_offsets: TileTensor[.uint32, Engine=ScalesOffsetsEngine, linear_idx_type=scales_offsets.linear_idx_type], ctx: DeviceContext, row_indices: OptionalReg[TileTensor[row_indices_dtype, RowIndicesLayoutType, ImmutAnyOrigin, Engine=RowIndicesEngine]] = None)

Quantizes grouped BF16 rows to NVFP4, each row with its own scale.

Equal to a chain of a row amax, row_scale = bf16(max(amax / 2688, 1e-30)), bf16(x * (1 / row_scale)) and grouped_quantize_dynamic_scaled_fp4_async with a tensor-wise scale of 1, but in one kernel with the same bits out. row_scales is the matmul's a_row_scales. SM100 only.

Parameters:

  • ​input_dtype (DType): Element type of the input (inferred, bfloat16).
  • ​output_dtype (DType): Element type of the packed output (inferred).
  • ​scales_dtype (DType): Element type of the block scales (inferred).
  • ​OutputEngine (TensorEngine): Engine policy of the output_tensor.
  • ​ScalesEngine (TensorEngine): Engine policy of the scales_tensor.
  • ​RowScalesEngine (TensorEngine): Engine policy of row_scales.
  • ​InputEngine (TensorEngine): Engine policy of the input_tensor.
  • ​RowOffsetsEngine (TensorEngine): Engine policy of the row_offsets tensor.
  • ​ScalesOffsetsEngine (TensorEngine): Engine policy of the scales_offsets tensor.
  • ​input_fn (Optional[def[dtype: DType, width: SIMDLength](SIMD[dtype, width]) thin -> SIMD[dtype, width]]): Optional elementwise transform quantized in place of the input, such as relu2 (see nvfp4_row_input_fn_type).
  • ​RowIndicesLayoutType (TensorLayout): Layout of the optional row_indices tensor.
  • ​RowIndicesEngine (TensorEngine): Engine policy of the row_indices tensor.
  • ​row_indices_dtype (DType): Integer element type of the row_indices tensor.
  • ​row_index_divisor (Int): Divisor applied to each row_indices entry to get the source row.
  • ​rows_per_block (Int): Rows (one per warp) each block quantizes.

Args:

Was this page helpful?