For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
grouped_quantize_nvfp4_row_amax
def grouped_quantize_nvfp4_row_amax[input_dtype: DType, output_dtype: DType, scales_dtype: DType, //, OutputEngine: TensorEngine, ScalesEngine: TensorEngine, RowScalesEngine: TensorEngine, InputEngine: TensorEngine, RowOffsetsEngine: TensorEngine, ScalesOffsetsEngine: TensorEngine, *, input_fn: Optional[def[dtype: DType, width: SIMDLength](SIMD[dtype, width]) thin -> SIMD[dtype, width]] = None, RowIndicesLayoutType: TensorLayout = Layout[TypeList[Int64](), TypeList[ComptimeInt[Int(1)]]()], RowIndicesEngine: TensorEngine = DefaultEngine, row_indices_dtype: DType = .uint32, row_index_divisor: Int = Int(1), rows_per_block: Int = Int(4)](output_tensor: TileTensor[output_dtype, Engine=OutputEngine, linear_idx_type=output_tensor.linear_idx_type], scales_tensor: TileTensor[scales_dtype, Engine=ScalesEngine, linear_idx_type=scales_tensor.linear_idx_type], row_scales: TileTensor[.bfloat16, Engine=RowScalesEngine, linear_idx_type=row_scales.linear_idx_type], input_tensor: TileTensor[input_dtype, Engine=InputEngine, linear_idx_type=input_tensor.linear_idx_type], row_offsets: TileTensor[.uint32, Engine=RowOffsetsEngine, linear_idx_type=row_offsets.linear_idx_type], scales_offsets: TileTensor[.uint32, Engine=ScalesOffsetsEngine, linear_idx_type=scales_offsets.linear_idx_type], ctx: DeviceContext, row_indices: OptionalReg[TileTensor[row_indices_dtype, RowIndicesLayoutType, ImmutAnyOrigin, Engine=RowIndicesEngine]] = None)
Quantizes grouped BF16 rows to NVFP4, each row with its own scale.
Equal to a chain of a row amax, row_scale = bf16(max(amax / 2688, 1e-30)), bf16(x * (1 / row_scale)) and
grouped_quantize_dynamic_scaled_fp4_async with a tensor-wise scale of 1,
but in one kernel with the same bits out. row_scales is the matmul's
a_row_scales. SM100 only.
Parameters:
- input_dtype (
DType): Element type of the input (inferred,bfloat16). - output_dtype (
DType): Element type of the packed output (inferred). - scales_dtype (
DType): Element type of the block scales (inferred). - OutputEngine (
TensorEngine): Engine policy of theoutput_tensor. - ScalesEngine (
TensorEngine): Engine policy of thescales_tensor. - RowScalesEngine (
TensorEngine): Engine policy ofrow_scales. - InputEngine (
TensorEngine): Engine policy of theinput_tensor. - RowOffsetsEngine (
TensorEngine): Engine policy of therow_offsetstensor. - ScalesOffsetsEngine (
TensorEngine): Engine policy of thescales_offsetstensor. - input_fn (
Optional[def[dtype: DType, width: SIMDLength](SIMD[dtype, width]) thin -> SIMD[dtype, width]]): Optional elementwise transform quantized in place of the input, such as relu2 (seenvfp4_row_input_fn_type). - RowIndicesLayoutType (
TensorLayout): Layout of the optionalrow_indicestensor. - RowIndicesEngine (
TensorEngine): Engine policy of therow_indicestensor. - row_indices_dtype (
DType): Integer element type of therow_indicestensor. - row_index_divisor (
Int): Divisor applied to eachrow_indicesentry to get the source row. - rows_per_block (
Int): Rows (one per warp) each block quantizes.
Args:
- output_tensor (
TileTensor[output_dtype, Engine=OutputEngine, linear_idx_type=output_tensor.linear_idx_type]): Packed FP4 output,[rows, K / 2]. - scales_tensor (
TileTensor[scales_dtype, Engine=ScalesEngine, linear_idx_type=scales_tensor.linear_idx_type]): Block scales,[tiles, ceil(K / 64), 32, 4, 4]. - row_scales (
TileTensor[.bfloat16, Engine=RowScalesEngine, linear_idx_type=row_scales.linear_idx_type]): Each row's BF16 global scale,[rows]; may be strided. - input_tensor (
TileTensor[input_dtype, Engine=InputEngine, linear_idx_type=input_tensor.linear_idx_type]): BF16 input,[source rows, K]. - row_offsets (
TileTensor[.uint32, Engine=RowOffsetsEngine, linear_idx_type=row_offsets.linear_idx_type]): Row boundaries per expert,[num_experts + 1]. - scales_offsets (
TileTensor[.uint32, Engine=ScalesOffsetsEngine, linear_idx_type=scales_offsets.linear_idx_type]): Per-expert offset intoscales_tensor. - ctx (
DeviceContext): Device context used to enqueue the kernel. - row_indices (
OptionalReg[TileTensor[row_indices_dtype, RowIndicesLayoutType, ImmutAnyOrigin, Engine=RowIndicesEngine]]): Optional source row of each output row; output rowireads input rowrow_indices[i] // row_index_divisor.