For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
quantize_dynamic_scaled_fp4fp8
def quantize_dynamic_scaled_fp4fp8[out_dtype: DType, scales_dtype: DType, in_dtype: DType, //, *, SF_VECTOR_SIZE: Int = Int(16), num_max_threads: Int = Int(512), gated: Bool = False, activation: def(Float32, Float32) thin -> Float32 = comptime #kgen.func.symbol<@linalg::@swiglu_activation::@"swiglu[::SIMD[DType.float32, 1]](::SIMD[DType.float32, 1],::SIMD[DType.float32, 1])"<:!lit.struct<@std::@simd::@SIMD<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype f32}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}>> apply(:!lit.generator<("value": !lit.struct<@std::@builtin::@float_literal::@FloatLiteral<:!pop.float_literal #pop.float_literal<0|1>>>, |) -> !lit.struct<@std::@simd::@SIMD<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype f32}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}>>> @std::@simd::@SIMD::@"__init__[!pop.float_literal](::FloatLiteral[$2])"<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype f32}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}, :!pop.float_literal #pop.float_literal<0|1>>, {})>>, rope_dim: Int = Int(0), rope_head_dim: Int = Int(0), rope_inverse: Bool = False, rope_layout: TensorLayout = Layout[TypeList[Int64, Int64](), TypeList[Int64, ComptimeInt[Int(1)]]()], grid_by_occupancy: Bool = False](ctx: DeviceContext, output_tile: TileTensor[out_dtype, Engine=output_tile.Engine, linear_idx_type=output_tile.linear_idx_type], scales_tile: TileTensor[scales_dtype, Engine=scales_tile.Engine, linear_idx_type=scales_tile.linear_idx_type], input_tile: TileTensor[in_dtype, Engine=input_tile.Engine, linear_idx_type=input_tile.linear_idx_type], num_cols: Int, num_cols_padded: Int, tensor_sf: Float32 = 1, rope_freqs: OptionalReg[TileTensor[.float32, rope_layout, ImmUnsafeAnyOrigin]] = None)
Launches the SM100 kernel that quantizes BF16 input to NVFP4, MXFP4, or MXFP8 with per-block scale factors.
Each thread processes ELEMENTS_PER_THREAD elements, cooperatively
reduces the per-group maximum across warp lanes, derives the scale
factor, and writes the quantized output and scales tensors.
Parameters:
- out_dtype (
DType): Element type of the quantized output tensor. Must beuint8for NVFP4 or MXFP4, orfloat8_e4m3fnfor MXFP8 (inferred). - scales_dtype (
DType): Element type of the block scale-factor tensor. Must befloat8_e4m3fnfor NVFP4 orfloat8_e8m0fnufor MXFP4 and MXFP8 (inferred). - in_dtype (
DType): Element type of the input activation tensor. Must bebfloat16(inferred). - SF_VECTOR_SIZE (
Int): Number of input elements covered by each block scale factor: 16 for NVFP4 or MXFP4, 32 for MXFP8 (defaults to 16). - num_max_threads (
Int): Maximum number of threads per block for the launch grid (defaults to 512). - gated (
Bool): Whether each input row is[gate | up], twice the output width, and the kernel quantizesactivation(gate, up)rounded toin_dtypein place of the input. - activation (
def(Float32, Float32) thin -> Float32): The gated activation; read only whengated. - rope_dim (
Int): Trailing columns of each head that the kernel rotates byrope_freqsbefore it quantizes; 0 rotates nothing. - rope_head_dim (
Int): Columns per head; read only whenrope_dim > 0. - rope_inverse (
Bool): Whether to rotate by the conjugate ofrope_freqs. - rope_layout (
TensorLayout): Layout ofrope_freqs(inferred). - grid_by_occupancy (
Bool): Whether to size the grid by the blocks an SM holds at the kernel's register use, instead of by threads.
Args:
- ctx (
DeviceContext): Device context used to enqueue the kernel. - output_tile (
TileTensor[out_dtype, Engine=output_tile.Engine, linear_idx_type=output_tile.linear_idx_type]): Output tile for the quantized elements of shape[num_rows, num_cols // 2]for packed FP4uint8or[num_rows, num_cols]for MXFP8. - scales_tile (
TileTensor[scales_dtype, Engine=scales_tile.Engine, linear_idx_type=scales_tile.linear_idx_type]): Output tile for the per-block scale factors. - input_tile (
TileTensor[in_dtype, Engine=input_tile.Engine, linear_idx_type=input_tile.linear_idx_type]): Inputbfloat16activation tile of shape[num_rows, num_cols], or[num_rows, 2 * num_cols]whengated. - num_cols (
Int): Number of columns in the input tensor before padding. - num_cols_padded (
Int): Number of columns in the input tensor after padding to a multiple ofSF_VECTOR_SIZE; padded columns are zeroed in the output and scales. - tensor_sf (
Float32): Tensor-wise scale factor applied to the quantization (defaults to 1.0). - rope_freqs (
OptionalReg[TileTensor[.float32, rope_layout, ImmUnsafeAnyOrigin]]):[num_rows, rope_dim]float32interleaved(cos, sin)pairs, one row per input row; required whenrope_dim > 0. Each head's rotated columns are rounded toin_dtypebefore they are quantized.