IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

quantize_dynamic_scaled_fp4fp8

def quantize_dynamic_scaled_fp4fp8[out_dtype: DType, scales_dtype: DType, in_dtype: DType, //, *, SF_VECTOR_SIZE: Int = Int(16), num_max_threads: Int = Int(512), gated: Bool = False, activation: def(Float32, Float32) thin -> Float32 = comptime #kgen.func.symbol<@linalg::@swiglu_activation::@"swiglu[::SIMD[DType.float32, 1]](::SIMD[DType.float32, 1],::SIMD[DType.float32, 1])"<:!lit.struct<@std::@simd::@SIMD<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype f32}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}>> apply(:!lit.generator<("value": !lit.struct<@std::@builtin::@float_literal::@FloatLiteral<:!pop.float_literal #pop.float_literal<0|1>>>, |) -> !lit.struct<@std::@simd::@SIMD<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype f32}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}>>> @std::@simd::@SIMD::@"__init__[!pop.float_literal](::FloatLiteral[$2])"<:!lit.struct<@std::@builtin::@dtype::@DType> {:dtype f32}, :!lit.struct<@std::@builtin::@simd_length::@SIMDLength> {1}, :!pop.float_literal #pop.float_literal<0|1>>, {})>>, rope_dim: Int = Int(0), rope_head_dim: Int = Int(0), rope_inverse: Bool = False, rope_layout: TensorLayout = Layout[TypeList[Int64, Int64](), TypeList[Int64, ComptimeInt[Int(1)]]()], grid_by_occupancy: Bool = False](ctx: DeviceContext, output_tile: TileTensor[out_dtype, Engine=output_tile.Engine, linear_idx_type=output_tile.linear_idx_type], scales_tile: TileTensor[scales_dtype, Engine=scales_tile.Engine, linear_idx_type=scales_tile.linear_idx_type], input_tile: TileTensor[in_dtype, Engine=input_tile.Engine, linear_idx_type=input_tile.linear_idx_type], num_cols: Int, num_cols_padded: Int, tensor_sf: Float32 = 1, rope_freqs: OptionalReg[TileTensor[.float32, rope_layout, ImmUnsafeAnyOrigin]] = None)

Launches the SM100 kernel that quantizes BF16 input to NVFP4, MXFP4, or MXFP8 with per-block scale factors.

Each thread processes ELEMENTS_PER_THREAD elements, cooperatively reduces the per-group maximum across warp lanes, derives the scale factor, and writes the quantized output and scales tensors.

Parameters:

  • ​out_dtype (DType): Element type of the quantized output tensor. Must be uint8 for NVFP4 or MXFP4, or float8_e4m3fn for MXFP8 (inferred).
  • ​scales_dtype (DType): Element type of the block scale-factor tensor. Must be float8_e4m3fn for NVFP4 or float8_e8m0fnu for MXFP4 and MXFP8 (inferred).
  • ​in_dtype (DType): Element type of the input activation tensor. Must be bfloat16 (inferred).
  • ​SF_VECTOR_SIZE (Int): Number of input elements covered by each block scale factor: 16 for NVFP4 or MXFP4, 32 for MXFP8 (defaults to 16).
  • ​num_max_threads (Int): Maximum number of threads per block for the launch grid (defaults to 512).
  • ​gated (Bool): Whether each input row is [gate | up], twice the output width, and the kernel quantizes activation(gate, up) rounded to in_dtype in place of the input.
  • ​activation (def(Float32, Float32) thin -> Float32): The gated activation; read only when gated.
  • ​rope_dim (Int): Trailing columns of each head that the kernel rotates by rope_freqs before it quantizes; 0 rotates nothing.
  • ​rope_head_dim (Int): Columns per head; read only when rope_dim > 0.
  • ​rope_inverse (Bool): Whether to rotate by the conjugate of rope_freqs.
  • ​rope_layout (TensorLayout): Layout of rope_freqs (inferred).
  • ​grid_by_occupancy (Bool): Whether to size the grid by the blocks an SM holds at the kernel's register use, instead of by threads.

Args:

Was this page helpful?