IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

rms_norm_quantize_dynamic_scaled_fp8

def rms_norm_quantize_dynamic_scaled_fp8[in_dtype: DType, out_dtype: DType, scales_dtype: DType, InputFnType: def[width: Int, alignment: Int](row: Int, col: Int) -> SIMD[in_dtype, width] & RegisterPassable & ImplicitlyCopyable, //, group_size: Int, num_cols: Int, multiply_before_cast: Bool, pdl_level: PDLLevel = PDLLevel.ON](input_fn: InputFnType, output: TileTensor[out_dtype, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type], scales: TileTensor[scales_dtype, Engine=scales.Engine, address_space=scales.address_space, linear_idx_type=scales.linear_idx_type], gamma: TileTensor[in_dtype, Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type], epsilon: Scalar[in_dtype], weight_offset: Scalar[in_dtype], scale_ub: Float32, ctx: DeviceContext, num_rows: Int)

Fused rms_norm + quantize_dynamic_scaled_fp8 for group (blockwise) activation scales: one launch, no normalized round trip through HBM.

Produces the outputs of rms_norm[multiply_before_cast=...] followed by quantize_dynamic_scaled_fp8[group_size_or_per_token=group_size]: output is [rows, num_cols] FP8 and scales is [num_cols // group_size, rows] (row r's scale for column group g at scales[g, r]; a padded trailing scale column is left untouched). Per-token scales (group_size == num_cols) stay on rms_norm_fused_quantize_dynamic_scaled_fp8.

Was this page helpful?