For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
rms_norm_quantize_dynamic_scaled_fp8
def rms_norm_quantize_dynamic_scaled_fp8[in_dtype: DType, out_dtype: DType, scales_dtype: DType, InputFnType: def[width: Int, alignment: Int](row: Int, col: Int) -> SIMD[in_dtype, width] & RegisterPassable & ImplicitlyCopyable, //, group_size: Int, num_cols: Int, multiply_before_cast: Bool, pdl_level: PDLLevel = PDLLevel.ON](input_fn: InputFnType, output: TileTensor[out_dtype, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type], scales: TileTensor[scales_dtype, Engine=scales.Engine, address_space=scales.address_space, linear_idx_type=scales.linear_idx_type], gamma: TileTensor[in_dtype, Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type], epsilon: Scalar[in_dtype], weight_offset: Scalar[in_dtype], scale_ub: Float32, ctx: DeviceContextBase, num_rows: Int)
Fused rms_norm + quantize_dynamic_scaled_fp8 for group (blockwise) activation scales: one launch, no normalized round trip through HBM.
Produces the outputs of rms_norm[multiply_before_cast=...] followed by
quantize_dynamic_scaled_fp8[group_size_or_per_token=group_size]:
output is [rows, num_cols] FP8 and scales is
[num_cols // group_size, rows] (row r's scale for column group g
at scales[g, r]; a padded trailing scale column is left untouched).
Per-token scales (group_size == num_cols) stay on
rms_norm_fused_quantize_dynamic_scaled_fp8.