IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

quantize_dynamic_scaled_fp8_grouped_mx

def quantize_dynamic_scaled_fp8_grouped_mx[out_dtype: DType, in_dtype: DType, scales_dtype: DType, InputFnType: def[width: Int, alignment: Int](row: Int, col: Int) -> SIMD[in_dtype, width] & RegisterPassable & ImplicitlyCopyable, //, num_cols: Int, pdl_level: PDLLevel = PDLLevel.ON](input_fn: InputFnType, scaled_output: TileTensor[out_dtype, Engine=scaled_output.Engine, address_space=scaled_output.address_space, linear_idx_type=scaled_output.linear_idx_type], scales_mx: TileTensor[scales_dtype, Engine=scales_mx.Engine, linear_idx_type=scales_mx.linear_idx_type], group_starts: TileTensor[.uint32, Engine=group_starts.Engine, linear_idx_type=group_starts.linear_idx_type], scale_offsets: TileTensor[.uint32, Engine=scale_offsets.Engine, linear_idx_type=scale_offsets.linear_idx_type], ctx: DeviceContext, num_rows: Int, amax_floor: Float32 = 0)

1x128 e8m0 dynamic fp8 quantization of group-sorted rows, with the scales written straight into the 5D MX scale tiles of a grouped block-scaled GEMM.

The codes and scales are those of quantize_dynamic_scaled_fp8 with group_size_or_per_token=128 and float8_e8m0fnu scales, bit for bit. Each 128-element scale is stored over its four 32-element MX groups, at the scale row the grouped GEMM reads for that activation row: row m of group g (group_starts[g] <= m < group_starts[g + 1]) lands in tile (m - start) // 128 + start // 128 + scale_offsets[g], row (m - start) % 128. The rest of each group's last tile is zeroed; tiles no group reaches are not written. Rows at or past group_starts[-1] (rows no GEMM group covers) get codes only.

Parameters:

  • ​out_dtype (DType): Code dtype; must be float8_e4m3fn.
  • ​in_dtype (DType): Input dtype (inferred).
  • ​scales_dtype (DType): Scale dtype; must be float8_e8m0fnu.
  • ​InputFnType (def[width: Int, alignment: Int](row: Int, col: Int) -> SIMD[in_dtype, width] & RegisterPassable & ImplicitlyCopyable): Input lambda type (inferred).
  • ​num_cols (Int): K, a multiple of 128.
  • ​pdl_level (PDLLevel): Programmatic dependent launch level.

Args:

Was this page helpful?