For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
quantize_dynamic_scaled_fp8_grouped_mx
def quantize_dynamic_scaled_fp8_grouped_mx[out_dtype: DType, in_dtype: DType, scales_dtype: DType, InputFnType: def[width: Int, alignment: Int](row: Int, col: Int) -> SIMD[in_dtype, width] & RegisterPassable & ImplicitlyCopyable, //, num_cols: Int, pdl_level: PDLLevel = PDLLevel.ON](input_fn: InputFnType, scaled_output: TileTensor[out_dtype, Engine=scaled_output.Engine, address_space=scaled_output.address_space, linear_idx_type=scaled_output.linear_idx_type], scales_mx: TileTensor[scales_dtype, Engine=scales_mx.Engine, linear_idx_type=scales_mx.linear_idx_type], group_starts: TileTensor[.uint32, Engine=group_starts.Engine, linear_idx_type=group_starts.linear_idx_type], scale_offsets: TileTensor[.uint32, Engine=scale_offsets.Engine, linear_idx_type=scale_offsets.linear_idx_type], ctx: DeviceContext, num_rows: Int, amax_floor: Float32 = 0)
1x128 e8m0 dynamic fp8 quantization of group-sorted rows, with the scales written straight into the 5D MX scale tiles of a grouped block-scaled GEMM.
The codes and scales are those of quantize_dynamic_scaled_fp8 with
group_size_or_per_token=128 and float8_e8m0fnu scales, bit for bit.
Each 128-element scale is stored over its four 32-element MX groups, at
the scale row the grouped GEMM reads for that activation row: row m of
group g (group_starts[g] <= m < group_starts[g + 1]) lands in tile
(m - start) // 128 + start // 128 + scale_offsets[g], row
(m - start) % 128. The rest of each group's last tile is zeroed; tiles
no group reaches are not written. Rows at or past group_starts[-1]
(rows no GEMM group covers) get codes only.
Parameters:
- out_dtype (
DType): Code dtype; must befloat8_e4m3fn. - in_dtype (
DType): Input dtype (inferred). - scales_dtype (
DType): Scale dtype; must befloat8_e8m0fnu. - InputFnType (
def[width: Int, alignment: Int](row: Int, col: Int) -> SIMD[in_dtype, width]&RegisterPassable&ImplicitlyCopyable): Input lambda type (inferred). - num_cols (
Int): K, a multiple of 128. - pdl_level (
PDLLevel): Programmatic dependent launch level.
Args:
- input_fn (
InputFnType): Loadswidthinput elements at(row, col). - scaled_output (
TileTensor[out_dtype, Engine=scaled_output.Engine, address_space=scaled_output.address_space, linear_idx_type=scaled_output.linear_idx_type]):[num_rows, K]fp8 codes. - scales_mx (
TileTensor[scales_dtype, Engine=scales_mx.Engine, linear_idx_type=scales_mx.linear_idx_type]):[scale_rows / 128, K / 128, 32, 4, 4]e8m0 MX scale tiles. - group_starts (
TileTensor[.uint32, Engine=group_starts.Engine, linear_idx_type=group_starts.linear_idx_type]):[groups + 1]exclusive prefix sum of the group sizes. - scale_offsets (
TileTensor[.uint32, Engine=scale_offsets.Engine, linear_idx_type=scale_offsets.linear_idx_type]):[groups]per-group scale tile offsets, as passed to the grouped GEMM. - ctx (
DeviceContext): Device context used to enqueue the kernel. - num_rows (
Int): Number of activation rows. - amax_floor (
Float32): Lower bound on a group's max-abs, as inquantize_dynamic_scaled_fp8.