IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

batched_quantize_dynamic_scaled_fp8

def batched_quantize_dynamic_scaled_fp8[out_dtype: DType, in_dtype: DType, scales_dtype: DType, InputFnType: def[width: Int, alignment: Int](batch: Int, row: Int, col: Int) -> SIMD[in_dtype, width] & RegisterPassable & ImplicitlyCopyable, //, group_size_or_per_token: Int, num_cols: Int, pdl_level: PDLLevel = PDLLevel.OVERLAP_AT_BEGINNING](input_fn: InputFnType, scaled_output: TileTensor[out_dtype, Engine=scaled_output.Engine, address_space=scaled_output.address_space, linear_idx_type=scaled_output.linear_idx_type], scales: TileTensor[scales_dtype, Engine=scales.Engine, address_space=scales.address_space, linear_idx_type=scales.linear_idx_type], scale_ub: Float32, ctx: DeviceContext, num_rows: Int, batch_size: Int)

TileTensor primary implementation of batched dynamic scaled FP8 quantization.

pdl_level picks when the kernel releases its dependents, as in quantize_dynamic_scaled_fp8: right after the wait for the predecessor grid at the default PDLLevel.OVERLAP_AT_BEGINNING, at kernel exit for PDLLevel.ON.

Was this page helpful?