For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
quantize_dynamic_scaled_fp8
def quantize_dynamic_scaled_fp8[out_dtype: DType, in_dtype: DType, scales_dtype: DType, InputFnType: def[width: Int, alignment: Int](row: Int, col: Int) -> SIMD[in_dtype, width] & RegisterPassable & ImplicitlyCopyable, //, group_size_or_per_token: Int, num_cols: Int, pdl_level: PDLLevel = PDLLevel.OVERLAP_AT_BEGINNING, row_bounded: Bool = False, pow2_scales: Bool = False](input_fn: InputFnType, scaled_output: TileTensor[out_dtype, Engine=scaled_output.Engine, address_space=scaled_output.address_space, linear_idx_type=scaled_output.linear_idx_type], scales: TileTensor[scales_dtype, Engine=scales.Engine, address_space=scales.address_space, linear_idx_type=scales.linear_idx_type], scale_ub: Float32, ctx: DeviceContext, num_rows: Int, amax_floor: Float32 = 0, row_limit: Optional[Pointer[UInt32, ImmUntrackedOrigin]] = None)
TileTensor primary implementation of dynamic scaled FP8 quantization.
amax_floor lower-bounds a group's max-abs before the scale is derived,
which is how QAT recipes such as DeepSeek-V4's act_quant (floor 1e-4)
keep near-zero activation groups on the scale grid the checkpoint was
trained against. It is only honored on the float8_e8m0fnu scale path;
0 (the default) leaves the scale unchanged.
pow2_scales derives each scale as the float8_e8m0fnu path does and
stores it widened to scales_dtype, which a consumer that wants float32
UE8M0 scales would otherwise get from a separate cast.
When row_bounded, row_limit points at a device-resident count of live
rows and the kernel grid-strides over [0, row_limit) instead of covering
all num_rows. Rows at or past that count keep whatever the output and
scales buffers already held.
pdl_level picks when the kernel releases its dependents. The kernel
always waits for the predecessor grid before it reads anything a
predecessor wrote, including row_limit. At the default
PDLLevel.OVERLAP_AT_BEGINNING it releases dependents right after that
wait, so a PDL-aware consumer can run its predecessor-independent setup
while the quantizer runs; PDLLevel.ON releases them when the kernel
exits, and PDLLevel.OFF launches without PDL. A dependent's own wait
still blocks until this grid has completed, so every level is safe for a
consumer that waits before reading the outputs.