IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

rms_norm_quantize_mxfp8_q_kv

def rms_norm_quantize_mxfp8_q_kv(output_device: TileTensor[.float8_e4m3fn, Engine=output_device.Engine, linear_idx_type=output_device.linear_idx_type], scales_device: TileTensor[.float8_e8m0fnu, Engine=scales_device.Engine, linear_idx_type=scales_device.linear_idx_type], kv_output_device: TileTensor[.bfloat16, Engine=kv_output_device.Engine, linear_idx_type=kv_output_device.linear_idx_type], input_device: TileTensor[.bfloat16, Engine=input_device.Engine, linear_idx_type=input_device.linear_idx_type], q_gamma_device: TileTensor[.bfloat16, Engine=q_gamma_device.Engine, linear_idx_type=q_gamma_device.linear_idx_type], kv_gamma_device: TileTensor[.bfloat16, Engine=kv_gamma_device.Engine, linear_idx_type=kv_gamma_device.linear_idx_type], epsilon: Float32, ctx: DeviceContext)

RMS-normalizes the q and kv parts of each row and quantizes the q part to MXFP8 with block scales.

One block per row holds the whole [q | kv] row in registers. Its q warps reduce the sum of squares of the q part, its kv warps that of the kv part. The q part is quantized as quantize_dynamic_block_scaled would, and the kv part is written normalized in bfloat16. Both normalized parts are rounded to bfloat16 where rms_norm rounds them, so the bytes match the unfused kernels up to the order of the sum of squares. SM100 and SM120 only.

Args:

Was this page helpful?