For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
rms_norm_quantize_mxfp8_q_kv
def rms_norm_quantize_mxfp8_q_kv(output_device: TileTensor[.float8_e4m3fn, Engine=output_device.Engine, linear_idx_type=output_device.linear_idx_type], scales_device: TileTensor[.float8_e8m0fnu, Engine=scales_device.Engine, linear_idx_type=scales_device.linear_idx_type], kv_output_device: TileTensor[.bfloat16, Engine=kv_output_device.Engine, linear_idx_type=kv_output_device.linear_idx_type], input_device: TileTensor[.bfloat16, Engine=input_device.Engine, linear_idx_type=input_device.linear_idx_type], q_gamma_device: TileTensor[.bfloat16, Engine=q_gamma_device.Engine, linear_idx_type=q_gamma_device.linear_idx_type], kv_gamma_device: TileTensor[.bfloat16, Engine=kv_gamma_device.Engine, linear_idx_type=kv_gamma_device.linear_idx_type], epsilon: Float32, ctx: DeviceContext)
RMS-normalizes the q and kv parts of each row and quantizes the q part to MXFP8 with block scales.
One block per row holds the whole [q | kv] row in registers. Its q warps
reduce the sum of squares of the q part, its kv warps that of the kv part.
The q part is quantized as quantize_dynamic_block_scaled would, and the
kv part is written normalized in bfloat16. Both normalized parts are
rounded to bfloat16 where rms_norm rounds them, so the bytes match the
unfused kernels up to the order of the sum of squares. SM100 and SM120
only.
Args:
- output_device (
TileTensor[.float8_e4m3fn, Engine=output_device.Engine, linear_idx_type=output_device.linear_idx_type]): Quantized q of shape[M, q_cols]. - scales_device (
TileTensor[.float8_e8m0fnu, Engine=scales_device.Engine, linear_idx_type=scales_device.linear_idx_type]): Block scales of q in the 5D TCGEN interleaved layout. - kv_output_device (
TileTensor[.bfloat16, Engine=kv_output_device.Engine, linear_idx_type=kv_output_device.linear_idx_type]): Normalized kv of shape[M, kv_cols]. - input_device (
TileTensor[.bfloat16, Engine=input_device.Engine, linear_idx_type=input_device.linear_idx_type]): Input of shape[M, q_cols + kv_cols], q then kv. - q_gamma_device (
TileTensor[.bfloat16, Engine=q_gamma_device.Engine, linear_idx_type=q_gamma_device.linear_idx_type]): The q weight,[q_cols]. - kv_gamma_device (
TileTensor[.bfloat16, Engine=kv_gamma_device.Engine, linear_idx_type=kv_gamma_device.linear_idx_type]): The kv weight,[kv_cols]. - epsilon (
Float32): Added to each mean square. Rounded to bfloat16 first, asrms_normdoes. - ctx (
DeviceContext): Device context used to enqueue the kernel.