For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
rope_quantize_dynamic_block_scaled
def rope_quantize_dynamic_block_scaled[out_dtype: DType, scales_dtype: DType, in_dtype: DType, //, *, SF_VECTOR_SIZE: Int, head_dim: Int, inverse: Bool](output_device: TileTensor[out_dtype, Engine=output_device.Engine, linear_idx_type=output_device.linear_idx_type], scales_device: TileTensor[scales_dtype, Engine=scales_device.Engine, linear_idx_type=scales_device.linear_idx_type], input_device: TileTensor[in_dtype, Engine=input_device.Engine, linear_idx_type=input_device.linear_idx_type], freqs_device: TileTensor[.float32, Engine=freqs_device.Engine, linear_idx_type=freqs_device.linear_idx_type], tensor_sf: Float32, ctx: DeviceContext)
Rotates the trailing columns of each head by RoPE and quantizes the result to NVFP4 or MXFP8 with block scales.
One kernel computes what rotating each row's heads with rope_ragged's
interleaved arithmetic and then quantize_dynamic_block_scaled would,
without writing the rotated input out. SM100 and SM120 only.
Parameters:
- out_dtype (
DType): Element type of the quantized output (inferred).uint8for NVFP4 orfloat8_e4m3fnfor MXFP8. - scales_dtype (
DType): Element type of the block scales (inferred).float8_e4m3fnfor NVFP4 orfloat8_e8m0fnufor MXFP8. - in_dtype (
DType): Element type of the input (inferred). Must bebfloat16. - SF_VECTOR_SIZE (
Int): Elements per block scale: 16 for NVFP4, 32 for MXFP8. - head_dim (
Int): Columns per head. The input's width is a whole number of heads. - inverse (
Bool): Whether to rotate by the conjugate of each frequency.
Args:
- output_device (
TileTensor[out_dtype, Engine=output_device.Engine, linear_idx_type=output_device.linear_idx_type]): Quantized output of shape[M, N // 2]for packed FP4 or[M, N]for MXFP8. - scales_device (
TileTensor[scales_dtype, Engine=scales_device.Engine, linear_idx_type=scales_device.linear_idx_type]): Block scales in the 5D TCGEN interleaved layout. - input_device (
TileTensor[in_dtype, Engine=input_device.Engine, linear_idx_type=input_device.linear_idx_type]): Input of shape[M, N]. - freqs_device (
TileTensor[.float32, Engine=freqs_device.Engine, linear_idx_type=freqs_device.linear_idx_type]):[M, rope_dim]interleaved(cos, sin)pairs, one row per input row. Each head's lastrope_dimcolumns are rotated. - tensor_sf (
Float32): Tensor-wise scale factor multiplied into each block scale. - ctx (
DeviceContext): Device context used to enqueue the kernel.
Raises:
Error: If freqs_device does not hold one row per input row, or
output_device is not [M, N] for MXFP8 or [M, N // 2] for
packed FP4.