IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

rope_quantize_dynamic_block_scaled

def rope_quantize_dynamic_block_scaled[out_dtype: DType, scales_dtype: DType, in_dtype: DType, //, *, SF_VECTOR_SIZE: Int, head_dim: Int, inverse: Bool](output_device: TileTensor[out_dtype, Engine=output_device.Engine, linear_idx_type=output_device.linear_idx_type], scales_device: TileTensor[scales_dtype, Engine=scales_device.Engine, linear_idx_type=scales_device.linear_idx_type], input_device: TileTensor[in_dtype, Engine=input_device.Engine, linear_idx_type=input_device.linear_idx_type], freqs_device: TileTensor[.float32, Engine=freqs_device.Engine, linear_idx_type=freqs_device.linear_idx_type], tensor_sf: Float32, ctx: DeviceContext)

Rotates the trailing columns of each head by RoPE and quantizes the result to NVFP4 or MXFP8 with block scales.

One kernel computes what rotating each row's heads with rope_ragged's interleaved arithmetic and then quantize_dynamic_block_scaled would, without writing the rotated input out. SM100 and SM120 only.

Parameters:

  • ​out_dtype (DType): Element type of the quantized output (inferred). uint8 for NVFP4 or float8_e4m3fn for MXFP8.
  • ​scales_dtype (DType): Element type of the block scales (inferred). float8_e4m3fn for NVFP4 or float8_e8m0fnu for MXFP8.
  • ​in_dtype (DType): Element type of the input (inferred). Must be bfloat16.
  • ​SF_VECTOR_SIZE (Int): Elements per block scale: 16 for NVFP4, 32 for MXFP8.
  • ​head_dim (Int): Columns per head. The input's width is a whole number of heads.
  • ​inverse (Bool): Whether to rotate by the conjugate of each frequency.

Args:

Raises:

Error: If freqs_device does not hold one row per input row, or output_device is not [M, N] for MXFP8 or [M, N // 2] for packed FP4.

Was this page helpful?