For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
dequant_nvfp4
def dequant_nvfp4(ctx: DeviceContext, output: TileTensor[Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type], input: TileTensor[Engine=input.Engine, address_space=input.address_space, linear_idx_type=input.linear_idx_type], scales: TileTensor[Engine=scales.Engine, address_space=scales.address_space, linear_idx_type=scales.linear_idx_type], tensor_sf: Float32, num_rows: Int, num_cols: Int)
Dequantize NVFP4 packed weights to bfloat16 or float32.
Args:
- ctx (
DeviceContext): Device context for kernel launch. - output (
TileTensor[Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type]): Output tensor[num_rows, num_cols]of bfloat16 or float32. - input (
TileTensor[Engine=input.Engine, address_space=input.address_space, linear_idx_type=input.linear_idx_type]): Packed weights[num_rows, num_cols // 2]of uint8. - scales (
TileTensor[Engine=scales.Engine, address_space=scales.address_space, linear_idx_type=scales.linear_idx_type]): Block scales offloat8_e4m3fn: rank 2[num_rows, num_cols // 16], or the rank-5 SM100 interleaved layout[ceildiv(num_rows, 128), ceildiv(num_cols, 64), 32, 4, 4]. - tensor_sf (
Float32): Tensor-wide decode scale (the checkpoint'sweight_scale_2). - num_rows (
Int): Number of rows (N dimension for weights). - num_cols (
Int): Number of columns (K dimension, unpacked). Must be a multiple of 16.
Raises:
If num_cols is not a multiple of 16, or if the shapes of output,
input and scales do not agree with num_rows and num_cols. The
kernels read scales on trust, so a short scale tensor would otherwise
be an out-of-bounds read.