For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
swiglu_quantize_dynamic_block_scaled
def swiglu_quantize_dynamic_block_scaled[out_dtype: DType, scales_dtype: DType, in_dtype: DType, //, *, SF_VECTOR_SIZE: Int, activation: def(Float32, Float32) thin -> Float32](output_device: TileTensor[out_dtype, Engine=output_device.Engine, linear_idx_type=output_device.linear_idx_type], scales_device: TileTensor[scales_dtype, Engine=scales_device.Engine, linear_idx_type=scales_device.linear_idx_type], gate_up_device: TileTensor[in_dtype, Engine=gate_up_device.Engine, linear_idx_type=gate_up_device.linear_idx_type], tensor_sf: Float32, ctx: DeviceContext)
Quantizes the gated activation of a [gate | up] input to NVFP4, MXFP4, or MXFP8 with block scales.
One kernel computes what activation(gate, up) rounded to in_dtype
and then quantize_dynamic_block_scaled would, without writing the
activation out. SM100 and SM120 run every supported format; AMD CDNA4
(MI355X) runs MXFP4 and MXFP8, with rank-2 row-major E8M0 scales.
Parameters:
- out_dtype (
DType): Element type of the quantized output (inferred).uint8for NVFP4/MXFP4 orfloat8_e4m3fnfor MXFP8. - scales_dtype (
DType): Element type of the block scales (inferred).float8_e4m3fnfor NVFP4 orfloat8_e8m0fnufor MXFP4/MXFP8. - in_dtype (
DType): Element type of the input (inferred). Must bebfloat16. - SF_VECTOR_SIZE (
Int): Elements per block scale: 16 for NVFP4, 32 for MXFP4 and MXFP8. - activation (
def(Float32, Float32) thin -> Float32): The gated activation, applied to each(gate, up)pair.
Args:
- output_device (
TileTensor[out_dtype, Engine=output_device.Engine, linear_idx_type=output_device.linear_idx_type]): Quantized output of shape[M, N // 2]for packed FP4 or[M, N]for MXFP8. - scales_device (
TileTensor[scales_dtype, Engine=scales_device.Engine, linear_idx_type=scales_device.linear_idx_type]): Block scales in the 5D TCGEN interleaved layout on SM100/SM120, or rank-2[M, N // SF_VECTOR_SIZE]on CDNA4. - gate_up_device (
TileTensor[in_dtype, Engine=gate_up_device.Engine, linear_idx_type=gate_up_device.linear_idx_type]): Input of shape[M, 2 * N]:gatein the firstNcolumns,upin the lastN. - tensor_sf (
Float32): Tensor-wise scale factor multiplied into each block scale; ignored on CDNA4, whose MXFP8 path is purely block scaled. - ctx (
DeviceContext): Device context used to enqueue the kernel.
Raises:
Error: If output_device is not [M, N] for MXFP8 or [M, N // 2]
for packed FP4.