For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
quantization
Registers quantization and dequantization graph ops backed by the quantization and linalg kernels.
comptime values
logger
comptime logger = Logger(stdout, prefix=String(""), source_location=False)
Structs
-
GGMLQ40Dequantize: Registers theggml_q4_0_dequantizegraph op with the graph compiler. -
GGMLQ4KDequantize: Registers theggml_q4_k_dequantizegraph op with the graph compiler. -
GGMLQ6KDequantize: Registers theggml_q6_k_dequantizegraph op with the graph compiler. -
LayerNormQuantizeDynamicScaledFloat8: Registers themo.layer_norm_quantize_dynamic_scaled_float8graph op: a biasedlayer_norm(float32 statistics, rounded to the input dtype) fused with a group-scaled (blockwise-FP8)mo.quantize_dynamic_scaled_float8. Same outputs and scale layout as the unfused pair;group_size_or_per_tokenmust be a proper divisor of the row, and the row width must be static. -
QMatmulGPU_b4_g128: Registers theqmatmul_b4_g128graph op with the graph compiler. -
QMatmulGPURepackGPTQ_b4_g128: Registers theGPTQ_gpu_repack_b4_g128graph op with the graph compiler. -
QMatmulGPURepackGPTQ_b4_g128_desc_act: Registers theGPTQ_gpu_repack_b4_g128_desc_actgraph op with the graph compiler. -
QuantizeDynamicScaledFloat8: Registers themo.composite.quantize_dynamic_scaled_fp8graph op with the graph compiler. -
QuantizeDynamicScaledFloat8GroupedMx: Registers thequantization.quantize_dynamic_scaled_float8.grouped_mxgraph op. -
QuantizeDynamicScaledFloat8Pow2Scales: Registers thequantization.quantize_dynamic_scaled_float8.pow2_scalesgraph op. -
QuantizeDynamicScaledFloat8RowBounded: Registers thequantization.quantize_dynamic_scaled_float8.row_boundedgraph op. -
QuantizeStaticScaledFloat8: Registers thequantization.quantize_static_scaled_float8graph op with the graph compiler. -
QuantizeTensorDynamicScaledFloat8: Registers thequantization.quantize_tensor_dynamic_scaled_float8graph op with the graph compiler. -
ResizeBicubic: Registers themo.resize.bicubicgraph op with the graph compiler. -
ResizeLinear: Registers themo.resize.lineargraph op with the graph compiler. -
ResizeNearest: Registers themo.resize.nearestgraph op with the graph compiler. -
RMSNormFusedQuantizeDynamicScaledFP8: Registers themo.composite.rms_norm_fused_quantize_dynamic_scaled_fp8graph op with the graph compiler. -
RMSNormQuantizeDynamicScaledFloat8: Registers thequantization.rms_norm_quantize_dynamic_scaled_float8graph op:mo.composite.reduce.rms_normfused with a group-scaled (blockwise-FP8)mo.composite.quantize_dynamic_scaled_fp8. Same outputs and scale layout as the unfused pair;group_size_or_per_tokenmust be a proper divisor of the row (per-token scales stay on the rms_norm + quantize composite). -
RMSNormQuantizeDynamicScaledFloat8Dual: Registers themo.rms_norm_quantize_dynamic_scaled_float8.dualgraph op:mo.reduce.rms_normfused with a group-scaled (blockwise-FP8)mo.quantize_dynamic_scaled_float8, keeping the normalized row as a third output for the norm's other readers. Same outputs and scale layout as the unfused pair;group_size_or_per_tokenmust be a proper divisor of the row. -
Struct_block_scaled_preshuffle_b_5d: Run the AMD CDNA4 MXFP4 B 5D preshuffle as a custom op. -
Struct_block_scaled_preshuffle_scale_4d_per_expert: Per-step A-scale preshuffle for the AMD CDNA4 preb grouped matmul. -
Struct_dequant_mxfp4: Registers thequantization.dequant.mxfp4graph op with the graph compiler. -
Struct_dequant_mxfp6: Registers thequantization.dequant.mxfp6graph op with the graph compiler. -
Struct_dequant_mxfp8: Registers thequantization.dequant.mxfp8graph op with the graph compiler. -
Struct_dequant_nvfp4: Registers thequantization.dequant.nvfp4graph op with the graph compiler. -
Struct_grouped_quantize_dynamic_block_scaled: Registers thequantization.grouped.quantize.dynamic.block.scaledgraph op with the graph compiler. -
Struct_grouped_quantize_dynamic_block_scaled_with_row_indices: Registers thequantization.grouped.quantize.dynamic.block.scaled.with_row_indicesgraph op with the graph compiler. -
Struct_grouped_quantize_nvfp4_row_scaled: Registers thequantization.grouped.quantize.nvfp4.row_scaledgraph op with the graph compiler. -
Struct_grouped_quantize_nvfp4_row_scaled_with_row_indices: Registers thequantization.grouped.quantize.nvfp4.row_scaled.with_row_indicesgraph op with the graph compiler. -
Struct_interleave_block_scales: Registers thequantization.interleave.block.scalesgraph op with the graph compiler. -
Struct_moe_gather_quantize_dynamic_block_scaled: Registers themo.moe.gather.quantize.dynamic.block.scaledgraph op with the graph compiler. -
Struct_moe_sort_gather_quantize_dynamic_block_scaled: Registers themo.moe.sort.gather.quantize.dynamic.block.scaledgraph op with the graph compiler. -
Struct_quantize_dynamic_block_scaled: Registers thequantization.quantize.dynamic.block.scaledgraph op with the graph compiler. -
Struct_quantize_dynamic_block_scaled_mxfp4: Registers thequantization.quantize.dynamic.block.scaled.mxfp4graph op with the graph compiler. -
Struct_quantize_dynamic_block_scaled_mxfp6: Registers thequantization.quantize.dynamic.block.scaled.mxfp6graph op. -
Struct_rms_norm_quantize_mxfp8_q_kv: Registers thequantization.rms_norm.quantize.mxfp8.q_kvgraph op with the graph compiler. -
Struct_rope_quantize_dynamic_block_scaled: Registers thequantization.rope.quantize.dynamic.block.scaledgraph op with the graph compiler. -
Struct_swiglu_quantize_dynamic_block_scaled: Registers thequantization.swiglu.quantize.dynamic.block.scaledgraph op with the graph compiler. -
Struct_unfused_qkv_matmul_ragged_paged_gguf_quantized: Registers thekv_cache.unfused_qkv_matmul.ragged.paged.gguf_quantizedgraph op with the graph compiler. -
VroomQ40Matmul: Registers thevroom_q4_0_matmulgraph op with the graph compiler. -
VroomQ40RepackWeights: Registers thevroom_q4_0_repack_weightsgraph op with the graph compiler. -
VroomQ4KMatmul: Registers thevroom_q4_k_matmulgraph op with the graph compiler. -
VroomQ4KRepackWeights: Registers thevroom_q4_k_repack_weightsgraph op with the graph compiler. -
VroomQ6KMatmul: Registers thevroom_q6_k_matmulgraph op with the graph compiler. -
VroomQ6KRepackWeights: Registers thevroom_q6_k_repack_weightsgraph op with the graph compiler.
Functions
-
composite_rms_norm_fused_quantize_dynamic_scaled_fp8_shape: Computes the output shapes for the fused RMS norm and dynamic scaled FP8 quantization op. -
ggml_q4_0_dequantize_shape: Computes the output shape for theggml_q4_0_dequantizegraph op. -
ggml_q4_k_dequantize_shape: Computes the output shape for theggml_q4_k_dequantizegraph op. -
ggml_q6_k_dequantize_shape: Computes the output shape for theggml_q6_k_dequantizegraph op. -
GPTQ_gpu_repack_b4_g128_desc_act_shape: Computes the output shape for theGPTQ_gpu_repack_b4_g128_desc_actgraph op. -
GPTQ_gpu_repack_b4_g128_shape: Computes the output shape for theGPTQ_gpu_repack_b4_g128graph op. -
qmatmul_b4_g128_shape: Computes the output shape for theqmatmul_b4_g128graph op. -
resize_bicubic_shape: Computes the output shape for themo.resize.bicubicgraph op. -
resize_linear_shape: Computes the output shape for themo.resize.lineargraph op. -
resize_nearest_shape: Computes the output shape for themo.resize.nearestgraph op. -
vroom_q4_0_matmul_shape: Computes the output shape for thevroom_q4_0_matmulgraph op. -
vroom_q4_0_repack_weights_shape: Computes the output shape for thevroom_q4_0_repack_weightsgraph op. -
vroom_q4_k_matmul_shape: Computes the output shape for thevroom_q4_k_matmulgraph op. -
vroom_q4_k_repack_weights_shape: Computes the output shape for thevroom_q4_k_repack_weightsgraph op. -
vroom_q6_k_matmul_shape: Computes the output shape for thevroom_q6_k_matmulgraph op. -
vroom_q6_k_repack_weights_shape: Computes the output shape for thevroom_q6_k_repack_weightsgraph op.