For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
block_scaled_quantization
Provides block-scaled quantization kernels for NVFP4, MXFP4, and MXFP8.
comptime values
logger
comptime logger = Logger(stdout, prefix=String(""), source_location=False)
nvfp4_row_input_fn_type
comptime nvfp4_row_input_fn_type = def[dtype: DType, width: SIMDLength](SIMD[dtype, width]) thin -> SIMD[dtype, width]
Elementwise transform applied to each input vector before quantizing.
grouped_quantize_nvfp4_row_amax quantizes input_fn(x) in place of x,
for both the row amax and the payload; the result is rounded to the input
dtype, as the graph op it replaces would round it. The graph op passes the
Nemotron-H relu(x) * relu(x) built from nn.activations.relu.
Functions
-
block_scaled_matmul: Dispatches the SM100 block-scaled matmul for NVFP4 or MXFP8, selecting between the Mojo heuristic kernel and the vendor fallback. -
block_scaled_matmul_with_epilogue: Our sm100 block scaled matmul kernel still does not support fusion of elementwise operations. This is a temporary implementation that uses our sm100 block scaled matmul kernel and dispatch a separate epilogue kernel to apply the elementwise operations. Callers must allocatec; withhas_epilogue_fnorhas_compute_fnthe matmul result is written intocand then read back by the closure. -
block_scales_interleave: Reinterleaves rank-2 scale factors into the 5D TCGEN layout on SM100 hardware. -
block_scales_interleave_fp4: Launches the SM100 kernel that reinterleaves rank-2 scale factors into the 5D TCGEN layout. -
block_scales_interleave_fp4_kernel: GPU kernel that reinterleaves rank-2 scale factors into the 5D TCGEN layout. -
grouped_matmul_block_scaled_amd: Launches the grouped per-expert MXFP4 block-scaled matmul kernel on AMD CDNA4. -
grouped_matmul_block_scaled_amd_kernel: -
grouped_quantize_dynamic_scaled_fp4_async: Launches the grouped per-expert quantization kernel for NVFP4/MXFP4/MXFP8 on SM100 hardware. -
grouped_quantize_dynamic_scaled_fp4_async_kernel: GPU kernel that quantizes per-expert BF16 activation tiles to NVFP4/MXFP4/MXFP8 with TMA-based scale-factor stores. -
grouped_quantize_nvfp4_row_amax: Quantizes grouped BF16 rows to NVFP4, each row with its own scale. -
grouped_quantize_nvfp4_row_amax_kernel: Quantizes BF16 rows to NVFP4, each with its own global scale. -
matmul_dynamic_block_scaled_amd: Launches the AMD CDNA4 MXFP4 block-scaled matmul kernel. -
matmul_dynamic_block_scaled_amd_kernel: -
moe_cast_window_bf16: Launches_moe_cast_window_bf16_kerneloverinput's rows. -
moe_gather_quantize_mx_amd: Gathers each sorted MoE row and quantizes it to MXFP8 in one launch. -
moe_sort_gather_quantize_fused_compilable: Whether the one-launch sort-fused kernel can be instantiated at all. -
moe_sort_gather_quantize_fused_supported: Whether the one-launch sort+gather+quantize covers this launch. -
moe_sort_gather_quantize_mx_amd: Sorts, gathers and MX-quantizes the MoE routing glue in one launch. -
moe_source_scatter_quantize_mx_amd: Scatters each token's MXFP8-quantized row to its sorted slots. -
moe_source_scatter_quantize_supported: Whether the source-major scatter+quantize kernel covers this launch. -
naive_block_scaled_matmul: Reference block-scaled matmul that emulates TCGEN scale-factor accumulation on SM100 hardware. -
naive_block_scaled_matmul_kernel: Naive GPU kernel that emulates a block-scaled matmul using TCGEN scale factors. -
quantize_dynamic_block_scaled: Dispatches dynamic block-scaled quantization to the appropriate hardware-specific kernel. -
quantize_dynamic_block_scaled_mxfp4: Launches the AMD CDNA4 MXFP4 quantization kernel over a flat input buffer. -
quantize_dynamic_block_scaled_mxfp4_kernel: -
quantize_dynamic_scaled_async_kernel: GPU kernel that quantizes BF16 tiles to NVFP4 or MXFP8 using TMA async copies and warp-specialized PDL. -
quantize_dynamic_scaled_fp4_async: Launches the TMA-based NVFP4 or MXFP8 quantization kernel on SM100 hardware. -
quantize_dynamic_scaled_fp4fp8: Launches the SM100 kernel that quantizes BF16 input to NVFP4, MXFP4, or MXFP8 with per-block scale factors. -
quantize_dynamic_scaled_fp4fp8_kernel: -
quantize_mx_amd: Quantize BF16 activations to MXFP4 or MXFP8 on AMD CDNA4 (MI355X). -
quantize_mxfp8_lane_group: Quantizes one thread's slice of an MX block to MXFP8, cooperatively. -
rms_norm_quantize_mxfp8_q_kv: RMS-normalizes the q and kv parts of each row and quantizes the q part to MXFP8 with block scales. -
rms_norm_quantize_mxfp8_q_kv_kernel: -
rope_quantize_dynamic_block_scaled: Rotates the trailing columns of each head by RoPE and quantizes the result to NVFP4 or MXFP8 with block scales. -
swiglu_quantize_dynamic_block_scaled: Quantizes the gated activation of a[gate | up]input to NVFP4, MXFP4, or MXFP8 with block scales. -
swiglu_quantize_mx_amd: Quantizes the gated activation of a BF16[gate | up]tensor to MXFP4 or MXFP8 on AMD CDNA4 (MI355X).