For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
allreduce_rmsnorm
def allreduce_rmsnorm[in_dtype: DType, out_dtype: DType, scales_dtype: DType, ngpus: Int, in_layout: TensorLayout, in_origin: Origin[mut=in_origin.mut], //](input_buffers: Array[TileTensor[in_dtype, in_layout, in_origin], ngpus], output: TileTensor[out_dtype, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type], gamma: TileTensor[in_dtype, Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type], epsilon: Float32, weight_offset: Scalar[in_dtype], scale_ub: Float32, scale_output: TileTensor[scales_dtype, Engine=scale_output.Engine, address_space=scale_output.address_space, linear_idx_type=scale_output.linear_idx_type], rank_sigs: Array[Pointer[Signal, MutAnyOrigin], ngpus], ctx: DeviceContextBase)
Fused allreduce + RMSNorm with optional FP8 quantization.
Combines a P2P allreduce across GPUs, RMSNorm, and (when the output dtype
differs from the input dtype) FP8 dynamic quantization into a single
kernel launch, eliminating the global memory round-trip between allreduce
output and RMSNorm input. When out_dtype == in_dtype the quantization is
skipped: the normalized value is written directly in the input dtype and
scale_ub and scale_output are ignored.
Note: Both kernels finish all P2P reads of the input buffers before they exit (the 1-stage kernel with an end barrier, the 2-stage kernel through its mid barrier), so the inputs are free to be reused once this op retires. The output and scale buffers are safe to read only on the local GPU (stream ordering guarantees visibility). If a remote GPU needs to read these outputs, the caller must insert an explicit barrier.
Signal buffer sizing: 1-stage path (payload < threshold): size_ofSignal only. 2-stage path (payload > threshold): size_ofSignal + ceildiv(rows, ngpus) * cols * sizeof(out_dtype) (output) + align_up(ceildiv(rows, ngpus) * sizeof(scales_dtype), simd_width) (scales + pad, if quantizing)
Parameters:
- in_dtype (
DType): Input data type (e.g. bfloat16). - out_dtype (
DType): Output data type. Either a float8 type (fuses quantization) or equal toin_dtype(no quantization). - scales_dtype (
DType): Scale factor data type (e.g. float32). Ignored whenout_dtype == in_dtype. - ngpus (
Int): Number of GPUs participating. - in_layout (
TensorLayout): Layout of the input TileTensors. - in_origin (
Origin[mut=in_origin.mut]): Origin of the input TileTensors.
Args:
- input_buffers (
Array[TileTensor[in_dtype, in_layout, in_origin], ngpus]): Per-GPU input buffers as TileTensors. - output (
TileTensor[out_dtype, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type]): Output buffer (FP8 values when quantizing, elsein_dtype). - gamma (
TileTensor[in_dtype, Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type]): RMSNorm gamma weights (1D TileTensor of length cols). - epsilon (
Float32): RMSNorm epsilon for numerical stability. - weight_offset (
Scalar[in_dtype]): Additive offset for gamma weights. - scale_ub (
Float32): Upper bound for FP8 scale clamping (ignored when not quantizing). - scale_output (
TileTensor[scales_dtype, Engine=scale_output.Engine, address_space=scale_output.address_space, linear_idx_type=scale_output.linear_idx_type]): Output buffer for per-row FP8 scales (ignored, and not written, when not quantizing). - rank_sigs (
Array[Pointer[Signal, MutAnyOrigin], ngpus]): Per-GPU signal pointers for synchronization. - ctx (
DeviceContextBase): Device context for this GPU.