IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

allreduce_rmsnorm

def allreduce_rmsnorm[in_dtype: DType, out_dtype: DType, scales_dtype: DType, ngpus: Int, in_layout: TensorLayout, in_origin: Origin[mut=in_origin.mut], in_engine: TensorEngine, //, multiply_before_cast: Bool = True, pdl_level: PDLLevel = PDLLevel.OVERLAP_AT_BEGINNING if (ngpus <= Int(2)) else PDLLevel.OFF](input_buffers: Array[TileTensor[in_dtype, in_layout, in_origin, Engine=in_engine], ngpus], output: TileTensor[out_dtype, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type], gamma: TileTensor[Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type], epsilon: Float32, weight_offset: Scalar[in_dtype], scale_ub: Float32, scale_output: TileTensor[scales_dtype, Engine=scale_output.Engine, address_space=scale_output.address_space, linear_idx_type=scale_output.linear_idx_type], rank_sigs: Array[Pointer[Signal, MutAnyOrigin], ngpus], ctx: DeviceContext, my_rank: Optional[Int] = None, scratch_bytes: Optional[Int] = None)

Fused allreduce + RMSNorm with optional FP8 quantization.

Combines a P2P allreduce across GPUs, RMSNorm, and (when the output dtype differs from the input dtype) FP8 dynamic quantization into a single kernel launch, eliminating the global memory round-trip between allreduce output and RMSNorm input. When out_dtype == in_dtype the quantization is skipped: the normalized value is written directly in the input dtype and scale_ub and scale_output are ignored.

Note: Both kernels finish all P2P reads of the input buffers before they exit (the 1-stage kernel with an end barrier, the 2-stage kernel through its mid barrier), so the inputs are free to be reused once this op retires. The output and scale buffers are safe to read only on the local GPU (stream ordering guarantees visibility). If a remote GPU needs to read these outputs, the caller must insert an explicit barrier.

Signal buffer sizing: 1-stage path (payload < threshold): size_ofSignal only. 2-stage path (payload > threshold): size_ofSignal + ceildiv(rows, ngpus) * cols * sizeof(out_dtype) (output) + align_up(ceildiv(rows, ngpus) * sizeof(scales_dtype), simd_width) (scales + pad, if quantizing)

Parameters:

  • ​in_dtype (DType): Input data type (e.g. bfloat16).
  • ​out_dtype (DType): Output data type. Either a float8 type (fuses quantization) or equal to in_dtype (no quantization).
  • ​scales_dtype (DType): Scale factor data type (e.g. float32). Ignored when out_dtype == in_dtype.
  • ​ngpus (Int): Number of GPUs participating.
  • ​in_layout (TensorLayout): Layout of the input TileTensors.
  • ​in_origin (Origin[mut=mut]): Origin of the input TileTensors.
  • ​multiply_before_cast (Bool): Matches ops.rms_norm. True applies gamma in float32 before the single cast to the output dtype; False casts the normalized value to in_dtype first and applies gamma in in_dtype. False requires out_dtype == in_dtype.
  • ​pdl_level (PDLLevel): Programmatic dependent launch level. The fused kernels wait for their predecessor before they signal their peers, and release the next kernel after the barrier that retires the peers' reads (ON) or right after the wait (OVERLAP_AT_BEGINNING). On the split path the allreduce always releases the norm at its exit, and only the norm honors OVERLAP_AT_BEGINNING. That level requires the next kernel not to write this kernel's inputs before its own PDL wait, because peers may still be reading them. The Lamport branch always uses PDL.

Args:

Was this page helpful?