For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
allreduce_rmsnorm
def allreduce_rmsnorm[in_dtype: DType, out_dtype: DType, scales_dtype: DType, ngpus: Int, in_layout: TensorLayout, in_origin: Origin[mut=in_origin.mut], in_engine: TensorEngine, //, multiply_before_cast: Bool = True, pdl_level: PDLLevel = PDLLevel.OVERLAP_AT_BEGINNING if (ngpus <= Int(2)) else PDLLevel.OFF](input_buffers: Array[TileTensor[in_dtype, in_layout, in_origin, Engine=in_engine], ngpus], output: TileTensor[out_dtype, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type], gamma: TileTensor[Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type], epsilon: Float32, weight_offset: Scalar[in_dtype], scale_ub: Float32, scale_output: TileTensor[scales_dtype, Engine=scale_output.Engine, address_space=scale_output.address_space, linear_idx_type=scale_output.linear_idx_type], rank_sigs: Array[Pointer[Signal, MutAnyOrigin], ngpus], ctx: DeviceContext, my_rank: Optional[Int] = None, scratch_bytes: Optional[Int] = None)
Fused allreduce + RMSNorm with optional FP8 quantization.
Combines a P2P allreduce across GPUs, RMSNorm, and (when the output dtype
differs from the input dtype) FP8 dynamic quantization into a single
kernel launch, eliminating the global memory round-trip between allreduce
output and RMSNorm input. When out_dtype == in_dtype the quantization is
skipped: the normalized value is written directly in the input dtype and
scale_ub and scale_output are ignored.
Note: Both kernels finish all P2P reads of the input buffers before they exit (the 1-stage kernel with an end barrier, the 2-stage kernel through its mid barrier), so the inputs are free to be reused once this op retires. The output and scale buffers are safe to read only on the local GPU (stream ordering guarantees visibility). If a remote GPU needs to read these outputs, the caller must insert an explicit barrier.
Signal buffer sizing: 1-stage path (payload < threshold): size_ofSignal only. 2-stage path (payload > threshold): size_ofSignal + ceildiv(rows, ngpus) * cols * sizeof(out_dtype) (output) + align_up(ceildiv(rows, ngpus) * sizeof(scales_dtype), simd_width) (scales + pad, if quantizing)
Parameters:
- in_dtype (
DType): Input data type (e.g. bfloat16). - out_dtype (
DType): Output data type. Either a float8 type (fuses quantization) or equal toin_dtype(no quantization). - scales_dtype (
DType): Scale factor data type (e.g. float32). Ignored whenout_dtype == in_dtype. - ngpus (
Int): Number of GPUs participating. - in_layout (
TensorLayout): Layout of the input TileTensors. - in_origin (
Origin[mut=mut]): Origin of the input TileTensors. - multiply_before_cast (
Bool): Matchesops.rms_norm. True applies gamma in float32 before the single cast to the output dtype; False casts the normalized value toin_dtypefirst and applies gamma inin_dtype. False requiresout_dtype == in_dtype. - pdl_level (
PDLLevel): Programmatic dependent launch level. The fused kernels wait for their predecessor before they signal their peers, and release the next kernel after the barrier that retires the peers' reads (ON) or right after the wait (OVERLAP_AT_BEGINNING). On the split path the allreduce always releases the norm at its exit, and only the norm honorsOVERLAP_AT_BEGINNING. That level requires the next kernel not to write this kernel's inputs before its own PDL wait, because peers may still be reading them. The Lamport branch always uses PDL.
Args:
- input_buffers (
Array[TileTensor[in_dtype, in_layout, in_origin, Engine=in_engine], ngpus]): Per-GPU input buffers as TileTensors. - output (
TileTensor[out_dtype, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type]): Output buffer (FP8 values when quantizing, elsein_dtype). - gamma (
TileTensor[Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type]): RMSNorm gamma weights (1D TileTensor of length cols), inin_dtypeor, on a float32 input, bfloat16/float16 (widened exactly in registers). - epsilon (
Float32): RMSNorm epsilon for numerical stability. - weight_offset (
Scalar[in_dtype]): Additive offset for gamma weights. - scale_ub (
Float32): Upper bound for FP8 scale clamping (ignored when not quantizing). - scale_output (
TileTensor[scales_dtype, Engine=scale_output.Engine, address_space=scale_output.address_space, linear_idx_type=scale_output.linear_idx_type]): Output buffer for per-row FP8 scales (ignored, and not written, when not quantizing). - rank_sigs (
Array[Pointer[Signal, MutAnyOrigin], ngpus]): Per-GPU signal pointers for synchronization. - ctx (
DeviceContext): Device context for this GPU. - my_rank (
Optional[Int]): This GPU's index ininput_buffersandrank_sigs. Defaults toctx.device().id, which is right only when device ordinals match list positions. - scratch_bytes (
Optional[Int]): Bytes after eachSignalheader that kernels may stage through. Paths that need more take the 1-stage kernel. None means the caller sized it for every path.