IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

allreduce_lamport_rmsnorm

Fused one-shot Lamport allreduce + RMSNorm (high-perf protocol).

Grafts the barrier-free Lamport comm protocol from comm/lamport.mojo + comm/allreduce.mojo (in-register local pack, poll-all-peers arrival, a sum in global rank order, 3-way generation rotation, fused clear, the device-resident Signal.lamport_region / lamport_state) onto a row-blocked RMSNorm epilogue so the whole TP allreduce+RMSNorm producer is a single kernel.

Layout: with the 128-bit Lamport pack (atomic_width = 16B / sizeof(dtype) = 8 for bf16) one thread owns exactly one pack = one column group of a row (token). By default one block owns whole rows. For decode-sized messages on 8 GPUs, where pushing every pack of a row from one SM is the bottleneck, a row is instead split across a thread-block cluster of smaller blocks. After the allreduce reduces each pack in-register, the block (or cluster) reduces the row's sum-of-squares and normalizes.

An optional residual is added to the reduced sum before the norm, and the pre-norm sum is written out as the new residual stream. That is the form the graph compiler's fused allreduce + add + RMSNorm op dispatches to for latency-bound payloads (comm.allreduce_residual_rmsnorm).

PDL is on by default: wait for the producer, then trigger the consumer immediately (early trigger) so a following GEMM overlaps this kernel; consumer correctness is its own wait_on_dependent_grids. The signal buffers must be sentinel-initialized once (lamport_init) before the first call.

Functions​

Was this page helpful?