For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
lamport_allreduce_rmsnorm
def lamport_allreduce_rmsnorm[dtype: DType, ngpus: Int](my_rank: Int, src: ImmPointer[Scalar[dtype], ImmutAnyOrigin], dst: MutPointer[Scalar[dtype], MutAnyOrigin], gamma: ImmPointer[Scalar[dtype], ImmutAnyOrigin], rank_sigs: Array[Pointer[Signal, MutAnyOrigin], ngpus], rows: Int, cols: Int, epsilon: Scalar[dtype], ctx: DeviceContext)
Per-device fused Lamport allreduce (sum) + RMSNorm (high-perf protocol).
src/dst are [rows, cols] (tokens x hidden); RMSNorm runs over cols.
Uses the device-resident Signal.lamport_region / lamport_state (so the
signal buffers must be lamport_init-ed once before the first call, then one
host sync). The generation flag advances in-kernel, so repeated calls --
including CUDA-graph replay -- rotate the buffers correctly with no per-call
fill or phase argument.
The launch carries the PDL attribute, and the kernel waits for its predecessor before any read and releases its dependents at entry so a following GEMM overlaps the whole AR+RMSNorm.
Parameters: