IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

lamport_allreduce_rmsnorm

def lamport_allreduce_rmsnorm[dtype: DType, ngpus: Int](my_rank: Int, src: ImmPointer[Scalar[dtype], ImmutAnyOrigin], dst: MutPointer[Scalar[dtype], MutAnyOrigin], gamma: ImmPointer[Scalar[dtype], ImmutAnyOrigin], rank_sigs: Array[Pointer[Signal, MutAnyOrigin], ngpus], rows: Int, cols: Int, epsilon: Scalar[dtype], ctx: DeviceContext)

Per-device fused Lamport allreduce (sum) + RMSNorm (high-perf protocol).

src/dst are [rows, cols] (tokens x hidden); RMSNorm runs over cols. Uses the device-resident Signal.lamport_region / lamport_state (so the signal buffers must be lamport_init-ed once before the first call, then one host sync). The generation flag advances in-kernel, so repeated calls -- including CUDA-graph replay -- rotate the buffers correctly with no per-call fill or phase argument.

The launch carries the PDL attribute, and the kernel waits for its predecessor before any read and releases its dependents at entry so a following GEMM overlaps the whole AR+RMSNorm.

Parameters:

  • ​dtype (DType): Element type (bf16/fp16/fp32 transport).
  • ​ngpus (Int): Number of participating GPUs.

Was this page helpful?