IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

DistributedAllReducePostNormResidualRMSNorm

struct DistributedAllReducePostNormResidualRMSNorm

Registers mo.composite.distributed.allreduce_post_norm_residual_rms_norm.

One Lamport launch per device sums inputs, norms the sum with gammas, joins it to residuals as mix[0] * normed + mix[1] * residual and norms the join with gammas2; see comm.allreduce_lamport_rmsnorm.lamport_allreduce_post_norm_residual_rmsnorm.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[Joined: TypeList[Joined.values], Normed: TypeList[Normed.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], Residuals: TypeList[Residuals.values], Gammas: TypeList[Gammas.values], Gammas2: TypeList[Gammas2.values], Mixes: TypeList[Mixes.values], //, _trace_name: StaticString, epsilon: Float32](outputs_joined: TensorTuple[*Joined.values], outputs_normed: TensorTuple[*Normed.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], residuals: TensorTuple[*Residuals.values], gammas: TensorTuple[*Gammas.values], gammas2: TensorTuple[*Gammas2.values], mixes: TensorTuple[*Mixes.values], dev_ctxs_input: DeviceContextArray)

Fused allreduce + post-norm residual join (bf16 in/out).

Parameters:

Args:

Was this page helpful?