For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
DistributedAllReducePostNormResidualRMSNorm
struct DistributedAllReducePostNormResidualRMSNorm
Registers mo.composite.distributed.allreduce_post_norm_residual_rms_norm.
One Lamport launch per device sums inputs, norms the sum with gammas,
joins it to residuals as mix[0] * normed + mix[1] * residual and
norms the join with gammas2; see
comm.allreduce_lamport_rmsnorm.lamport_allreduce_post_norm_residual_rmsnorm.
Implemented traits
Methods
execute
static def execute[Joined: TypeList[Joined.values], Normed: TypeList[Normed.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], Residuals: TypeList[Residuals.values], Gammas: TypeList[Gammas.values], Gammas2: TypeList[Gammas2.values], Mixes: TypeList[Mixes.values], //, _trace_name: StaticString, epsilon: Float32](outputs_joined: TensorTuple[*Joined.values], outputs_normed: TensorTuple[*Normed.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], residuals: TensorTuple[*Residuals.values], gammas: TensorTuple[*Gammas.values], gammas2: TensorTuple[*Gammas2.values], mixes: TensorTuple[*Mixes.values], dev_ctxs_input: DeviceContextArray)
Fused allreduce + post-norm residual join (bf16 in/out).
Parameters:
- Joined (
TypeList[values]): The element types ofoutputs_joined(inferred). - Normed (
TypeList[values]): The element types ofoutputs_normed(inferred). - Ins (
TypeList[values]): The element types ofinputs(inferred). - Sigs (
TypeList[values]): The element types ofsignal_buffers(inferred). - Residuals (
TypeList[values]): The element types ofresiduals(inferred). - Gammas (
TypeList[values]): The element types ofgammas(inferred). - Gammas2 (
TypeList[values]): The element types ofgammas2(inferred). - Mixes (
TypeList[values]): The element types ofmixes(inferred). - _trace_name (
StaticString): Trace name for profiling. - epsilon (
Float32): Both norms' epsilon.
Args:
- outputs_joined (
TensorTuple[*Joined.values]): Per-device joined residual stream. - outputs_normed (
TensorTuple[*Normed.values]): Per-device join normed withgammas2. - inputs (
TensorTuple[*Ins.values]): Per-device[rows, cols]partial sums to reduce. - signal_buffers (
TensorTuple[*Sigs.values]): Per-device synchronization buffers. - residuals (
TensorTuple[*Residuals.values]): Per-device residual stream, replicated. - gammas (
TensorTuple[*Gammas.values]): Per-device first norm weight,[cols]. - gammas2 (
TensorTuple[*Gammas2.values]): Per-device second norm weight,[cols]. - mixes (
TensorTuple[*Mixes.values]): Per-device[2]float32 weights of the normed sum and of the residual. - dev_ctxs_input (
DeviceContextArray): Device contexts for participating GPUs.