IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

DistributedAllGatherGatedResidualRMSNorm

struct DistributedAllGatherGatedResidualRMSNorm

Registers mo.composite.distributed.allgather_gated_residual_rms_norm.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[Streams: TypeList[Streams.values], Normed: TypeList[Normed.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], Residuals: TypeList[Residuals.values], PostWeights: TypeList[PostWeights.values], Gates: TypeList[Gates.values], NormWeights: TypeList[NormWeights.values], //, _trace_name: StaticString, post_eps: Float32, norm_eps: Float32, temperature: Float32, gate_eps: Float32, scale_branch: Bool](outputs_stream: TensorTuple[*Streams.values], outputs_normed: TensorTuple[*Normed.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], residuals: TensorTuple[*Residuals.values], post_weights: TensorTuple[*PostWeights.values], gates: TensorTuple[*Gates.values], norm_weights: TensorTuple[*NormWeights.values], dev_ctxs_input: DeviceContextArray)

Gathers a branch's column shards and joins it to each device's stream.

Parameters:

  • ​Streams (TypeList[values]): The element types of outputs_stream (inferred).
  • ​Normed (TypeList[values]): The element types of outputs_normed (inferred).
  • ​Ins (TypeList[values]): The element types of inputs (inferred).
  • ​Sigs (TypeList[values]): The element types of signal_buffers (inferred).
  • ​Residuals (TypeList[values]): The element types of residuals (inferred).
  • ​PostWeights (TypeList[values]): The element types of post_weights (inferred).
  • ​Gates (TypeList[values]): The element types of gates (inferred).
  • ​NormWeights (TypeList[values]): The element types of norm_weights (inferred).
  • ​_trace_name (StaticString): Trace name for profiling.
  • ​post_eps (Float32): The branch norm's epsilon.
  • ​norm_eps (Float32): The next pre-norm's epsilon.
  • ​temperature (Float32): The gate's temperature.
  • ​gate_eps (Float32): The floor under the residual coefficient's root.
  • ​scale_branch (Bool): Whether the branch is scaled by 1 + post_weight before its norm.

Args:

Was this page helpful?