For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
DistributedAllGatherGatedResidualRMSNorm
struct DistributedAllGatherGatedResidualRMSNorm
Registers mo.composite.distributed.allgather_gated_residual_rms_norm.
Implemented traits
Methods
execute
static def execute[Streams: TypeList[Streams.values], Normed: TypeList[Normed.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], Residuals: TypeList[Residuals.values], PostWeights: TypeList[PostWeights.values], Gates: TypeList[Gates.values], NormWeights: TypeList[NormWeights.values], //, _trace_name: StaticString, post_eps: Float32, norm_eps: Float32, temperature: Float32, gate_eps: Float32, scale_branch: Bool](outputs_stream: TensorTuple[*Streams.values], outputs_normed: TensorTuple[*Normed.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], residuals: TensorTuple[*Residuals.values], post_weights: TensorTuple[*PostWeights.values], gates: TensorTuple[*Gates.values], norm_weights: TensorTuple[*NormWeights.values], dev_ctxs_input: DeviceContextArray)
Gathers a branch's column shards and joins it to each device's stream.
Parameters:
- Streams (
TypeList[values]): The element types ofoutputs_stream(inferred). - Normed (
TypeList[values]): The element types ofoutputs_normed(inferred). - Ins (
TypeList[values]): The element types ofinputs(inferred). - Sigs (
TypeList[values]): The element types ofsignal_buffers(inferred). - Residuals (
TypeList[values]): The element types ofresiduals(inferred). - PostWeights (
TypeList[values]): The element types ofpost_weights(inferred). - Gates (
TypeList[values]): The element types ofgates(inferred). - NormWeights (
TypeList[values]): The element types ofnorm_weights(inferred). - _trace_name (
StaticString): Trace name for profiling. - post_eps (
Float32): The branch norm's epsilon. - norm_eps (
Float32): The next pre-norm's epsilon. - temperature (
Float32): The gate's temperature. - gate_eps (
Float32): The floor under the residual coefficient's root. - scale_branch (
Bool): Whether the branch is scaled by1 + post_weightbefore its norm.
Args:
- outputs_stream (
TensorTuple[*Streams.values]): Per-device new residual stream[rows, cols]. - outputs_normed (
TensorTuple[*Normed.values]): Per-device next pre-norm of it[rows, cols]. - inputs (
TensorTuple[*Ins.values]): Per-device branch column shards[rows, cols / N]. - signal_buffers (
TensorTuple[*Sigs.values]): Per-device synchronization buffers. - residuals (
TensorTuple[*Residuals.values]): Per-device incoming residual stream[rows, cols]. - post_weights (
TensorTuple[*PostWeights.values]): Per-device branch input scale weight[cols]. - gates (
TensorTuple[*Gates.values]): Per-device residual gate logits[cols]. - norm_weights (
TensorTuple[*NormWeights.values]): Per-device next pre-norm weight[cols]. - dev_ctxs_input (
DeviceContextArray): Device contexts for participating GPUs.