IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

DistributedAllReduceRMSNormExitConcat

struct DistributedAllReduceRMSNormExitConcat

Registers mo.composite.distributed.allreduce_rms_norm_exit_concat.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[Out: TypeList[Out.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], Gammas: TypeList[Gammas.values], GateUps: TypeList[GateUps.values], //, _trace_name: StaticString, epsilon: Float32, beta: Float32, linear_beta: Float32](outputs: TensorTuple[*Out.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], gammas: TensorTuple[*Gammas.values], gate_ups: TensorTuple[*GateUps.values], dev_ctxs_input: DeviceContextArray)

Reduces, norms, re-shards and activates the exit's tail: one launch at decode sizes, the plain allreduce plus a norm above the size gate, and the same chain under a vendor-CCL build or without P2P access.

Limitations: - The fused arm requires P2P access between the participating GPUs; without it, and under a vendor-CCL build, the launcher runs the chain instead. The group must be the full world, of at most 8 devices. - float32 or bf16 partials and bf16 gated projections only.

Parameters:

  • ​Out (TypeList[values]): The element types of outputs (inferred).
  • ​Ins (TypeList[values]): The element types of inputs (inferred).
  • ​Sigs (TypeList[values]): The element types of signal_buffers (inferred).
  • ​Gammas (TypeList[values]): The element types of gammas (inferred).
  • ​GateUps (TypeList[values]): The element types of gate_ups (inferred).
  • ​_trace_name (StaticString): Trace name for profiling.
  • ​epsilon (Float32): RMSNorm epsilon, shared by every device.
  • ​beta (Float32): The SiTU gate's beta, shared by every device.
  • ​linear_beta (Float32): The SiTU up-half saturation beta; 0 disables it.

Args:

  • ​outputs (TensorTuple[*Out.values]): Per-device [rows, cols / N + half] bf16 activation.
  • ​inputs (TensorTuple[*Ins.values]): Per-device float32 or bf16 [rows, cols] partial sums.
  • ​signal_buffers (TensorTuple[*Sigs.values]): Per-device synchronization buffers; the bytes behind each Signal header are the collective's staging scratch (the plain allreduce's partition, which is all a large prefill batch needs).
  • ​gammas (TensorTuple[*Gammas.values]): Per-device [cols] norm weights, in the inputs' dtype or, on float32 inputs, bfloat16/float16 (widened in registers). Every device must pass the same replicated weight: one device normalizes the columns it holds for every device's rows.
  • ​gate_ups (TensorTuple[*GateUps.values]): Per-device bf16 [rows, 2 * half] shared gate/up projection, gate half first.
  • ​dev_ctxs_input (DeviceContextArray): Device contexts for participating GPUs.

Was this page helpful?