For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
DistributedAllReduceRMSNormExitConcat
struct DistributedAllReduceRMSNormExitConcat
Registers mo.composite.distributed.allreduce_rms_norm_exit_concat.
Implemented traits
Methods
execute
static def execute[Out: TypeList[Out.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], Gammas: TypeList[Gammas.values], GateUps: TypeList[GateUps.values], //, _trace_name: StaticString, epsilon: Float32, beta: Float32, linear_beta: Float32](outputs: TensorTuple[*Out.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], gammas: TensorTuple[*Gammas.values], gate_ups: TensorTuple[*GateUps.values], dev_ctxs_input: DeviceContextArray)
Reduces, norms, re-shards and activates the exit's tail: one launch at decode sizes, the plain allreduce plus a norm above the size gate, and the same chain under a vendor-CCL build or without P2P access.
Limitations: - The fused arm requires P2P access between the participating GPUs; without it, and under a vendor-CCL build, the launcher runs the chain instead. The group must be the full world, of at most 8 devices. - float32 or bf16 partials and bf16 gated projections only.
Parameters:
- Out (
TypeList[values]): The element types ofoutputs(inferred). - Ins (
TypeList[values]): The element types ofinputs(inferred). - Sigs (
TypeList[values]): The element types ofsignal_buffers(inferred). - Gammas (
TypeList[values]): The element types ofgammas(inferred). - GateUps (
TypeList[values]): The element types ofgate_ups(inferred). - _trace_name (
StaticString): Trace name for profiling. - epsilon (
Float32): RMSNorm epsilon, shared by every device. - beta (
Float32): The SiTU gate's beta, shared by every device. - linear_beta (
Float32): The SiTU up-half saturation beta; 0 disables it.
Args:
- outputs (
TensorTuple[*Out.values]): Per-device[rows, cols / N + half]bf16 activation. - inputs (
TensorTuple[*Ins.values]): Per-device float32 or bf16[rows, cols]partial sums. - signal_buffers (
TensorTuple[*Sigs.values]): Per-device synchronization buffers; the bytes behind eachSignalheader are the collective's staging scratch (the plain allreduce's partition, which is all a large prefill batch needs). - gammas (
TensorTuple[*Gammas.values]): Per-device[cols]norm weights, in the inputs' dtype or, on float32 inputs, bfloat16/float16 (widened in registers). Every device must pass the same replicated weight: one device normalizes the columns it holds for every device's rows. - gate_ups (
TensorTuple[*GateUps.values]): Per-device bf16[rows, 2 * half]shared gate/up projection, gate half first. - dev_ctxs_input (
DeviceContextArray): Device contexts for participating GPUs.