For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
DistributedAllGatherRMSNorm
struct DistributedAllGatherRMSNorm
Registers the mo.composite.distributed.allgather_rms_norm graph op with the graph compiler.
Implemented traits
Methods
execute
static def execute[Normed: TypeList[Normed.values], Residuals: TypeList[Residuals.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], Gammas: TypeList[Gammas.values], //, _trace_name: StaticString, epsilon: Float32, weight_offset: Float32, group_size: Int = Int(0), gather_axis: Int = Int(0), round_offset_gamma: Bool = False](outputs_normed: TensorTuple[*Normed.values], outputs_residual: TensorTuple[*Residuals.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], gammas: TensorTuple[*Gammas.values], dev_ctxs_input: DeviceContextArray)
Fused all-gather + RMSNorm (bf16 in/out, no quantization).
All-gathers inputs (per-device [shard_i, cols] row-shards) into the
full replicated [rows, cols] stream and RMSNorms every gathered row in
one launch: normed to outputs_normed, gathered residual to
outputs_residual (both full and replicated).
Limitations: - Maximum of 8 GPUs supported (matches MAX_GPUS in comm/sync.mojo). - Requires P2P; within a group, shard shapes may differ only in the gathered axis.
Parameters:
- Normed (
TypeList[values]): The element types ofoutputs_normed(inferred). - Residuals (
TypeList[values]): The element types ofoutputs_residual(inferred). - Ins (
TypeList[values]): The element types ofinputs(inferred). - Sigs (
TypeList[values]): The element types ofsignal_buffers(inferred). - Gammas (
TypeList[values]): The element types ofgammas(inferred). - _trace_name (
StaticString): Trace name for profiling. - epsilon (
Float32): RMSNorm epsilon, shared by every device. - weight_offset (
Float32): Gamma offset, shared by every device. - group_size (
Int): Number of contiguous devices per independent all-gather group; must be at least 2 and must evenly divide the total number of devices. Equal tonum_devicesfor a full-world collective. The builder always sets it; the0attribute default is not a usable value. - gather_axis (
Int): 0 to gather row shards, 1 column shards. - round_offset_gamma (
Bool): Whethergamma + weight_offsetis rounded todtypebefore the f32 multiply.
Args:
- outputs_normed (
TensorTuple[*Normed.values]): Per-device normed output (the group's gathered[rows, cols], replicated within the group). - outputs_residual (
TensorTuple[*Residuals.values]): Per-device gathered residual (same shape asoutputs_normed; the residual stream). - inputs (
TensorTuple[*Ins.values]): Per-device input row-shards to gather. - signal_buffers (
TensorTuple[*Sigs.values]): Per-device synchronization buffers. - gammas (
TensorTuple[*Gammas.values]): Per-device RMSNorm gamma weights (in_dtype, length cols). - dev_ctxs_input (
DeviceContextArray): Device contexts for participating GPUs.