IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

DistributedAllGatherRMSNormQuantMXFP8

struct DistributedAllGatherRMSNormQuantMXFP8

Registers mo.composite.distributed.allgather_rms_norm_quant_mxfp8.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[Normed: TypeList[Normed.values], Quant: TypeList[Quant.values], Scales: TypeList[Scales.values], Residuals: TypeList[Residuals.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], Gammas: TypeList[Gammas.values], //, _trace_name: StaticString, epsilon: Float32, weight_offset: Float32, group_size: Int = Int(0), gather_quant_only: Bool = False](outputs_normed: TensorTuple[*Normed.values], outputs_quant: TensorTuple[*Quant.values], outputs_scale: TensorTuple[*Scales.values], outputs_residual: TensorTuple[*Residuals.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], gammas: TensorTuple[*Gammas.values], dev_ctxs_input: DeviceContextArray)

Fused all-gather + RMSNorm that also emits an MXFP8 copy.

DistributedAllGatherRMSNorm plus outputs_quant/outputs_scale. The quantize rides the collective's epilogue on the same bf16 that lands in outputs_normed, so it is byte-identical to a standalone quantize.

Limitations: - Targets AMD (CDNA4): the fallback calls quantize_mx_amd and the scale layout is its rank-2 one -- what block_scaled_matmul_amd takes as a_scales, not the SM100 SF atom and not the preshuffled order the _preb kernel needs. - Maximum of 8 GPUs; requires P2P.

Parameters:

  • ​Normed (TypeList[values]): The element types of outputs_normed (inferred).
  • ​Quant (TypeList[values]): The element types of outputs_quant (inferred).
  • ​Scales (TypeList[values]): The element types of outputs_scale (inferred).
  • ​Residuals (TypeList[values]): The element types of outputs_residual (inferred).
  • ​Ins (TypeList[values]): The element types of inputs (inferred).
  • ​Sigs (TypeList[values]): The element types of signal_buffers (inferred).
  • ​Gammas (TypeList[values]): The element types of gammas (inferred).
  • ​_trace_name (StaticString): Trace name for profiling.
  • ​epsilon (Float32): RMSNorm epsilon, shared by every device.
  • ​weight_offset (Float32): Gamma offset, shared by every device.
  • ​group_size (Int): Devices per independent all-gather group; see DistributedAllGatherRMSNorm.
  • ​gather_quant_only (Bool): When every group gathers at least AG_QUANT_ONLY_THRESHOLD bytes of bf16 rows (~1024 rows), norm and quantize only this device's row shard and gather just the MXFP8 values and scales, about half the bytes of the bf16 residual; below it the extra scales gather costs more than the bytes save. Each shard is normed with its owner's gamma, so the result matches the default only when every device in a group holds the same gamma. outputs_normed and outputs_residual then hold valid rows only for this device's shard. The fused path is unchanged.

Args:

Was this page helpful?