For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
DistributedAllGatherRMSNormQuantMXFP8
struct DistributedAllGatherRMSNormQuantMXFP8
Registers mo.composite.distributed.allgather_rms_norm_quant_mxfp8.
Implemented traits
Methods
execute
static def execute[Normed: TypeList[Normed.values], Quant: TypeList[Quant.values], Scales: TypeList[Scales.values], Residuals: TypeList[Residuals.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], Gammas: TypeList[Gammas.values], //, _trace_name: StaticString, epsilon: Float32, weight_offset: Float32, group_size: Int = Int(0), gather_quant_only: Bool = False](outputs_normed: TensorTuple[*Normed.values], outputs_quant: TensorTuple[*Quant.values], outputs_scale: TensorTuple[*Scales.values], outputs_residual: TensorTuple[*Residuals.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], gammas: TensorTuple[*Gammas.values], dev_ctxs_input: DeviceContextArray)
Fused all-gather + RMSNorm that also emits an MXFP8 copy.
DistributedAllGatherRMSNorm plus outputs_quant/outputs_scale. The
quantize rides the collective's epilogue on the same bf16 that lands in
outputs_normed, so it is byte-identical to a standalone quantize.
Limitations:
- Targets AMD (CDNA4): the fallback calls quantize_mx_amd and
the scale layout is its rank-2 one -- what
block_scaled_matmul_amd takes as a_scales, not the SM100 SF
atom and not the preshuffled order the _preb kernel needs.
- Maximum of 8 GPUs; requires P2P.
Parameters:
- Normed (
TypeList[values]): The element types ofoutputs_normed(inferred). - Quant (
TypeList[values]): The element types ofoutputs_quant(inferred). - Scales (
TypeList[values]): The element types ofoutputs_scale(inferred). - Residuals (
TypeList[values]): The element types ofoutputs_residual(inferred). - Ins (
TypeList[values]): The element types ofinputs(inferred). - Sigs (
TypeList[values]): The element types ofsignal_buffers(inferred). - Gammas (
TypeList[values]): The element types ofgammas(inferred). - _trace_name (
StaticString): Trace name for profiling. - epsilon (
Float32): RMSNorm epsilon, shared by every device. - weight_offset (
Float32): Gamma offset, shared by every device. - group_size (
Int): Devices per independent all-gather group; seeDistributedAllGatherRMSNorm. - gather_quant_only (
Bool): When every group gathers at leastAG_QUANT_ONLY_THRESHOLDbytes of bf16 rows (~1024 rows), norm and quantize only this device's row shard and gather just the MXFP8 values and scales, about half the bytes of the bf16 residual; below it the extra scales gather costs more than the bytes save. Each shard is normed with its owner's gamma, so the result matches the default only when every device in a group holds the same gamma.outputs_normedandoutputs_residualthen hold valid rows only for this device's shard. The fused path is unchanged.
Args:
- outputs_normed (
TensorTuple[*Normed.values]): Per-device normed output (the group's gathered[rows, cols], replicated within the group). - outputs_quant (
TensorTuple[*Quant.values]): Per-device MXFP8 copy ofoutputs_normed. - outputs_scale (
TensorTuple[*Scales.values]): Per-device E8M0 block scales,[rows, cols / 32]. - outputs_residual (
TensorTuple[*Residuals.values]): Per-device gathered residual. - inputs (
TensorTuple[*Ins.values]): Per-device input row-shards to gather. - signal_buffers (
TensorTuple[*Sigs.values]): Per-device synchronization buffers. - gammas (
TensorTuple[*Gammas.values]): Per-device RMSNorm gamma weights. - dev_ctxs_input (
DeviceContextArray): Device contexts for participating GPUs.