For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
DistributedAllGatherRMSNormQuantMXFP6
struct DistributedAllGatherRMSNormQuantMXFP6
Registers mo.composite.distributed.allgather_rms_norm_quant_mxfp6.
Implemented traits
Methods
execute
static def execute[Normed: TypeList[Normed.values], Quant: TypeList[Quant.values], Scales: TypeList[Scales.values], Residuals: TypeList[Residuals.values], Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], Gammas: TypeList[Gammas.values], //, _trace_name: StaticString, fp6_format: Int, epsilon: Float32, weight_offset: Float32, group_size: Int = Int(0)](outputs_normed: TensorTuple[*Normed.values], outputs_quant: TensorTuple[*Quant.values], outputs_scale: TensorTuple[*Scales.values], outputs_residual: TensorTuple[*Residuals.values], inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], gammas: TensorTuple[*Gammas.values], dev_ctxs_input: DeviceContextArray)
Fused all-gather + RMSNorm that also emits a packed MXFP6 copy.
DistributedAllGatherRMSNorm plus outputs_quant/outputs_scale. The
quantize rides the collective's epilogue on the same bf16 that lands in
outputs_normed, so it is byte-identical to a standalone quantize.
Limitations:
- Targets AMD (CDNA4): the fallback calls quantize_mx_amd and
the scale layout is its rank-2 one -- what
block_scaled_matmul_amd takes as a_scales, not the SM100 SF
atom and not the preshuffled order the _preb kernel needs.
- Maximum of 8 GPUs; requires P2P.
Parameters:
- Normed (
TypeList[values]): The element types ofoutputs_normed(inferred). - Quant (
TypeList[values]): The element types ofoutputs_quant(inferred). - Scales (
TypeList[values]): The element types ofoutputs_scale(inferred). - Residuals (
TypeList[values]): The element types ofoutputs_residual(inferred). - Ins (
TypeList[values]): The element types ofinputs(inferred). - Sigs (
TypeList[values]): The element types ofsignal_buffers(inferred). - Gammas (
TypeList[values]): The element types ofgammas(inferred). - _trace_name (
StaticString): Trace name for profiling. - fp6_format (
Int): FP6 encoding, 0 for E2M3 and 1 for E3M2. - epsilon (
Float32): RMSNorm epsilon, shared by every device. - weight_offset (
Float32): Gamma offset, shared by every device. - group_size (
Int): Devices per independent all-gather group; seeDistributedAllGatherRMSNorm.
Args:
- outputs_normed (
TensorTuple[*Normed.values]): Per-device normed output (the group's gathered[rows, cols], replicated within the group). - outputs_quant (
TensorTuple[*Quant.values]): Per-device packed MXFP6 copy ofoutputs_normed,[rows, cols * 3 // 4]. - outputs_scale (
TensorTuple[*Scales.values]): Per-device E8M0 block scales,[rows, cols / 32]. - outputs_residual (
TensorTuple[*Residuals.values]): Per-device gathered residual. - inputs (
TensorTuple[*Ins.values]): Per-device input row-shards to gather. - signal_buffers (
TensorTuple[*Sigs.values]): Per-device synchronization buffers. - gammas (
TensorTuple[*Gammas.values]): Per-device RMSNorm gamma weights. - dev_ctxs_input (
DeviceContextArray): Device contexts for participating GPUs.