IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo struct

BundledAllReduceAddRMSNormQuantFP8

struct BundledAllReduceAddRMSNormQuantFP8

Registers the mo.composite.bundled.allreduce_add_rms_norm_quant_fp8 graph op with the graph compiler.

Implemented traits​

AnyType, Deinitable, Movable

Methods​

execute​

static def execute[Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], //, _trace_name: StaticString, epsilon: Float32, weight_offset: Float32](output: T, out_scale: T, out_residual: T, inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], residual: T, gamma: T, scale_ub: T, ctx: DeviceContext)

Per-device fused allreduce.sum + add + rms_norm + fp8 quantize.

Single-device analog of DistributedAllReduceAddRMSNormQuantFP8, for use inside mo.parallel. The parallel framework launches one instance per GPU; this kernel invokes the same underlying primitive (allreduce_residual_rmsnorm) that the distributed variant calls from within _launch_device_collective, but for a single device.

Parameters:

  • ​Ins (TypeList[values]): The element types of inputs (inferred).
  • ​Sigs (TypeList[values]): The element types of signal_buffers (inferred).
  • ​_trace_name (StaticString): Trace name used for profiling and debugging.
  • ​epsilon (Float32): RMSNorm epsilon scalar.
  • ​weight_offset (Float32): RMSNorm weight offset scalar.

Args:

  • ​output (T): FP8 quantized output tensor for THIS GPU.
  • ​out_scale (T): Per-token scale tensor for THIS GPU.
  • ​out_residual (T): Post-add residual tensor for THIS GPU.
  • ​inputs (TensorTuple[*Ins.values]): Input tensors from ALL participating GPUs.
  • ​signal_buffers (TensorTuple[*Sigs.values]): Signal buffers for ALL participating GPUs.
  • ​residual (T): Residual tensor for THIS GPU.
  • ​gamma (T): RMSNorm weight for THIS GPU.
  • ​scale_ub (T): Quantization scale upper bound scalar (host).
  • ​ctx (DeviceContext): Device context for THIS GPU.

Was this page helpful?