For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo struct
BundledAllReduceAddRMSNormQuantFP8
struct BundledAllReduceAddRMSNormQuantFP8
Registers the mo.composite.bundled.allreduce_add_rms_norm_quant_fp8 graph op with the graph compiler.
Implemented traits
Methods
execute
static def execute[Ins: TypeList[Ins.values], Sigs: TypeList[Sigs.values], //, _trace_name: StaticString, epsilon: Float32, weight_offset: Float32](output: T, out_scale: T, out_residual: T, inputs: TensorTuple[*Ins.values], signal_buffers: TensorTuple[*Sigs.values], residual: T, gamma: T, scale_ub: T, ctx: DeviceContext)
Per-device fused allreduce.sum + add + rms_norm + fp8 quantize.
Single-device analog of DistributedAllReduceAddRMSNormQuantFP8, for
use inside mo.parallel. The parallel framework launches one
instance per GPU; this kernel invokes the same underlying primitive
(allreduce_residual_rmsnorm) that the distributed variant calls
from within _launch_device_collective, but for a single device.
Parameters:
- Ins (
TypeList[values]): The element types ofinputs(inferred). - Sigs (
TypeList[values]): The element types ofsignal_buffers(inferred). - _trace_name (
StaticString): Trace name used for profiling and debugging. - epsilon (
Float32): RMSNorm epsilon scalar. - weight_offset (
Float32): RMSNorm weight offset scalar.
Args:
- output (
T): FP8 quantized output tensor for THIS GPU. - out_scale (
T): Per-token scale tensor for THIS GPU. - out_residual (
T): Post-add residual tensor for THIS GPU. - inputs (
TensorTuple[*Ins.values]): Input tensors from ALL participating GPUs. - signal_buffers (
TensorTuple[*Sigs.values]): Signal buffers for ALL participating GPUs. - residual (
T): Residual tensor for THIS GPU. - gamma (
T): RMSNorm weight for THIS GPU. - scale_ub (
T): Quantization scale upper bound scalar (host). - ctx (
DeviceContext): Device context for THIS GPU.