For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
allgather_rmsnorm_quant
def allgather_rmsnorm_quant[in_dtype: DType, ngpus: Int, in_layout: TensorLayout, in_origin: Origin[mut=in_origin.mut], QuantEpilogueType: def[width: Int](row: Int, col: Int, val: SIMD[in_dtype, width]) -> None & RegisterPassable & ImplicitlyCopyable, //, group_size: Int = ngpus, pdl_level: PDLLevel = PDLLevel()](input_buffers: Array[TileTensor[in_dtype, in_layout, in_origin], ngpus], normed_out: TileTensor[in_dtype, Engine=normed_out.Engine, address_space=normed_out.address_space, linear_idx_type=normed_out.linear_idx_type], sum_out: TileTensor[in_dtype, Engine=sum_out.Engine, address_space=sum_out.address_space, linear_idx_type=sum_out.linear_idx_type], gamma: TileTensor[in_dtype, Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type], epsilon: Float32, weight_offset: Scalar[in_dtype], rank_sigs: Array[Pointer[Signal, MutAnyOrigin], ngpus], ctx: DeviceContext, quant_epilogue: QuantEpilogueType, my_rank: Optional[Int] = None)
allgather_rmsnorm that also hands each normed value to an epilogue.
The epilogue sees the bf16 that lands in normed_out, so a folded-in
quantizer emits the same bytes one launch fewer. Caller-supplied because
comm cannot depend on linalg (which already depends on comm).
Parameters:
- in_dtype (
DType): Input/output data type (bf16). - ngpus (
Int): Total number of devices in the world. - in_layout (
TensorLayout): Layout of the input shard TileTensors. - in_origin (
Origin[mut=mut]): Origin of the input shard TileTensors. - QuantEpilogueType (
def[width: Int](row: Int, col: Int, val: SIMD[in_dtype, width]) -> None&RegisterPassable&ImplicitlyCopyable): The type ofquant_epilogue(inferred). - group_size (
Int): Number of devices per independent all-gather group; see_allgather_rmsnorm_impl. - pdl_level (
PDLLevel): Programmatic dependent launch level. AboveOFFthe grid waits on its predecessor before it reads any input, including gamma, and releases its dependents after the end barrier that retires the peers' reads (ON) or right after that wait (OVERLAP_AT_BEGINNING). That level requires the next kernel not to write this kernel's inputs before its own PDL wait, because peers may still be reading them.ONis the level that is safe against a dependent that does.
Args:
- input_buffers (
Array[TileTensor[in_dtype, in_layout, in_origin], ngpus]): Input buffers from ALLngpusdevices as TileTensors, indexed by GLOBAL device rank. - normed_out (
TileTensor[in_dtype, Engine=normed_out.Engine, address_space=normed_out.address_space, linear_idx_type=normed_out.linear_idx_type]): This GPU's full normed output[rows, cols]. - sum_out (
TileTensor[in_dtype, Engine=sum_out.Engine, address_space=sum_out.address_space, linear_idx_type=sum_out.linear_idx_type]): This GPU's full gathered residual[rows, cols]. - gamma (
TileTensor[in_dtype, Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type]): RMSNorm gamma weights (1D TileTensor of length cols). - epsilon (
Float32): RMSNorm epsilon for numerical stability. - weight_offset (
Scalar[in_dtype]): Additive offset for gamma weights. - rank_sigs (
Array[Pointer[Signal, MutAnyOrigin], ngpus]): Allngpusdevices' Signal pointers, indexed by GLOBAL device rank. - ctx (
DeviceContext): Device context for this GPU. - quant_epilogue (
QuantEpilogueType): Normed-value epilogue; see_allgather_rmsnorm_impl. - my_rank (
Optional[Int]): Optional GLOBAL rank of THIS GPU in[0, ngpus).