For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
allreduce_rmsnorm_exit_concat
def allreduce_rmsnorm_exit_concat[in_dtype: DType, dtype: DType, ngpus: Int, in_layout: TensorLayout, in_origin: Origin[mut=in_origin.mut], //](input_buffers: Array[TileTensor[in_dtype, in_layout, in_origin], ngpus], gamma: TileTensor[Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type], gate_up: TileTensor[dtype, Engine=gate_up.Engine, address_space=gate_up.address_space, linear_idx_type=gate_up.linear_idx_type], output: TileTensor[dtype, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type], epsilon: Float32, beta: Float32, linear_beta: Float32, rank_sigs: Array[Pointer[Signal, MutAnyOrigin], ngpus], ctx: DeviceContext, my_rank: Optional[Int] = None, force_arm: Int = Int(0))
Fused allreduce sum + RMSNorm + exit SiTU-and-mul concat.
Runs the fused kernel at or below the size gate
(EXIT_CONCAT_FUSE_THRESHOLD bytes) and the chain it replaced above it;
see the module docstring for both arms and why they are bit-identical. A
vendor-CCL build and a host without P2P access run the chain too, with the
vendor or the plain allreduce reducing in front of it.
Parameters:
- in_dtype (
DType): The partials' element type; bfloat16 or float32. - dtype (
DType): The gated projection's and the output's type; bfloat16 only. - ngpus (
Int): Number of cooperating devices. - in_layout (
TensorLayout): Layout of the input partial TileTensors. - in_origin (
Origin[mut=mut]): Origin of the input partial TileTensors.
Args:
- input_buffers (
Array[TileTensor[in_dtype, in_layout, in_origin], ngpus]): Every device's[rows, cols]partial, indexed by GLOBAL device rank (the fused arm needs peer access). - gamma (
TileTensor[Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type]): This device's[cols]norm weight, inin_dtypeor, on a float32 input, bfloat16/float16. - gate_up (
TileTensor[dtype, Engine=gate_up.Engine, address_space=gate_up.address_space, linear_idx_type=gate_up.linear_idx_type]): This device's[rows, 2 * half]shared gate/up projection. - output (
TileTensor[dtype, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type]): This device's[rows, cols / ngpus + half]destination. - epsilon (
Float32): The norm's epsilon. - beta (
Float32): The SiTU gate's beta. - linear_beta (
Float32): The SiTU up-half saturation beta; 0 disables it. - rank_sigs (
Array[Pointer[Signal, MutAnyOrigin], ngpus]): Every device'sSignalpointer, indexed by GLOBAL rank. The bytes behind each header hold the collective's staging scratch, never its sum. - ctx (
DeviceContext): Device context for THIS device. - my_rank (
Optional[Int]): Optional GLOBAL rank of THIS device in[0, ngpus); defaults to the physical device id. - force_arm (
Int): Override for threshold experiments.0uses the size gate,1forces the fused kernel and2forces the chain. Every rank of a call must agree, exactly as the gate itself requires.
Raises:
On any shape, dtype, rank or arm this op does not implement. A world size the size gate has no threshold for takes the chain.