IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

allreduce_rmsnorm_exit_concat

Fused allreduce sum + RMSNorm + column-shard cast + SiTU-and-mul concat.

One launch replaces the four-kernel exit chain of a tensor-parallel latent MoE (Kimi K3 at TP8):

  1. allreduce.sum over every device's [rows, cols] partial (float32 or bfloat16),
  2. rms_norm of every reduced row over all cols columns,
  3. the column-shard slice/cast: device r keeps columns [r * cols / ngpus, (r + 1) * cols / ngpus) of the normed row, in bf16,
  4. moe.exit.act.concat: that shard written beside the SiTU-and-mul of the device's own [rows, 2 * half] gated shared-expert projection.

The result is [rows, cols / ngpus + half] bf16 per device, column-sharded -- exactly the exit GEMM's activation.

Shape of the kernel​

Every rank holds a partial of every column, so the collective is a reduce-scatter into per-rank scratch (the bytes behind each Signal header) followed by a gather that rebuilds whole rows:

Stage 0 (before the start barrier, no peer data): each block writes the SiTU half of its rows from the LOCAL gate_up. It needs nothing from a peer, so it hides behind the barrier.

Stage 1 (reduce-scatter): block b on rank r reduces columns [r * cols / ngpus, (r + 1) * cols / ngpus) of its rows from every peer's partials in float32 and stores the sum, rounded to the input dtype as the plain allreduce rounds it, in rank r's own scratch, laid out [rows, cols / ngpus].

Barrier 2 (need_fence=True): the barriers index their counters by block id, so block b may only read what block b wrote on the other ranks; the row-ownership below is a function of the block id alone, which is what makes that hold.

Stage 2: block b reads the whole row back, slice p from peer p, to fold the mean square in registers, then re-reads its OWN slice (local bytes) to write the normalized bf16 shard. Stage 0 already wrote those rows' SiTU half.

Bit-identity​

Both the sum and the mean square are computed in the same order the ops this replaces use, so the fused output is byte-identical to the unfused chain wherever that chain's allreduce.sum takes its 2-stage kernel -- the 1-stage caveat in Shape dispatch below applies to it too:

  • The peer sum starts at the rank that OWNS each element's SIMD vector in allreduce's flat reduce-scatter partition and rotates from there, which is the order _load_reduce sums in (the 2-stage allreduce hands it the tiles already rotated by the partition owner). The owner of row m's slice-r vectors is (m * ngpus + r) // rows: with the partition's contiguous chunks, a row's column slice lies inside one chunk, and every row inside a chunk shares it.

  • The mean square folds a lane-wise SIMD[f32, simd_width] accumulator over the row in ascending column tiles, collapses it with reduce_add, and block-reduces that scalar -- ReduceSum's accumulate / reduce / join_parallel in builtin_kernels/monoids.mojo, which is what the graph compiler's generated rowwise block-tier norm kernel runs. That tier's block size is a fixed 256 threads and its tile stride BLOCK_SIZE * simd_width, so thread t folds the same columns in the same order here.

    The norm itself is ops.rms_norm's Llama form: x * rsqrt(mean(x**2) + eps) in f32, rounded to the input dtype, times gamma, cast to the output dtype last (weight_offset 0, multiply_before_cast False).

  • The SiTU half repeats moe.exit.act.concat's body in the same operation order. comm cannot import nn, so the math is copied, not called.

Shape dispatch​

The fused kernel gives a block one row, so its reduce-scatter pulls cols bytes per row from every peer in ngpus messages of cols / ngpus bytes. That is the right shape for a decode step (a few dozen rows: the messages ride the latency, and folding the norm into the collective's epilogue deletes two launches), and the wrong shape for a prefill chunk (thousands of rows: the per-row messages leave the fabric idle and the norm's block reduction runs on cols / ngpus columns per row instead of a whole row). In isolation the fused kernel is still ahead of the chain at every row count measured on MI355 at 2 and at 8 devices, so a microbenchmark alone cannot place a gate. The gate exists because routing prefill chunks through the fused kernel raised Kimi K3's end-to-end time to first token on 8 devices, while decode-sized calls gained.

So the launcher gates on size, at run time, from rows/cols -- the same value on every rank, so every rank takes the same branch:

  • at or below EXIT_CONCAT_FUSE_ROWS (EXIT_CONCAT_FUSE_THRESHOLD bytes) the fused kernel runs, unchanged;
  • every other call -- above the cut, or at a world size the threshold was not calibrated for (any but 2, 4 or 8 devices) -- runs the chain this op replaced: the plain allreduce into a full [rows, cols] sum, then _allreduce_rmsnorm_exit_concat_chain_kernel, which folds the mean square over the local sum, normalizes this rank's slice and writes the SiTU half.

Where the chain's plain allreduce takes its 2-stage kernel, both arms write the same bytes: its reduce-scatter sums each partition from the partition's owner first, rotating exactly as the fused kernel does, the chain reads that peer sum rounded to in_dtype (which is what the fused kernel's reduce-scatter writes) and folds the mean square in the same tile order with the same block reduction. The plain allreduce's 1-stage kernel sums in global rank order instead, and CDNA3's tuning keeps 1-stage at every size, so on such a world the two arms can differ by the rounding of the float32 sum alone -- ULP-level, far under the norm's own tolerance. The chain's [rows, cols] sum is a per-device workspace (comm.workspace), not the payload behind this rank's Signal header: only the plain allreduce's partition staging has to fit there, so a prefill chunk's signal buffers stay the plain allreduce's size.

A vendor-CCL build and a host without P2P run the chain too, with the vendor allreduce or the plain allreduce's staged fallback in front of it -- the same two-launch form allreduce_rms_norm takes in those two situations. Neither consults the size gate, so neither can pick the fused kernel.

The plain allreduce kernels are untouched: this file adds its own kernels and its own scratch layout.

comptime values​

ARNEC_THREADS​

comptime ARNEC_THREADS = 256

Threads per block.

Fixed rather than tuned: the mean square has to be block-reduced in the same tree shape as the norm it replaces, whose rowwise block tier hard-codes BLOCK_SIZE = 256 (algorithm/rowwise.mojo's launch), and that tier's per-thread tile stride is BLOCK_SIZE * simd_width -- this kernel folds each thread's columns in the same order only at the same block size.

EXIT_CONCAT_CALIBRATED_COLS​

comptime EXIT_CONCAT_CALIBRATED_COLS = 3584

Cols the fuse threshold is stated at (Kimi K3's latent).

A different width moves the cut in bytes, so recalibrate before trusting it there.

EXIT_CONCAT_FUSE_ROWS​

comptime EXIT_CONCAT_FUSE_ROWS = 128

Rows at EXIT_CONCAT_CALIBRATED_COLS at or below which the fused kernel runs.

A decode/prefill cut, not a measured crossover (see the module docstring). A decode step is 32 rows at batch 4 with DSpark spec decode, well below it, and a prefill chunk is thousands of rows, well above it. A perf cut, not correctness: both arms write the same bytes.

EXIT_CONCAT_FUSE_THRESHOLD is that row count as [rows, cols] payload bytes, the meter AR_NORM_FUSE_THRESHOLD uses: a fused arm's cost is set by the traffic per row, cols bytes from every peer, which does not shrink with the world size, unlike the plain allreduce's per-rank partition. It is stated in float32 bytes, so a bfloat16 call fuses up to twice the rows.

EXIT_CONCAT_FUSE_THRESHOLD​

comptime EXIT_CONCAT_FUSE_THRESHOLD = (Int(458752) * size_of[DType.float32]())

[rows, cols] payload bytes at or below which the fused kernel runs.

Functions​

Was this page helpful?