For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
allreduce_rmsnorm_exit_concat
Fused allreduce sum + RMSNorm + column-shard cast + SiTU-and-mul concat.
One launch replaces the four-kernel exit chain of a tensor-parallel latent MoE (Kimi K3 at TP8):
allreduce.sumover every device's[rows, cols]partial (float32 or bfloat16),rms_normof every reduced row over allcolscolumns,- the column-shard slice/cast: device
rkeeps columns[r * cols / ngpus, (r + 1) * cols / ngpus)of the normed row, in bf16, moe.exit.act.concat: that shard written beside the SiTU-and-mul of the device's own[rows, 2 * half]gated shared-expert projection.
The result is [rows, cols / ngpus + half] bf16 per device, column-sharded --
exactly the exit GEMM's activation.
Shape of the kernel
Every rank holds a partial of every column, so the collective is a
reduce-scatter into per-rank scratch (the bytes behind each Signal header)
followed by a gather that rebuilds whole rows:
Stage 0 (before the start barrier, no peer data): each block writes the
SiTU half of its rows from the LOCAL gate_up. It needs nothing from a
peer, so it hides behind the barrier.
Stage 1 (reduce-scatter): block b on rank r reduces columns
[r * cols / ngpus, (r + 1) * cols / ngpus) of its rows from every
peer's partials in float32 and stores the sum, rounded to the input dtype
as the plain allreduce rounds it, in rank r's own scratch, laid out
[rows, cols / ngpus].
Barrier 2 (need_fence=True): the barriers index their counters by block
id, so block b may only read what block b wrote on the other ranks;
the row-ownership below is a function of the block id alone, which is
what makes that hold.
Stage 2: block b reads the whole row back, slice p from peer p, to
fold the mean square in registers, then re-reads its OWN slice (local
bytes) to write the normalized bf16 shard. Stage 0 already wrote those
rows' SiTU half.
Bit-identity
Both the sum and the mean square are computed in the same order the ops this
replaces use, so the fused output is byte-identical to the unfused chain
wherever that chain's allreduce.sum takes its 2-stage kernel -- the
1-stage caveat in Shape dispatch below applies to it too:
-
The peer sum starts at the rank that OWNS each element's SIMD vector in
allreduce's flat reduce-scatter partition and rotates from there, which is the order_load_reducesums in (the 2-stage allreduce hands it the tiles already rotated by the partition owner). The owner of rowm's slice-rvectors is(m * ngpus + r) // rows: with the partition's contiguous chunks, a row's column slice lies inside one chunk, and every row inside a chunk shares it. -
The mean square folds a lane-wise
SIMD[f32, simd_width]accumulator over the row in ascending column tiles, collapses it withreduce_add, and block-reduces that scalar --ReduceSum'saccumulate/reduce/join_parallelinbuiltin_kernels/monoids.mojo, which is what the graph compiler's generatedrowwiseblock-tier norm kernel runs. That tier's block size is a fixed 256 threads and its tile strideBLOCK_SIZE * simd_width, so threadtfolds the same columns in the same order here.The norm itself is
ops.rms_norm's Llama form:x * rsqrt(mean(x**2) + eps)in f32, rounded to the input dtype, times gamma, cast to the output dtype last (weight_offset0,multiply_before_castFalse). -
The SiTU half repeats
moe.exit.act.concat's body in the same operation order.commcannot importnn, so the math is copied, not called.
Shape dispatch
The fused kernel gives a block one row, so its reduce-scatter pulls cols
bytes per row from every peer in ngpus messages of cols / ngpus bytes. That
is the right shape for a decode step (a few dozen rows: the messages ride the
latency, and folding the norm into the collective's epilogue deletes two
launches), and the wrong shape for a prefill chunk (thousands of rows: the
per-row messages leave the fabric idle and the norm's block reduction runs on
cols / ngpus columns per row instead of a whole row). In isolation the
fused kernel is still ahead of the chain at every row count measured on MI355
at 2 and at 8 devices, so a microbenchmark alone cannot place a gate. The gate
exists because routing prefill chunks through the fused kernel raised Kimi
K3's end-to-end time to first token on 8 devices, while decode-sized calls
gained.
So the launcher gates on size, at run time, from rows/cols -- the same
value on every rank, so every rank takes the same branch:
- at or below
EXIT_CONCAT_FUSE_ROWS(EXIT_CONCAT_FUSE_THRESHOLDbytes) the fused kernel runs, unchanged; - every other call -- above the cut, or at a world size the threshold was
not calibrated for (any but 2, 4 or 8 devices) -- runs the chain this op
replaced: the plain allreduce into a full
[rows, cols]sum, then_allreduce_rmsnorm_exit_concat_chain_kernel, which folds the mean square over the local sum, normalizes this rank's slice and writes the SiTU half.
Where the chain's plain allreduce takes its 2-stage kernel, both arms write
the same bytes: its reduce-scatter sums each partition from the partition's
owner first, rotating exactly as the fused kernel does, the chain reads that
peer sum rounded to in_dtype (which is what the fused kernel's
reduce-scatter writes) and folds the mean square in the same tile order with
the same block reduction. The plain allreduce's 1-stage kernel sums in
global rank order instead, and CDNA3's tuning keeps 1-stage at every size,
so on such a world the two arms can differ by the rounding of the float32
sum alone -- ULP-level, far under the norm's own tolerance. The chain's
[rows, cols] sum is a per-device workspace (comm.workspace), not the
payload behind this rank's Signal header: only the plain allreduce's
partition staging has to fit there, so a prefill chunk's signal buffers stay
the plain allreduce's size.
A vendor-CCL build and a host without P2P run the chain too, with the vendor
allreduce or the plain allreduce's staged fallback in front of it -- the same
two-launch form allreduce_rms_norm takes in those two situations. Neither
consults the size gate, so neither can pick the fused kernel.
The plain allreduce kernels are untouched: this file adds its own kernels and
its own scratch layout.
comptime values
ARNEC_THREADS
comptime ARNEC_THREADS = 256
Threads per block.
Fixed rather than tuned: the mean square has to be block-reduced in the same
tree shape as the norm it replaces, whose rowwise block tier hard-codes
BLOCK_SIZE = 256 (algorithm/rowwise.mojo's launch), and that tier's
per-thread tile stride is BLOCK_SIZE * simd_width -- this kernel folds each
thread's columns in the same order only at the same block size.
EXIT_CONCAT_CALIBRATED_COLS
comptime EXIT_CONCAT_CALIBRATED_COLS = 3584
Cols the fuse threshold is stated at (Kimi K3's latent).
A different width moves the cut in bytes, so recalibrate before trusting it there.
EXIT_CONCAT_FUSE_ROWS
comptime EXIT_CONCAT_FUSE_ROWS = 128
Rows at EXIT_CONCAT_CALIBRATED_COLS at or below which the fused kernel runs.
A decode/prefill cut, not a measured crossover (see the module docstring). A decode step is 32 rows at batch 4 with DSpark spec decode, well below it, and a prefill chunk is thousands of rows, well above it. A perf cut, not correctness: both arms write the same bytes.
EXIT_CONCAT_FUSE_THRESHOLD is that row count as [rows, cols] payload bytes,
the meter AR_NORM_FUSE_THRESHOLD uses: a fused arm's cost is set by the
traffic per row, cols bytes from every peer, which does not shrink with the
world size, unlike the plain allreduce's per-rank partition. It is stated in
float32 bytes, so a bfloat16 call fuses up to twice the rows.
EXIT_CONCAT_FUSE_THRESHOLD
comptime EXIT_CONCAT_FUSE_THRESHOLD = (Int(458752) * size_of[DType.float32]())
[rows, cols] payload bytes at or below which the fused kernel runs.
Functions
-
allreduce_rmsnorm_exit_concat: Fused allreduce sum + RMSNorm + exit SiTU-and-mul concat.