For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
mhc
Manifold-Constrained Hyper-Connections (mHC) gates and stream write-back.
Reference: Xie et al. 2026, "Manifold-Constrained Hyper-Connections", section 2.2 equation 8.
An mHC site replaces the residual add between two transformer sublayers with a
learned mixing of hc_mult parallel residual streams. A single projection of
the normalized streams produces 2 * hc_mult + hc_mult**2 values per token,
which this module turns into the three gate tensors the site applies:
pre: stream-collapse weights, folding the parallel streams into the single sequence the sublayer consumes.post: where the sublayer output lands across the streams, in[0, 2].comb: thehc_mult x hc_multstream mixer, softmax-initialized and then Sinkhorn-Knopp projected onto the doubly-stochastic manifold.
After the sublayer runs, mhc_post writes its output back into the streams:
streams'[k] = post[k] * y + sum_j comb[j, k] * streams[j].
hyper_connection_gates leaves the projection and the final collapse to the
graph. On gfx950 mhc_pre fuses the whole pre site instead: projection, RMS
scale, gates, collapse and an optional RMSNorm in one op, which is one launch
at large token counts and two below them, where K is split across blocks.
Functions
-
hyper_connection_gates: Launches the mHC gate kernel, one warp per row. -
hyper_connection_gates_kernel: Computes the mHCpre,postandcombgates. -
mhc_fn_split: Splits fp32 values into three bf16 terms whose sum is exactlyv. -
mhc_post: Launches the mHC write-back,(num_rows, k_blocks)blocks. -
mhc_post_block_size: Returns the threads per blockmhc_posttakes forhidden_size. -
mhc_post_kernel: Writes one token's slice of the mHC streams back. -
mhc_post_non_temporal: Returns whethermhc_postmarks its accesses non-temporal. -
mhc_post_tile: Writes onewidth-wide slice of one token's mHC streams back. -
mhc_pre: Launches the fused mHC pre site on blocks of 16 tokens and 256 threads. -
mhc_pre_finish_kernel: Finishes one token from its K-split partials: gates, then collapse. -
mhc_pre_kernel: Runs a whole mHC pre site for 16 tokens: projection, gates, collapse. -
mhc_pre_launch_config: Returns howmhc_prelaunches forTtokens athc_mult = 4. -
mhc_pre_partials_kernel: Projects 16 tokens over one K slice into fp32 partial sums.