IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

mhc

Manifold-Constrained Hyper-Connections (mHC) gates and stream write-back.

Reference: Xie et al. 2026, "Manifold-Constrained Hyper-Connections", section 2.2 equation 8.

An mHC site replaces the residual add between two transformer sublayers with a learned mixing of hc_mult parallel residual streams. A single projection of the normalized streams produces 2 * hc_mult + hc_mult**2 values per token, which this module turns into the three gate tensors the site applies:

  • pre: stream-collapse weights, folding the parallel streams into the single sequence the sublayer consumes.
  • post: where the sublayer output lands across the streams, in [0, 2].
  • comb: the hc_mult x hc_mult stream mixer, softmax-initialized and then Sinkhorn-Knopp projected onto the doubly-stochastic manifold.

After the sublayer runs, mhc_post writes its output back into the streams: streams'[k] = post[k] * y + sum_j comb[j, k] * streams[j].

hyper_connection_gates leaves the projection and the final collapse to the graph. On gfx950 mhc_pre fuses the whole pre site instead: projection, RMS scale, gates, collapse and an optional RMSNorm in one op, which is one launch at large token counts and two below them, where K is split across blocks.

Functions​

Was this page helpful?