For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

mhc

Manifold-Constrained Hyper-Connections (mHC) gate computation.

Reference: Xie et al. 2026, "Manifold-Constrained Hyper-Connections", section 2.2 equation 8.

An mHC site replaces the residual add between two transformer sublayers with a learned mixing of hc_mult parallel residual streams. A single projection of the normalized streams produces 2 * hc_mult + hc_mult**2 values per token, which this module turns into the three gate tensors the site applies:

• pre: stream-collapse weights, folding the parallel streams into the single sequence the sublayer consumes.
• post: where the sublayer output lands across the streams, in [0, 2].
• comb: the hc_mult x hc_mult stream mixer, softmax-initialized and then Sinkhorn-Knopp projected onto the doubly-stochastic manifold.

The projection itself and the final collapse stay outside: they are a GEMM and a broadcast-reduce that the graph compiler already handles well.

Functions

• ​hyper_connection_gates: Launches the mHC gate kernel, one warp per row.
• ​hyper_connection_gates_kernel: Computes the mHC pre, post and comb gates.