IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

ihc

Identity hyper-connection (iHC) site kernels.

An iHC site keeps hc_mult residual streams per token. Before its sublayer it projects the flattened streams (K = hc_mult * hidden values) onto num_mix float32 weight rows, scales the logits by the streams' inverse RMS, turns them into sigmoid gates, folds the streams into one sequence with the first hc_mult gates (pre), and RMS-normalizes that sequence.

Two kernels do it with two reads of the streams, which stay in their own dtype throughout:

  • ihc_mix_gemv: the projection and the streams' sum of squares, split over K, on tensor cores straight from the bfloat16 streams (the float32 weight as a bfloat16 pair), rather than converting the streams for a float32 (TF32) GEMM.
  • ihc_collapse_rms_norm: the gates, the collapse and the RMSNorm, one block per token.

nn.mhc.mhc_pre fuses the same kind of pre site for mHC, with the comb mixer and its Sinkhorn projection, and only on gfx950; iHC has neither. Two bfloat16 terms of the weight (where mhc_pre takes three) already exceed the TF32 a float32 matmul would use, and keep the gates within 1e-5.

comptime values​

IHC_POISONS​

comptime IHC_POISONS = 4

Planted faults, poison 1 through 4, each of which a test must catch: 1 drops the low half of the split weight, 2 sums only half of a row pair's squares, 3 scales pre by the post band's scale, 4 skips the last row.

Functions​

Was this page helpful?