For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
ihc
Identity hyper-connection (iHC) site kernels.
An iHC site keeps hc_mult residual streams per token. Before its sublayer it
projects the flattened streams (K = hc_mult * hidden values) onto num_mix
float32 weight rows, scales the logits by the streams' inverse RMS, turns them
into sigmoid gates, folds the streams into one sequence with the first
hc_mult gates (pre), and RMS-normalizes that sequence.
Two kernels do it with two reads of the streams, which stay in their own dtype throughout:
ihc_mix_gemv: the projection and the streams' sum of squares, split overK, on tensor cores straight from the bfloat16 streams (the float32 weight as a bfloat16 pair), rather than converting the streams for a float32 (TF32) GEMM.ihc_collapse_rms_norm: the gates, the collapse and the RMSNorm, one block per token.
nn.mhc.mhc_pre fuses the same kind of pre site for mHC, with the comb mixer
and its Sinkhorn projection, and only on gfx950; iHC has neither. Two
bfloat16 terms of the weight (where mhc_pre takes three) already exceed the
TF32 a float32 matmul would use, and keep the gates within 1e-5.
comptime values
IHC_POISONS
comptime IHC_POISONS = 4
Planted faults, poison 1 through 4, each of which a test must catch: 1 drops the low half of the split weight, 2 sums only half of a row pair's squares, 3 scales pre by the post band's scale, 4 skips the last row.
Functions
-
ihc_collapse_rms_norm: Launches the iHC collapse + RMSNorm kernel, one block per token. -
ihc_collapse_rms_norm_block: Collapses one token's streams and RMS-normalizes the result. -
ihc_collapse_rms_norm_kernel: Runsihc_collapse_rms_norm_blockfor tokenblock_idx.x. -
ihc_mix_gemv: Launches the split-K iHC projection and sum of squares. -
ihc_mix_gemv_block: Projects_GEMV_TILES16-row tiles onto oneKsplit on tensor cores. -
ihc_mix_gemv_kernel: Runsihc_mix_gemv_blockfor block(row_block, split). -
ihc_mix_gemv_splits: Returns the leading dimension ofihc_mix_gemv's partials. -
ihc_mix_gemv_warps: Returns the warps per block ofihc_mix_gemv.