For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
mhc_pre
def mhc_pre[hc_mult: Int, hidden_size: Int, hc_sinkhorn_iters: Int, has_norm: Bool, fn_terms: Int = Int(3)](post: TileTensor[.float32, Engine=post.Engine, address_space=post.address_space, linear_idx_type=post.linear_idx_type], comb: TileTensor[.float32, Engine=comb.Engine, address_space=comb.address_space, linear_idx_type=comb.linear_idx_type], collapsed: TileTensor[Engine=collapsed.Engine, address_space=collapsed.address_space, linear_idx_type=collapsed.linear_idx_type], pre: TileTensor[.float32, Engine=pre.Engine, address_space=pre.address_space, linear_idx_type=pre.linear_idx_type], streams: TileTensor[collapsed.dtype, Engine=streams.Engine, address_space=streams.address_space, linear_idx_type=streams.linear_idx_type], hc_fn: TileTensor[Engine=hc_fn.Engine, address_space=hc_fn.address_space, linear_idx_type=hc_fn.linear_idx_type], base: TileTensor[.float32, Engine=base.Engine, address_space=base.address_space, linear_idx_type=base.linear_idx_type], scale: TileTensor[.float32, Engine=scale.Engine, address_space=scale.address_space, linear_idx_type=scale.linear_idx_type], gamma: TileTensor[collapsed.dtype, Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type], rms_eps: Float32, hc_eps: Float32, norm_eps: Float32, context: DeviceContext, force_k_splits: Int = Int(0))
Launches the fused mHC pre site on blocks of 16 tokens and 256 threads.
Computes, per token t with x = streams[t] flattened to hc_mult * hidden_size values:
mix = (x @ hc_fn.T) * rsqrt(mean(x * x) + rms_eps);pre,postandcombfrommix, ashyper_connection_gatesdoes;collapsed[t] = sum_j pre[t, j] * streams[t, j], rounded to the streams' dtype, then withhas_normRMS-normalized bygamma.
The projection runs on bf16 MFMA with fp32 accumulation. A bf16 hc_fn is
one exact term; an fp32 hc_fn is split into fn_terms bf16 terms, and at
3 their products are exact, so the result is at least as accurate as an
fp32 GEMM. Requires gfx950.
mhc_pre_launch_config picks the K-splits KS that bring the grid to
about 256 blocks, and KS = 4 from 129 to 192 token blocks, where the
fused grid would leave CUs idle. At KS = 1 one kernel does the
whole site. Above it, KS blocks per 16 tokens each project a slice of K
into a per-call fp32 workspace, and a second launch on the same stream
sums each token's slices in a fixed order and finishes it, one block per
token. Both forms are deterministic.
Parameters:
- hc_mult (
Int): Number of parallel residual streams. Only 4. - hidden_size (
Int): Width of one stream: a multiple of 1024 in[1024, 8192]. - hc_sinkhorn_iters (
Int): Sinkhorn-Knopp iterations used to projectcomb. - has_norm (
Bool): Whether to RMS-normalize the collapsed row bygamma. - fn_terms (
Int): Test-only: number of bf16 terms per fp32hc_fnweight, 1 to 3. Callers keep the default 3; fewer drop precision and exist so tests can check that every term matters. Ignored for a bf16hc_fn.
Args:
- post (
TileTensor[.float32, Engine=post.Engine, address_space=post.address_space, linear_idx_type=post.linear_idx_type]): Sublayer-output placement weights. Shape:[T, hc_mult]. - comb (
TileTensor[.float32, Engine=comb.Engine, address_space=comb.address_space, linear_idx_type=comb.linear_idx_type]): Stream mixer;comb[t, j, k]weighs input streamjin output streamk. Shape:[T, hc_mult, hc_mult]. - collapsed (
TileTensor[Engine=collapsed.Engine, address_space=collapsed.address_space, linear_idx_type=collapsed.linear_idx_type]): The collapsed sequence, normalized withhas_norm. Shape:[T, hidden_size]. - pre (
TileTensor[.float32, Engine=pre.Engine, address_space=pre.address_space, linear_idx_type=pre.linear_idx_type]): Stream-collapse weights. Shape:[T, hc_mult]. - streams (
TileTensor[collapsed.dtype, Engine=streams.Engine, address_space=streams.address_space, linear_idx_type=streams.linear_idx_type]): The bf16 residual streams. Shape:[T, hc_mult, hidden_size]. - hc_fn (
TileTensor[Engine=hc_fn.Engine, address_space=hc_fn.address_space, linear_idx_type=hc_fn.linear_idx_type]): The fp32 or bf16 projection weights. Shape:[2 * hc_mult + hc_mult**2, hc_mult * hidden_size]. - base (
TileTensor[.float32, Engine=base.Engine, address_space=base.address_space, linear_idx_type=base.linear_idx_type]): Per-output bias. Shape:[2 * hc_mult + hc_mult**2]. - scale (
TileTensor[.float32, Engine=scale.Engine, address_space=scale.address_space, linear_idx_type=scale.linear_idx_type]): Per-output scale, inpre,post,comborder. Shape:[3]. - gamma (
TileTensor[collapsed.dtype, Engine=gamma.Engine, address_space=gamma.address_space, linear_idx_type=gamma.linear_idx_type]): RMSNorm weight. Shape:[hidden_size]withhas_norm, else[1]. - rms_eps (
Float32): Epsilon of the RMS scale on the projection. - hc_eps (
Float32): Epsilon guarding the gates and the Sinkhorn divisions. - norm_eps (
Float32): Epsilon of the collapsed-row RMSNorm. - context (
DeviceContext): The device context. - force_k_splits (
Int): Splits of K across blocks; 0 picks, 1 forces none.
Raises:
If force_k_splits is not one mhc_pre_launch_config accepts, a
shape disagrees with hc_mult, hidden_size or the token count of
streams, or a tensor is not laid out for the kernel's 16-byte
vector accesses.