For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
mhc_pre_finish_kernel
def mhc_pre_finish_kernel[dtype: DType, PostLayoutType: TensorLayout, CombLayoutType: TensorLayout, CollapsedLayoutType: TensorLayout, PreLayoutType: TensorLayout, StreamsLayoutType: TensorLayout, PartialLayoutType: TensorLayout, BaseLayoutType: TensorLayout, ScaleLayoutType: TensorLayout, GammaLayoutType: TensorLayout, hc_mult: Int, hidden_size: Int, hc_sinkhorn_iters: Int, has_norm: Bool](post: TileTensor[.float32, PostLayoutType, MutAnyOrigin], comb: TileTensor[.float32, CombLayoutType, MutAnyOrigin], collapsed: TileTensor[dtype, CollapsedLayoutType, MutAnyOrigin], pre: TileTensor[.float32, PreLayoutType, MutAnyOrigin], streams: TileTensor[dtype, StreamsLayoutType, ImmutAnyOrigin], partial: TileTensor[.float32, PartialLayoutType, ImmutAnyOrigin], base: TileTensor[.float32, BaseLayoutType, ImmutAnyOrigin], scale: TileTensor[.float32, ScaleLayoutType, ImmutAnyOrigin], gamma: TileTensor[dtype, GammaLayoutType, ImmutAnyOrigin], rms_eps: Float32, hc_eps: Float32, norm_eps: Float32, k_splits: Int32)
Finishes one token from its K-split partials: gates, then collapse.
One block per token keeps a decode batch of a few tokens on as many CUs.
The block sums the token's k_splits partials in a fixed order (lane
l of a wave adds splits l, l + 64, ... in turn, then a warp
reduction), so the result does not depend on scheduling.
Parameters:
- dtype (
DType): Element type of the streams and the collapsed output. - PostLayoutType (
TensorLayout): Layout ofpost. - CombLayoutType (
TensorLayout): Layout ofcomb. - CollapsedLayoutType (
TensorLayout): Layout ofcollapsed. - PreLayoutType (
TensorLayout): Layout ofpre. - StreamsLayoutType (
TensorLayout): Layout ofstreams. - PartialLayoutType (
TensorLayout): Layout ofpartial. - BaseLayoutType (
TensorLayout): Layout ofbase. - ScaleLayoutType (
TensorLayout): Layout ofscale. - GammaLayoutType (
TensorLayout): Layout ofgamma. - hc_mult (
Int): Number of parallel residual streams. - hidden_size (
Int): Width of one stream. - hc_sinkhorn_iters (
Int): Sinkhorn-Knopp iterations used to projectcomb. - has_norm (
Bool): Whether to RMS-normalize the collapsed row bygamma.
Args:
- post (
TileTensor[.float32, PostLayoutType, MutAnyOrigin]): Sublayer-output placement weights. Shape:[T, hc_mult]. - comb (
TileTensor[.float32, CombLayoutType, MutAnyOrigin]): Stream mixer. Shape:[T, hc_mult, hc_mult]. - collapsed (
TileTensor[dtype, CollapsedLayoutType, MutAnyOrigin]): The collapsed (optionally normalized) sequence. Shape:[T, hidden_size]. - pre (
TileTensor[.float32, PreLayoutType, MutAnyOrigin]): Stream-collapse weights. Shape:[T, hc_mult]. - streams (
TileTensor[dtype, StreamsLayoutType, ImmutAnyOrigin]): Residual streams. Shape:[T, hc_mult, hidden_size]. - partial (
TileTensor[.float32, PartialLayoutType, ImmutAnyOrigin]):mhc_pre_partials_kernel's sums, rows 16-byte aligned. - base (
TileTensor[.float32, BaseLayoutType, ImmutAnyOrigin]): Per-output bias. Shape:[2 * hc_mult + hc_mult**2]. - scale (
TileTensor[.float32, ScaleLayoutType, ImmutAnyOrigin]): Per-output scale, inpre,post,comborder. Shape:[3]. - gamma (
TileTensor[dtype, GammaLayoutType, ImmutAnyOrigin]): RMSNorm weight. Shape:[hidden_size], read only withhas_norm. - rms_eps (
Float32): Epsilon of the RMS scale on the projection. - hc_eps (
Float32): Epsilon guarding the gates and the Sinkhorn divisions. - norm_eps (
Float32): Epsilon of the collapsed-row RMSNorm. - k_splits (
Int32): Number of K slices inpartial, at most 256.