IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

mhc_pre_finish_kernel

def mhc_pre_finish_kernel[dtype: DType, PostLayoutType: TensorLayout, CombLayoutType: TensorLayout, CollapsedLayoutType: TensorLayout, PreLayoutType: TensorLayout, StreamsLayoutType: TensorLayout, PartialLayoutType: TensorLayout, BaseLayoutType: TensorLayout, ScaleLayoutType: TensorLayout, GammaLayoutType: TensorLayout, hc_mult: Int, hidden_size: Int, hc_sinkhorn_iters: Int, has_norm: Bool](post: TileTensor[.float32, PostLayoutType, MutAnyOrigin], comb: TileTensor[.float32, CombLayoutType, MutAnyOrigin], collapsed: TileTensor[dtype, CollapsedLayoutType, MutAnyOrigin], pre: TileTensor[.float32, PreLayoutType, MutAnyOrigin], streams: TileTensor[dtype, StreamsLayoutType, ImmutAnyOrigin], partial: TileTensor[.float32, PartialLayoutType, ImmutAnyOrigin], base: TileTensor[.float32, BaseLayoutType, ImmutAnyOrigin], scale: TileTensor[.float32, ScaleLayoutType, ImmutAnyOrigin], gamma: TileTensor[dtype, GammaLayoutType, ImmutAnyOrigin], rms_eps: Float32, hc_eps: Float32, norm_eps: Float32, k_splits: Int32)

Finishes one token from its K-split partials: gates, then collapse.

One block per token keeps a decode batch of a few tokens on as many CUs. The block sums the token's k_splits partials in a fixed order (lane l of a wave adds splits l, l + 64, ... in turn, then a warp reduction), so the result does not depend on scheduling.

Parameters:

  • ​dtype (DType): Element type of the streams and the collapsed output.
  • ​PostLayoutType (TensorLayout): Layout of post.
  • ​CombLayoutType (TensorLayout): Layout of comb.
  • ​CollapsedLayoutType (TensorLayout): Layout of collapsed.
  • ​PreLayoutType (TensorLayout): Layout of pre.
  • ​StreamsLayoutType (TensorLayout): Layout of streams.
  • ​PartialLayoutType (TensorLayout): Layout of partial.
  • ​BaseLayoutType (TensorLayout): Layout of base.
  • ​ScaleLayoutType (TensorLayout): Layout of scale.
  • ​GammaLayoutType (TensorLayout): Layout of gamma.
  • ​hc_mult (Int): Number of parallel residual streams.
  • ​hidden_size (Int): Width of one stream.
  • ​hc_sinkhorn_iters (Int): Sinkhorn-Knopp iterations used to project comb.
  • ​has_norm (Bool): Whether to RMS-normalize the collapsed row by gamma.

Args:

Was this page helpful?