IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

routed_rows_finalize

The MoE combine with the routed-rows norm folded in, in one launch.

For token t, slot k names its expert-output row restore[t * top_k + k]; the rows below the routed count offsets[last] are this device's:

out[t] = sum_k [row_k routed] * w[t, k]
               * bf16(rms_norm(bf16(x_k * (1 + gamma))))

in float32, k in order, with norm.routed_rows_rms_norm's roundings, then stored as out_dtype: float32, or bf16 rounded once from the float32 sum. Persistent CTAs take tokens cta, cta + G, ...: a load lane bulk-copies the token's top_k rows into a ring slot, routed or not (every restore value indexes inside the buffer, so the copies need not wait on the count), and eight work warps, a thread per two 16-byte pieces, sum each row's squares, meet, and write the combined row. A row past the count may hold NaN, so it is dropped, not weighed by zero. The schedule and the arithmetic are the Moka component's (Kernels/test/gpu/Moka/kernels/moe/routed_rows_finalize_plan.mojo), whose ops call the helpers below.

comptime values​

HIDDEN​

comptime HIDDEN = 4096

Muse Spark's expert width, the only width the kernel takes.

In​

comptime In[T: DType] = Pointer[Scalar[T], ImmUntrackedOrigin]

Parameters​

Out​

comptime Out[T: DType] = Pointer[Scalar[T], MutUntrackedOrigin]

Parameters​

Piece​

comptime Piece = SIMD[.bfloat16, 8]

PIECES​

comptime PIECES = 2

16-byte pieces of a row per work thread.

POISONS​

comptime POISONS = 4

Planted faults, poison 1 through 4, each of which a test must catch: 1 adds the slots past the routed count, 2 skips the input scale, 3 skips the last token, 4 truncates a bf16 output instead of rounding it.

ROW_BYTES​

comptime ROW_BYTES = 8192

VEC​

comptime VEC = 8

BF16 channels per 16-byte access.

Wide​

comptime Wide = SIMD[.float32, 8]

WORK_WARPS​

comptime WORK_WARPS = (Int(256) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)

WORKERS​

comptime WORKERS = ((Int(256) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size) * DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)

Structs​

Functions​

  • ​chan: The first channel of work thread rt's piece p: a warp's pieces are contiguous, so its 16-byte accesses coalesce.
  • ​combine: acc plus slot k's normalized piece by its router weight; an unrouted slot adds nothing.
  • ​narrow: The float32 sum as the output's dtype, rounded to nearest even.
  • ​routed_rows_finalize: Writes each token's combined row, float32 or bf16.
  • ​scaled: A row piece times 1 + gamma, rounded to bf16 as the graph does.
  • ​smem_bytes: The scratch (the warps' sums, double buffered by trip parity), the full and empty barriers, then the ring slots, 128-byte aligned.
  • ​square_sum: A thread's sum of squares over its pieces.
  • ​stages_for: Ring slots for top_k rows a slot: three while they fit in 200 KB.
  • ​sum_at: Where warp w keeps slot k's sum: by trip parity, then slot, then warp, so a slot's sums are one vector.

Was this page helpful?