For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
routed_rows_finalize
The MoE combine with the routed-rows norm folded in, in one launch.
For token t, slot k names its expert-output row restore[t * top_k + k];
the rows below the routed count offsets[last] are this device's:
out[t] = sum_k [row_k routed] * w[t, k]
* bf16(rms_norm(bf16(x_k * (1 + gamma))))in float32, k in order, with norm.routed_rows_rms_norm's roundings, then
stored as out_dtype: float32, or bf16 rounded once from the float32 sum.
Persistent CTAs take tokens cta, cta + G, ...: a load lane bulk-copies the
token's top_k rows into a ring slot, routed or not (every restore value
indexes inside the buffer, so the copies need not wait on the count), and
eight work warps, a thread per two 16-byte pieces, sum each row's squares,
meet, and write the combined row. A row past the count may hold NaN, so it
is dropped, not weighed by zero. The schedule and the arithmetic are the
Moka component's
(Kernels/test/gpu/Moka/kernels/moe/routed_rows_finalize_plan.mojo), whose
ops call the helpers below.
comptime values
HIDDEN
comptime HIDDEN = 4096
Muse Spark's expert width, the only width the kernel takes.
In
Out
Piece
comptime Piece = SIMD[.bfloat16, 8]
PIECES
comptime PIECES = 2
16-byte pieces of a row per work thread.
POISONS
comptime POISONS = 4
Planted faults, poison 1 through 4, each of which a test must catch: 1 adds the slots past the routed count, 2 skips the input scale, 3 skips the last token, 4 truncates a bf16 output instead of rounding it.
ROW_BYTES
comptime ROW_BYTES = 8192
VEC
comptime VEC = 8
BF16 channels per 16-byte access.
Wide
comptime Wide = SIMD[.float32, 8]
WORK_WARPS
comptime WORK_WARPS = (Int(256) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)
WORKERS
comptime WORKERS = ((Int(256) // DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size) * DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)
Structs
-
RoutedRowsFinalizeArgs: The combine's operands, one kernel parameter. Rows are dense.
Functions
-
chan: The first channel of work threadrt's piecep: a warp's pieces are contiguous, so its 16-byte accesses coalesce. -
combine:accplus slot k's normalized piece by its router weight; an unrouted slot adds nothing. -
narrow: The float32 sum as the output's dtype, rounded to nearest even. -
routed_rows_finalize: Writes each token's combined row, float32 or bf16. -
scaled: A row piece times1 + gamma, rounded to bf16 as the graph does. -
smem_bytes: The scratch (the warps' sums, double buffered by trip parity), the full and empty barriers, then the ring slots, 128-byte aligned. -
square_sum: A thread's sum of squares over its pieces. -
stages_for: Ring slots fortop_krows a slot: three while they fit in 200 KB. -
sum_at: Where warpwkeeps slot k's sum: by trip parity, then slot, then warp, so a slot's sums are one vector.