IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo module

moe_indices_packed

The MoE index build of packed routing rows, in one CTA.

routing is [tokens, 2 * top_k] float32: each token's expert ids, then their weights, as a router writes them and an allgather replicates them. Slot i = t * top_k + j goes to group id - expert_offset when that is below local_experts, else to the trailing group local_experts. The outputs are moe_create_indices' over local_experts + 1 groups, plus the groups' scale offsets and the weights copied dense.

A stable counting sort, so the order is deterministic: each warp counts its span of slots per group into its own row of a [WARPS, groups] table; after a meet, each group's column becomes the warps' exclusive bases; after another, one warp scans the group totals into the starts; after the last, each warp places its slots at start + base + its rank among the lanes with the same group. The schedule and the arithmetic are the Moka component's (Kernels/test/gpu/Moka/kernels/moe/moe_indices_packed_plan.mojo), whose ops call the helpers below.

comptime values​

ALIGN​

comptime ALIGN = 128

Scale rows of a group are padded to this many.

In​

comptime In[T: DType] = Pointer[Scalar[T], ImmUntrackedOrigin]

Parameters​

MAX_GROUPS​

comptime MAX_GROUPS = THREADS

A thread per group sums its column of the counts.

Out​

comptime Out[T: DType] = Pointer[Scalar[T], MutUntrackedOrigin]

Parameters​

POISONS​

comptime POISONS = 3

Planted faults, poison 1 through 3, each of which a test must catch: 1 files other devices' slots under group 0, 2 ranks a lane after every lane of its group, 3 skips the last slot's weight.

Smem​

comptime Smem = Pointer[UInt32, MutUntrackedOrigin, address_space=AddressSpace.SHARED]

THREADS​

comptime THREADS = (Int(32) * DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)

WARPS​

comptime WARPS = 32

Structs​

Functions​

  • ​column_bases: Thread t's group column: each warp's count becomes the counts of the warps before it, and the column's sum goes to totals.
  • ​count_span: Warp w counts its span per group into its row, and copies its slots' weights dense.
  • ​moe_indices_packed: Builds the indices of a.tokens packed routing rows.
  • ​scan_groups: Warp 0 scans the group totals into starts and writes the CSR, the ids, the scale offsets and the usage.
  • ​scatter_span: Warp w places its span's slots, in slot order within each group.
  • ​slot_group: Slot i's group; other devices' experts share the last one.
  • ​smem_words: The count table, then the group totals, then the starts.
  • ​warp_span: Warp w's slots [lo, hi): equal spans of whole 32-slot chunks.

Was this page helpful?