For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo module
moe_indices_packed
The MoE index build of packed routing rows, in one CTA.
routing is [tokens, 2 * top_k] float32: each token's expert ids, then
their weights, as a router writes them and an allgather replicates them.
Slot i = t * top_k + j goes to group id - expert_offset when that is
below local_experts, else to the trailing group local_experts. The
outputs are moe_create_indices' over local_experts + 1 groups, plus the
groups' scale offsets and the weights copied dense.
A stable counting sort, so the order is deterministic: each warp counts its
span of slots per group into its own row of a [WARPS, groups] table;
after a meet, each group's column becomes the warps' exclusive bases; after
another, one warp scans the group totals into the starts; after the last,
each warp places its slots at start + base + its rank among the lanes with
the same group. The schedule and the arithmetic are the Moka component's
(Kernels/test/gpu/Moka/kernels/moe/moe_indices_packed_plan.mojo), whose
ops call the helpers below.
comptime values
ALIGN
comptime ALIGN = 128
Scale rows of a group are padded to this many.
In
MAX_GROUPS
comptime MAX_GROUPS = THREADS
A thread per group sums its column of the counts.
Out
POISONS
comptime POISONS = 3
Planted faults, poison 1 through 3, each of which a test must catch: 1 files other devices' slots under group 0, 2 ranks a lane after every lane of its group, 3 skips the last slot's weight.
Smem
comptime Smem = Pointer[UInt32, MutUntrackedOrigin, address_space=AddressSpace.SHARED]
THREADS
comptime THREADS = (Int(32) * DefaultMaxPlugin if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 0](), True) else [PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if (xor _index_lt[PluginSelector._find[_current_target(), 0, trap_if_missing=False](), 3](), True) else CudaMaxPlugin, HipMaxPlugin, MetalMaxPlugin[PluginSelector._find[_current_target(), 0, trap_if_missing=False]()] if identical(_NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin), _NoMaxPlugin) else _NoMaxPlugin if (xor _index_lt[TargetPluginSelector._find[0](), len(#kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>)](), True) else #kgen.get_target_plugin<apply(:!lit.generator<() -> !kgen.target> @std::@sys::@info::@"_current_target()"), :param_list<trait<@std::@traits::@anytype::@AnyType>> [@max::@_plugin::@_impl::@_NoMaxPlugin]>[TargetPluginSelector._find[0]()](MaxPlugin).HookFor[CompilationTarget(Int(0))].warp_size)
WARPS
comptime WARPS = 32
Structs
-
PackedIndicesArgs: The index build's operands, one kernel parameter.
Functions
-
column_bases: Threadt's group column: each warp's count becomes the counts of the warps before it, and the column's sum goes tototals. -
count_span: Warpwcounts its span per group into itsrow, and copies its slots' weights dense. -
moe_indices_packed: Builds the indices ofa.tokenspacked routing rows. -
scan_groups: Warp 0 scans the group totals intostartsand writes the CSR, the ids, the scale offsets and the usage. -
scatter_span: Warpwplaces its span's slots, in slot order within each group. -
slot_group: Sloti's group; other devices' experts share the last one. -
smem_words: The count table, then the group totals, then the starts. -
warp_span: Warpw's slots[lo, hi): equal spans of whole 32-slot chunks.