For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
moe_finalize_gather_kernel
def moe_finalize_gather_kernel[down_type: DType, weight_type: DType, out_type: DType, OutputLayoutType: TensorLayout, DownLayoutType: TensorLayout, RestoreOrderLayoutType: TensorLayout, RouterWeightLayoutType: TensorLayout, ValidRowsLayoutType: TensorLayout, topk: Int, vec: Int, block_threads: Int, bounded: Bool](output: TileTensor[out_type, OutputLayoutType, MutUntrackedOrigin], down: TileTensor[down_type, DownLayoutType, ImmUntrackedOrigin], restore_order: TileTensor[.uint32, RestoreOrderLayoutType, ImmUntrackedOrigin], router_weight: TileTensor[weight_type, RouterWeightLayoutType, ImmUntrackedOrigin], valid_rows: TileTensor[.uint32, ValidRowsLayoutType, ImmUntrackedOrigin])
Sums one token's top-k down rows per thread, top-k known at compile time.
One thread owns vec consecutive columns of one token's row: it reads the
token's topk row ids and weights, issues all topk row loads, then
accumulates them in ascending k in fp32. The arithmetic is the per-term
acc += f32(down) * f32(weight) of the generic path, in the same order, so
a call is bit-identical to it. When bounded, a row at or past
valid_rows[0] is neither loaded nor accumulated, as in the generic path.
Parameters:
- down_type (
DType): DType of the permuted expert outputs. - weight_type (
DType): DType of the router weights. - out_type (
DType): DType of the combined output. - OutputLayoutType (
TensorLayout): Layout ofoutput(row-major). - DownLayoutType (
TensorLayout): Layout ofdown(row-major). - RestoreOrderLayoutType (
TensorLayout): Layout ofrestore_order. - RouterWeightLayoutType (
TensorLayout): Layout ofrouter_weight. - ValidRowsLayoutType (
TensorLayout): Layout ofvalid_rows. - topk (
Int):num_experts_per_token, as a compile-time constant. - vec (
Int): Columns per thread; one vector load ofdownper row. - block_threads (
Int): Threads per workgroup, 64 or 256. It also bounds what the register allocator keeps live, which is why the small-token launch declares its own 64-thread group. - bounded (
Bool): Skip the rows ofdownfromvalid_rows[0]on.
Args:
- output (
TileTensor[out_type, OutputLayoutType, MutUntrackedOrigin]):[num_tokens, row_chunks * vec], one combined row per token. - down (
TileTensor[down_type, DownLayoutType, ImmUntrackedOrigin]):[num_tokens * topk, row_chunks * vec], expert-permuted rows. - restore_order (
TileTensor[.uint32, RestoreOrderLayoutType, ImmUntrackedOrigin]):[num_tokens * topk], token-major index to adownrow. - router_weight (
TileTensor[weight_type, RouterWeightLayoutType, ImmUntrackedOrigin]):[num_tokens, topk], per-(token, expert) weight. - valid_rows (
TileTensor[.uint32, ValidRowsLayoutType, ImmUntrackedOrigin]):[1]row bound ofdown; read only whenbounded.