IMPORTANT: To view this page as Markdown, append `.md` to the URL (e.g. /get-started.md). For the complete documentation index, see llms.txt.
Skip to main content
For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).

Mojo function

moe_finalize_gather_kernel

def moe_finalize_gather_kernel[down_type: DType, weight_type: DType, out_type: DType, OutputLayoutType: TensorLayout, DownLayoutType: TensorLayout, RestoreOrderLayoutType: TensorLayout, RouterWeightLayoutType: TensorLayout, ValidRowsLayoutType: TensorLayout, topk: Int, vec: Int, block_threads: Int, bounded: Bool](output: TileTensor[out_type, OutputLayoutType, MutUntrackedOrigin], down: TileTensor[down_type, DownLayoutType, ImmUntrackedOrigin], restore_order: TileTensor[.uint32, RestoreOrderLayoutType, ImmUntrackedOrigin], router_weight: TileTensor[weight_type, RouterWeightLayoutType, ImmUntrackedOrigin], valid_rows: TileTensor[.uint32, ValidRowsLayoutType, ImmUntrackedOrigin])

Sums one token's top-k down rows per thread, top-k known at compile time.

One thread owns vec consecutive columns of one token's row: it reads the token's topk row ids and weights, issues all topk row loads, then accumulates them in ascending k in fp32. The arithmetic is the per-term acc += f32(down) * f32(weight) of the generic path, in the same order, so a call is bit-identical to it. When bounded, a row at or past valid_rows[0] is neither loaded nor accumulated, as in the generic path.

Parameters:

  • ​down_type (DType): DType of the permuted expert outputs.
  • ​weight_type (DType): DType of the router weights.
  • ​out_type (DType): DType of the combined output.
  • ​OutputLayoutType (TensorLayout): Layout of output (row-major).
  • ​DownLayoutType (TensorLayout): Layout of down (row-major).
  • ​RestoreOrderLayoutType (TensorLayout): Layout of restore_order.
  • ​RouterWeightLayoutType (TensorLayout): Layout of router_weight.
  • ​ValidRowsLayoutType (TensorLayout): Layout of valid_rows.
  • ​topk (Int): num_experts_per_token, as a compile-time constant.
  • ​vec (Int): Columns per thread; one vector load of down per row.
  • ​block_threads (Int): Threads per workgroup, 64 or 256. It also bounds what the register allocator keeps live, which is why the small-token launch declares its own 64-thread group.
  • ​bounded (Bool): Skip the rows of down from valid_rows[0] on.

Args:

Was this page helpful?