For the complete documentation index, see llms.txt. Markdown versions of all pages are available by appending .md to any URL (e.g. /get-started.md).
Mojo function
moe_finalize_bounded
def moe_finalize_bounded[down_type: DType, weight_type: DType, out_type: DType, //](output: MutTileTensor[out_type, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type], down: ImmTileTensor[down_type, Engine=down.Engine, address_space=down.address_space, linear_idx_type=down.linear_idx_type], restore_order: ImmTileTensor[.uint32, Engine=restore_order.Engine, address_space=restore_order.address_space, linear_idx_type=restore_order.linear_idx_type], router_weight: ImmTileTensor[weight_type, Engine=router_weight.Engine, address_space=router_weight.address_space, linear_idx_type=router_weight.linear_idx_type], valid_rows: ImmTileTensor[.uint32, Engine=valid_rows.Engine, address_space=valid_rows.address_space, linear_idx_type=valid_rows.linear_idx_type], context: DeviceContext)
Runs moe_finalize over only the rows of down below valid_rows[0].
A row of down at or past the bound adds nothing to its token's sum and
is never read, so it may hold anything, NaN included. Expert-parallel
layers sort the slots of experts held on other devices last, past the
local rows, and the bound drops them without masking down first.
Parameters:
- down_type (
DType): DType of the permuted expert outputs. - weight_type (
DType): DType of the router weights. - out_type (
DType): DType of the combined output.
Args:
- output (
MutTileTensor[out_type, Engine=output.Engine, address_space=output.address_space, linear_idx_type=output.linear_idx_type]): Asmoe_finalize. - down (
ImmTileTensor[down_type, Engine=down.Engine, address_space=down.address_space, linear_idx_type=down.linear_idx_type]): Asmoe_finalize; rows fromvalid_rows[0]on are ignored. - restore_order (
ImmTileTensor[.uint32, Engine=restore_order.Engine, address_space=restore_order.address_space, linear_idx_type=restore_order.linear_idx_type]): Asmoe_finalize. - router_weight (
ImmTileTensor[weight_type, Engine=router_weight.Engine, address_space=router_weight.address_space, linear_idx_type=router_weight.linear_idx_type]): Asmoe_finalize. - valid_rows (
ImmTileTensor[.uint32, Engine=valid_rows.Engine, address_space=valid_rows.address_space, linear_idx_type=valid_rows.linear_idx_type]):[1]count of the leading rows ofdownthat count, on the device. - context (
DeviceContext): The device context.