Different methods have been used in the literature in order to determine the number of experts for these models [1]}: growing models where the experts with the worst performance decompose into a MoE themselves [2]} [3]}, pruning models where they start with a large number of experts and reduce the number by combining/removing experts [4]}, and exhaustive search in cases where tree topologies are not overly complex [5]}.
