MoE contains a series of sub-models (i.e., experts) and performs conditional computation in an input-dependent fashion [1]}, [2]}, [3]}, [4]}, [5]}, based on learned or deterministic routing policies [6]}, [5]}. The traditional dense MoEs suffer from intensive computational costs since they select all experts [8]}. Recent studies [9]}, [10]}, [11]} in natural language processing (NLP) propose sparse MoE that sparsely activates a few experts during both training and inference, thus substantially reducing the cost and allowing gigantic language models even with trillions of parameters [11]}. Unfortunately, such a sparse-gated manner still has limitations of unstable training and imbalanced selections among experts. Various solutions are invented from regularization [13]}, [10]}, [11]} and optimization [16]}, [17]} perspectives. Moreover, MoE has drawn increasing popularity in computer vision [8]}, [19]}, [20]}, [21]}, [22]}, [23]}, [24]}, where it mainly focuses on considerably smaller network backbones compared to the ones in NLP. For instance, [21]} and [22]} formulate the channel and kernel of convolutional layers as experts and establish the MoE framework. Several pioneer investigations also explore MoE for multi-task learning, which are related to this work. Particularly, [27]}, [28]}, [29]} introduce task-specific gating networks to choose different parts of models for processing information from each task. They present certain possibilities of using MoE to solve MTL problems in some cases like classification for medical signals [28]}, digital number images (MNIST) [29]}, and recommendation
systems [27]}. We make a further attempt to adapt MoE into a compact model for dense prediction multi-task learning, along with software-hardware co-design.
