To evaluate the performance of our model, we further conduct experiments on different levels of MTL difficulties with different numbers of tasks. We compare between STL-B, MTL-B, SoTA work Cross-Stitch [1]}, MTL-B with ViT-small (M-ViT), and MTL-B with MoE ViT-small (M\(^3\) ViT); results are shown in Table REF . It can be observed that M\(^3\) ViT consistently outperforms MTL-B with less computational FLOPs on different numbers of tasks on both NYUD-v2 and PASCAL-Context. Compared to SoTA encoder-focused work Cross-Stitch, although M\(^3\) ViT performs slightly lower on NYUD-v2 with two tasks, it achieves better performance on all the other settings. In particular, it surpasses Cross-Stitch on NYUD-v2 when the number of tasks increases to four (—0.91% vs.  —3.26%), which demonstrates the strong capacity of our model on handling more tasks. On PASCAL-Context dataset, introducing MoE (M\(^3\) ViT) can achieve much better performance than Cross-Stitch. Noticing that M\(^3\) ViT performs slightly worse on normal estimation and saliency detection tasks, we speculate that it is because these two tasks require a relatively small receptive field to retain a detailed estimation, and Cross-Stitch allows to use limited local information (i.e., small receptive field) when fusing the activations from the different single-task networks. But for other tasks that require larger receptive fields, our model performs significantly better than Cross-Stitch, since our task-dependent MoE design helps effectively avoid different tasks’ training conflict. Meanwhile, M\(^3\) ViT consumes much less computational power than previous methods.
