As shown in Table REF , even using a vanilla MTL-B framework, introducing MoE (M\(^2\) ViT) can achieve the highest performance over all previous encoder-focused works (+2.71% MTL performance); meanwhile, it significantly reduces their single task inference FLOPs (particularly, reducing Cross-Stitch by 88%). Comparing against Uncertainty [1]}, DWA [2]}, GradNorm [3]}, and MGDA [4]}, the superior performance of M\(^2\) ViT demonstrates its strong capacity in handling training conflict. Moreover, leveraging the Model-Accelerator co-design helps us to consume less than one-tenth the energy cost when deploying our model on FPGA. Some qualitative results are shown in Figure REF .
