As we target an efficient MTL system under the single-task inference setting, we conduct experiments on encoder-focused architectures (more details in Section ). MTL-B [1]} is a vanilla multi-task learning baseline model which is composed of a shared backbone in combination with task-specific heads. Several state-of-the-art (SoTA) encoder-focused MTL models, including MTAN [2]}, Cross-Stitch [3]} and NDDR-CNN [4]}, improve MTL-B by proposing feature sharing methods in the encoder. Our methods are all conducted on vanilla MTL-B, namely, applying MTL on ViT (M-ViT), adding task-dependent MoE design (M\(^2\) ViT), and adding hardware co-design on FPGA (M\(^3\) ViT). We also compare with previous works that handle the multi-task training conflicts problem, including uncertainty weighting [5]}, GradNorm [6]}, DWA [7]}, and MGDA [2]}, and they are evaluated on MTL-B.
Single task learning baseline (STL-B) is used for MTL performance evaluation \(\Delta _{m}\) . As multi-gate MoE shows better performance than task-conditioned MoE, the reported M\(^2\) ViT and M\(^3\) ViT results are based on the multi-gate design.
