We further evaluate M\(^3\) ViT on different variants of ViT including ViT-tiny and ViT-base; results are shown in Table REF . We compare against STL-B, MTL-B, and SoTA encoder-focused MTL model TAPS[1]}, Cross-Stitch [2]}. For TAPS, we adopt joint MTL strategy for comparable training longitude. It can be observed that MoE ViT-base increases the SoTA performance by a large margin, achieving +4.00% on PASCAL-Context and +8.32% on NYUD-v2. Meanwhile, it also consumes lower FLOPs compared to previous ResNet-based methods. MoE ViT-tiny consumes much fewer FLOPs than all previous methods (in particular, less than 1/10 FLOPs of the previous SoTA method Cross-Stitch). Additionally, our hardware co-design of MoE ViT-tiny achieves energy consumption an order of magnitude lower than Cross-Stitch.
<TABLE>