The generic multi-task learning problem has been studied for a long history. Some non-deep learning-based methods propose to use distance metric [1]}, [2]}, [3]}, probabilistic prior  [4]}, [5]}, [6]}, [7]} to model the common information among tasks.
With the emergence of the deep learning technique, MTL [8]}, [9]}, [10]}, [11]}, [12]}, [13]} is performed to learn shared representation among tasks. The emergence of ViT further makes it possible to extend the task range from only vision tasks to other modalities tasks (e.g., text, audio) [14]}, [15]}, [16]}, [17]}, [18]}. Current MTL models can be roughly categorized into two types based on where the task interactions take place in the network. The encoder-focused architectures [10]}, [9]}, [21]}, [22]} only share information in the
encoder, before
decoding each task with an independent task-specific head. Cross-stitch networks [10]} introduce linear combination in each layer. NDDR-CNN [21]} improves it by dimensional reduction. MTAN [22]} leverages an attention mechanism to learn between tasks. TAPS [26]} adapts a base model to a new task by
modifying a small task-specific subset of layers.
The second type, decoder-focused models [12]}, [13]}, [29]}, [30]}, make initial task predictions in decoder and then leverage features from these initial predictions to further improve output. Although they report higher performance, their models consume a large number of FLOPs, according to [8]}. This makes it difficult to deploy them onto those real-world systems that are often resource-constrained or latency-sensitive. And they need to execute all the tasks for initial prediction, which is heavily inefficient in the common scenario when only one or few tasks
are needed.
Hence, we focus on encoder-focused architecture in this work. Many methods [32]}, [33]}, [34]}, [22]} are also proposed to handle the MTL training conflicts problem.
