Despite the promise, challenges persist to build an efficient MTL model suitable for real-world applications: ➊ during training, prior works [1]}, [2]}, [3]} indicate the competition of different tasks in training may degrade MTL, since the same weights might receive and be confused by conflicting update directions. Specifically,  [2]} reveals that negative cosine similarities between different tasks’ gradients are detrimental.  [5]}, [6]} confirm that conflicting gradients not only slow down convergence but also bias the learned representations against some tasks. That is only getting worse on compact models owing to their limited modeling capacity. To tackle the cross-task conflicts, solutions have been proposed by varying learning rate speeds of different tasks [3]}, using “cross-stitch” sharing [8]}, or re-balancing task gradients [2]}, [10]}, [3]}, [12]}. However, they either require task-specific design or significantly increase the model complexity which contradicts our efficiency goal.
➋ at inference, existing MTL regimes typically activate the entire backbone model unconditionally. However, many real systems only need to call upon one or a few tasks at
each moment, hence the “all activated” inference is heavily inefficient and non-scalable. For example, current regimes [13]}, [8]}, [15]}, [16]} have to activate the whole gigantic ResNet [17]} encoder even just to execute a single monocular depth estimation task or so.
If the number of tasks scale up [18]} and the backbone keeps growing bigger, the “per task” inference efficiency of the resultant MTL model could become catastrophically poor.
