Saliency-Regularized Deep Multi-Task Learning
Multitask learning is a framework that enforces multiple learning tasks to
share knowledge to improve their generalization abilities. While shallow
multitask learning can learn task relations, it can only handle predefined
features. Modern deep multitask learning can jointly learn latent features and
task sharing, but they are obscure in task relation. Also, they predefine which
layers and neurons should share across tasks and cannot learn adaptively. To
address these challenges, this paper proposes a new multitask learning
framework that jointly learns latent features and explicit task relations by
complementing the strength of existing shallow and deep multitask learning
scenarios. Specifically, we propose to model the task relation as the
similarity between task input gradients, with a theoretical analysis of their
equivalency. In addition, we innovatively propose a multitask learning
objective that explicitly learns task relations by a new regularizer.
Theoretical analysis shows that the generalizability error has been reduced
thanks to the proposed regularizer. Extensive experiments on several multitask
learning and image classification benchmarks demonstrate the proposed method
effectiveness, efficiency as well as reasonableness in the learned task
relation patterns.