Multi-task learning has been successful in modeling multiple related tasks with large, carefully curated labeled datasets. By leveraging the relationships among different tasks, multi-task learning framework can improve the performance significantly. However, most of the existing works are under the assumption that the predefined tasks are related to each other. Thus, their applications on real-world are limited, because rare real-world problems are closely related. Besides, the understanding of relationships among tasks has been ignored by most of the current methods. Along this line, we propose a novel multi-task learning framework - Learning To Transfer Via Modelling Multi-level Task Dependency, which constructed attention based dependency relationships among different tasks. At the same time, the dependency relationship can be used to guide what knowledge should be transferred, thus the performance of our model also be improved. To show the effectiveness of our model and the importance of considering multi-level dependency relationship, we conduct experiments on several public datasets, on which we obtain significant improvements over current methods. Multi-task learning (Caruana, 1997) aims to train a single model on multiple related tasks jointly, so that useful knowledge learned from one task can be transferred to enhance the generalization performance of other tasks. Over the last few years, different types of multi-task learning mechanisms (Sener & Koltun, 2018; Guo & Farooq, 2018; Ish, 2016; Lon, 2015) have been proposed and proved better than single-task learning methods from natural language processing (Palmer et al., 2017) and computer vision (Cortes et al., 2015) to chemical study (Ramsundar et al., 2015) . Despite the success of multi-task learning, when applying to 'discrete' data (graph/text), most of the current multi-task learning frameworks (Zamir et al., 2018; Ish, 2016) only leverage the general task dependency with the assumption that the task dependency remains the same for (1) different data samples; and (2) different sub-structures (node/word) in one data sample (graph/text). However, this assumption is not always true in many real-world problems. (1) Different data samples may have different task dependency. For example, when we want to predict the chemical properties of a particular toxic molecule, despite the general task dependency, its representations learned from toxicity prediction tasks should be more significant than the other tasks. (2) Even for the same data sample, different sub-structures may have different task dependency. Take sentence classification as an example. Words like 'good' or 'bad' may transfer more knowledge from sentiment analysis tasks, while words like 'because' or 'so' may transfer more from discourse relation identification tasks. In this work, to accurately learn the task dependency in both general level and data-specific level, we propose a novel framework, 'Learning to Transfer via ModellIng mulTi-level Task dEpeNdency' (L2T-MITTEN). The general task dependency is learned as a parameterized weighted dependency graph. And the data-specific task dependency is learned with the position-wise mutual attention mechanism. The two-level task dependency can be used by our framework to improve the performance on multiple tasks. And the objective function of multi-task learning can further enhance the quality of the learned task dependency. By iteratively mutual enhancement, our framework can not only perform better on multiple tasks, but also can extract high-quality dependency structures at different levels, which can reveal some hidden knowledge of the datasets. Another problem is that to transfer task-specific representations between every task pair, the number of transfer functions will grow quadratically as the number of tasks increases, which is unaffordable. To solve this, we develop a universal representation space where all task-specific representations get mapped to and all target tasks can be inferred from. This decomposition method reduces the space complexity from quadratic to linear. We validate our multi-task learning framework extensively on different tasks, including graph classication, node classification, and text classification. Our framework outperforms all the other state-ofthe-art (SOTA) multi-task methods. Besides, we show that L2T-MITTEN can be used as an analytic tool to extract interpretable task dependency structures at different levels on real-world datasets. Our contributions in this work are threefold: • We propose a novel multi-task learning framework to learn to both general task dependency and data-specific task dependency. The learned task dependency structures can be mutually enhanced with the objective function of multi-task learning. • We develop a decomposition method to reduce the space complexity needed by transfer functions from quadratic to linear. • We conduct extensive experiments on different real-world datasets to show the effectiveness of our framework and the importance of modelling multi-level task dependency. We propose L2T-MITTEN, a novel multi-task learning framework that (1) employs the positionwise mutual attention mechanism to learn the multi-level task dependency; (2) transfers the taskspecific representations between tasks with linear space-efficiency; and (3) uses the learned multilevel task dependency to guide the inference. We design three experimental settings where training data is sufficient, imbalanced or deficient, with multiple graph/text datasets. Experimental results demonstrate the superiority of our method against both classical and SOTA baselines. We also show that our framework can be used as an analytical tool to extract the task dependency structures at different levels, which can reveal some hidden knowledge of tasks and of datasets A DATASET SUMMARY Figure 4 , in the Encoder Block, we use several layers of graph convolutional layers (Kipf & Welling, 2016) followed by the layer normalization (Ba et al., 2016) . In the Readout Block, for graph-level task, we use set-to-set (Vinyals et al., 2015) as the global pooling operator to extract the graph-level representation which is later fed to a classifier; while for node-level task, we simply eliminate the global pooling layer and feed the node-level representation directly to the classifier. Figure 4: Graph convolutional networks architecture. Note that in node-level task, the Set2Set layer (global pooling) is eliminated.
We propose a novel multi-task learning framework which extracts multi-view dependency relationship automatically and use it to guide the knowledge transfer among different tasks.
{"src": "scitldr-AIC_train", "paper_id": "BklhsgSFvB"}