We investigate task clustering for deep learning-based multi-task and few-shot learning in the settings with large numbers of diverse tasks. Our method measures task similarities using cross-task transfer performance matrix. Although this matrix provides us critical information regarding similarities between tasks, the uncertain task-pairs, i.e., the ones with extremely asymmetric transfer scores, may collectively mislead clustering algorithms to output an inaccurate task-partition. Moreover, when the number of tasks is large, generating the full transfer performance matrix can be very time consuming. To overcome these limitations, we propose a novel task clustering algorithm to estimate the similarity matrix based on the theory of matrix completion. The proposed algorithm can work on partially-observed similarity matrices based on only sampled task-pairs with reliable scores, ensuring its efficiency and robustness. Our theoretical analysis shows that under mild assumptions, the reconstructed matrix perfectly matches the underlying “true” similarity matrix with an overwhelming probability. The final task partition is computed by applying an efficient spectral clustering algorithm to the recovered matrix. Our results show that the new task clustering method can discover task clusters that benefit both multi-task learning and few-shot learning setups for sentiment classification and dialog intent classification tasks. This paper leverages knowledge distilled from a large number of learning tasks BID0 BID19 , or MAny Task Learning (MATL), to achieve the goal of (i) improving the overall performance of all tasks, as in multi-task learning (MTL); and (ii) rapid-adaptation to a new task by using previously learned knowledge, similar to few-shot learning (FSL) and transfer learning. Previous work on multi-task learning and transfer learning used small numbers of related tasks (usually ∼10) picked by human experts. By contrast, MATL tackles hundreds or thousands of tasks BID0 BID19 , with unknown relatedness between pairs of tasks, introducing new challenges such as task diversity and model inefficiency.MATL scenarios are increasingly common in a wide range of machine learning applications with potentially huge impact. Examples include reinforcement learning for game playing -where many numbers of sub-goals are treated as tasks by the agents for joint-learning, e.g. BID19 achieved the state-of-the-art on the Ms. Pac-Man game by using a multi-task learning architecture to approximate rewards of over 1,000 sub-goals (reward functions). Another important example is enterprise AI cloud services -where many clients submit various tasks/datasets to train machine learning models for business-specific purposes. The clients could be companies who want to know opinion from their customers on products and services, agencies that monitor public reactions to policy changes, and financial analysts who analyze news as it can potentially influence the stock-market. Such MATL-based services thus need to handle the diverse nature of clients' tasks.Challenges on Handling Diverse (Heterogeneous) Tasks Previous multi-task learning and fewshot learning research usually work on homogeneous tasks, e.g. all tasks are binary classification problems, or tasks are close to each other (picked by human experts) so the positive transfer between tasks is guaranteed. However, with a large number of tasks in a MATL setting, the above assumption may not hold, i.e. we need to be able to deal with tasks with larger diversity. Such diversity can be reflected as (i) tasks with varying numbers of labels: when tasks are diverse, different tasks could have different numbers of labels; and the labels might be defined in different label spaces without relatedness. Most of the existing multi-task and few-shot learning methods will fail in this setting; and more importantly (ii) tasks with positive and negative transfers: since tasks are not guaranteed to be similar to each other in the MATL setting, they are not always able to help each other when trained together, i.e. negative transfer BID22 between tasks. For example, in dialog services, the sentences "What fast food do you have nearby" and "Could I find any Indian food" may belong to two different classes "fast_food" and "indian_food" for a restaurant recommendation service in a city; while for a travel-guide service for a park, those two sentences could belong to the same class "food_options". In this case the two tasks may hurt each other when trained jointly with a single representation function, since the first task turns to give similar representations to both sentences while the second one turns to distinguish them in the representation space.A Task Clustering Based Solution To deal with the second challenge above, we propose to partition the tasks to clusters, making the tasks in each cluster more likely to be related. Common knowledge is only shared across tasks within a cluster, thus the negative transfer problem is alleviated. There are a few task clustering algorithm proposed mainly for convex models BID12 BID9 BID5 BID0 , but they assume that the tasks have the same number of labels (usually binary classification). In order to handle tasks with varying numbers of labels, we adopt a similarity-based task clustering algorithm. The task similarity is measured by cross-task transfer performance, which is a matrix S whose (i, j)-entry S ij is the estimated accuracy by adapting the learned representations on the i-th (source) task to the j-th (target) task. The above task similarity computation does not require the source task and target task to have the same set of labels, as a result, our clustering algorithm could naturally handle tasks with varying numbers of labels.Although cross-task transfer performance can provide critical information of task similarities, directly using it for task clustering may suffer from both efficiency and accuracy issues. First and most importantly, evaluation of all entries in the matrix S involves conducting the source-target transfer learning O(n 2 ) times, where n is the number of tasks. For a large number of diverse tasks where the n can be larger than 1,000, evaluation of the full matrix is unacceptable (over 1M entries to evaluate). Second, the estimated cross-task performance (i.e. some S ij or S ji scores) is often unreliable due to small data size or label noises. When the number of the uncertain values is large, they can collectively mislead the clustering algorithm to output an incorrect task-partition.To address the aforementioned challenges, we propose a novel task clustering algorithm based on the theory of matrix completion BID2 . Specifically, we deal with the huge number of entries by randomly sample task pairs to evaluate the S ij and S ji scores; and deal with the unreliable entries by keeping only task pairs (i, j) with consistent S ij and S ji scores. Given a set of n tasks, we first construct an n × n partially-observed matrix Y, where its observed entries correspond to the sampled and reliable task pairs (i, j) with consistent S ij and S ji scores. Otherwise, if the task pairs (i, j) are not sampled to compute the transfer scores or the scores are inconsistent, we mark both Y ij and Y ji as unobserved. Given the constructed partially-observed matrix Y, our next step is to recover an n × n full similarity matrix using a robust matrix completion approach, and then generate the final task partition by applying spectral clustering to the completed similarity matrix. The proposed approach has a 2-fold advantage. First, our method carries a strong theoretical guarantee, showing that the full similarity matrix can be perfectly recovered if the number of observed correct entries in the partially observed similarity matrix is at least O(n log 2 n). This theoretical result allows us to only compute the similarities of O(n log 2 n) instead of O(n 2 ) pairs, thus greatly reduces the computation when the number of tasks is large. Second, by filtering out uncertain task pairs, the proposed algorithm will be less sensitive to noise, leading to a more robust clustering performance.The task clusters allow us to handle (i) diverse MTL problems, by model sharing only within clusters such that the negative transfer from irrelevant tasks can be alleviated; and (ii) diverse FSL problems, where a new task can be assigned a task-specific metric, which is a linear combination of the metrics defined by different clusters, such that the diverse few-shot tasks could derive different metrics from the previous learning experience. Our results show that the proposed task clustering algorithm, combined with the above MTL and FSL strategies, could give us significantly better deep MTL and FSL algorithms on sentiment classification and intent classification tasks. In this paper, we propose a robust task-clustering method that not only has strong theoretical guarantees but also demonstrates significantly empirical improvements when equipped by our MTL and FSL algorithms. Our empirical studies verify that (i) the proposed task clustering approach is very effective in the many-task learning setting especially when tasks are diverse; (ii) our approach could efficiently handle large number of tasks as suggested by our theory; and (iii) cross-task transfer performance can serve as a powerful task similarity measure. Our work opens up many future research directions, such as supporting online many-task learning with incremental computation on task similarities, and combining our clustering approach with the recent learning-to-learn methods (e.g. BID18 ), to enhance our MTL and FSL methods.
We propose a matrix-completion based task clustering algorithm for deep multi-task and few-shot learning in the settings with large numbers of diverse tasks.
{"src": "scitldr-AIC_train", "paper_id": "B11bwYgfM"}