Many real applications show a great deal of interest in learning multiple tasks from different data sources/modalities with unbalanced samples and dimensions. Unfortunately, existing cutting-edge deep multi-task learning (MTL) approaches cannot be directly applied to these settings, due to either heterogeneous input dimensions or the heterogeneity in the optimal network architectures of different tasks. It is thus demanding to develop knowledge-sharing mechanism to handle the intrinsic discrepancies among network architectures across tasks. To this end, we propose a flexible knowledge-sharing framework for jointly learning multiple tasks from distinct data sources/modalities. The proposed framework allows each task to own its task (data)-specific network design, via utilizing a compact tensor representation, while the sharing is achieved through the partially shared latent cores. By providing more elaborate sharing control with latent cores, our framework is effective in transferring task-invariant knowledge, yet also being efficient in learning task-specific features. Experiments on both single and multiple data sources/modalities settings display the promising results of the proposed method, especially favourable in insufficient data scenarios. Multi-task learning (MTL) (Caruana, 1997; Maurer et al., 2016) is an approach for boosting the overall performance of each individual task by learning multiple related tasks simultaneously. In the deep learning setting, jointly fitting sufficiently flexible deep neural networks (DNNs) to data of multiple tasks can be seen as adding an inductive bias to the deep models, which can facilitate the learning of feature representations that are preferable by all tasks. Recently, the deep MTL has been successfully explored in a broad range of applications, such as computer vision (Zhang et al., 2014; Misra et al., 2016) , natural language processing (Luong et al., 2015; , speech recognition Huang et al., 2015) and so on. Nevertheless, one key challenge in deep MTL remains largely unaddressed, that is, almost all existing deep MTL approaches (Yang & Hospedales, 2017; Long et al., 2017) restrict themselves only to the setting of multi-label learning (or multi-output regression) (Zhang & Yang, 2017) . In other words, different tasks must be fed with input data from the same source (or domain). Such requirement, however, seriously limits the applicability of those models to a more realistic scenario of deep MTL, where the tasks involve distinct data sources (domains) with unbalanced sample sizes or dimensions. More specifically, tasks from some domains with abundant samples or small input dimensions are relatively easy to handle, whereas tasks from other domains are quite challenging due to the insufficient training data and large dimensionality. For instance, classifying hand-written digits (MNIST dataset (LeCun et al., 1998) ) is somewhat similar to the recognition of hand-drawn characters (Omniglot dataset (Lake et al., 2015) ). The Omniglot task is much harder than the MNIST task, as each character in Omniglot has only 20 training samples, while the input dimensionality is about 15 times larger than MNIST digit. As another example, predicting binary attributes (i.e., 'young', 'bald', 'receding hairline') from human face images (CelebA dataset (Liu et al., 2015) ) ought to be related to the age group classification using human photos taken in the wild (Adience dataset (Eidinger et al., 2014) ). The Adience task turns out to be the more difficult one since the wild images are not preprocessed and 7.6 times fewer than CelebA samples. Hence, it makes good sense to jointly learn these multi-task representation learning (DMTRL) for CNN setting and our TRMTL (general setting and CNN setting) w.r.t. two tasks. The shared portion is depicted in yellow. MRN: original weights are totally shared at the lower layers and the relatedness between tasks at the top layers is modelled by tensor normal priors. DMTRL (TT or Tucker): all layer-wise weights must be equal-shape so as to be stacked and decomposed into factors. For each task, almost all the factors are shard at each layer except the very last 1D vector. Such pattern of sharing is identical at all layers. TRMTL (General): layer-wise weights are separately encoded into TR-formats for different tasks, and a subset of latent cores are selected to be tied across two tasks. The portions of sharing can be different from layer to layer. TRMTL (CNN): spatial cores (height and width cores) in the tensorized convolutional kernel are shared, while cores of input/output channels of the kernel are task-specific. tasks to extract better feature representations, especially for the hard tasks, which could be achieved through transferring domain-specific knowledge from easy tasks. Unfortunately, existing cutting-edge deep MTL models are only suited for the multi-label learning where different tasks share the same training inputs (i.e., X i = X j for i = j, where X i denotes the input for task T i ), and thus cannot be directly applied to above learning scenarios. This is due to those models fail to provide knowledge-sharing mechanisms that can cope with the intrinsic discrepancies among network architectures across tasks. Such discrepancies either arise from the heterogeneous dimensions of input data or from the heterogeneous designs of layer-wise structures. Conventionally, knowledge-sharing mechanisms of deep MTL can be hard or soft parameter sharing (Ruder, 2017) . Hard sharing models (Zhang et al., 2014; Yin & Liu, 2017) share all parameters at the lower layers but with no parameters being shared at the upper layers across tasks. Soft sharing models (Duong et al., 2015; Yang & Hospedales, 2016; Long & Wang, 2015) , on the other hand, learn one DNN per task with its own set of parameters, and the tasks are implicitly connected through imposing regularization terms on the aligned weights. The common issue with above mechanisms is that, for the sharing part, the network architectures of all tasks are strictly required to be identical. It turns out that some of the tasks have to compromise on a sub-optimal network architecture, which may lead to the deterioration in the overall performance. Ideally, at all potentially shared layers, each task should be capable of encoding both task-specific and task-independent portions of variation. To overcome this limitation, we propose a latent-subspace knowledge-sharing mechanism that allows to associate each task with distinct source (domain) of data. By utilizing tensor representation, different portions of parameters can be shared via latent cores as common knowledge at distinct layers, so that each task can better convey its private knowledge. In this work, we realize our proposed framework via tensor ring (TR) format and refer it as tensor ring multi-task learning (TRMTL), as shown in Figure 1 . Our main contributions are twofold: (1) we offer a new distributed knowledge-sharing mechanism that can address the discrepancies of network architectures among tasks. Compared to existing deep MTL models that are only for multi-label learning, the joint learning of tasks from multi-datasets (multi-domains) with heterogeneous architectures becomes feasible. (2) we provide a TR-based implementation of the proposed framework, which further enhances the performance of deep MTL models in terms of both compactness and expressive power. Our general TRMTL framework relies on the manual selection of shared cores, i.e., one need to specify the number of shared cores C at each layer if we choose to share the cores in a left-to-right order across tasks. Although we can employ some efficient heuristics, the search space of this hyperparameter may grow rapidly as number of the layers increase. Besides the greedy search, a more sophisticated and possible option is to automatically select sharable core pairs that have the highest similarity. We may consider two cores as a candidate pair if the same perturbation of the two cores induces similar changes in the errors of respective tasks. In this way, one can adaptively select most similar cores from tasks according to a certain threshold, leaving the rest as private cores. We should also point out that tensorization operation plays a key role in our proposed sharing mechanism. Due to the tensorization, the cores can be shared in a much finer granularity via our TRMTL framework. Furthermore, tensorizing weight matrix into high-order weight tensor yields more compact tensor network format (with much lower overall ranks), and thus a higher compression ratio for parameters. In contrast, DMTRL tends to produce a lot more parameters without tensorization. In this work, we have extended the conventional deep MTL to a broader paradigm where multiple tasks may involve more than one source data domain. To resolve the issues caused by the discrepancies among different tasks' network structures, we have introduced a novel knowledge sharing framework for deep MTL, by partially sharing latent cores via tensor network format. Our method is empirically verified on various learning settings and achieves the state-of-the-art results in helping tasks to improve their overall performance. of T tasks to be equal-sized, so that these weights could be stacked up into one weight matrix W ∈ R M ×T . The work (Kumar & Daume III, 2012 ) assumes W to be low-rank and factorizes it as W = LS. Here, L ∈ R M ×K consists of K task-independent latent basis vectors, whereas each column vector of S ∈ R K×T is task-specific and contains the mixing coefficients of these common latent bases. Yang & Hospedales (2017) extended this to its tensorial counterpart deep multi-task representation learning (DMTRL) by making use of tensor factorization. Likewise, DMTRL starts by putting the equal-shaped weight matrices side by side along the 'task' mode to form a 3rd-order weight tensor W ∈ R M ×N ×T . In the case of CNN, this weight tensor corresponds to a 5th-order filter tensor K ∈ R H×W ×U ×V ×T . DMTRL then factorizes W (or K), for instance via TT-format, into 3 TT-cores (or 5 TT-cores for K) Yang & Hospedales (2017) . Analogously, the first 2 TT-cores (or the first 4 TT-cores) play exactly the same role as L for the common knowledge; the very last TT-core is in fact a matrix (similar to S), with each column representing the task-specific information. The fundamental difference between our TRMTL and DMTRL is that ours can tailor heterogeneous network structures to various tasks. In contrast, DMTRL is not flexible enough to deal with such variations with tasks. Specifically, our TRMTL differs widely with DMTRL and generalizes DMTRL from a variety of aspects. In order to reach TRMTL from DMTRL-TT, one needs to take four major types of generalizations (G1-G4), as shown in Figure 6 . Firstly (in G1), TRMTL tensorizes the weight into a higher-order weight tensor before factorizing it. By doing so, the weight can be embedded into more latent cores than that of just 3 cores (or 5 cores) in DMTRL, which yields a more compact model and makes the sharing at a finer granularity feasible. Secondly (in G2), DMTRL stringently requires that the first D-1 cores (D is weight tensor's order) must be all shared at every hidden layer, only the last vector is kept for private knowledge. By contrast, TRMTL allows for any sharing pattern at distinct layer. Thirdly (in G3), there is no need for layerwise weights to be equal-sized and stacked into one big tensor as in TRMTL, each task may have its individual input domains. Finally (in G4), TRMTL further generalizes TT to TR-format. For each task in DMTRL, the first core must be a matrix and the last core must be a vector (with both border rank and outer mode size being 1). Notice that our TRMTL also conceptually subsumes DMTRLTucker in terms of the first three aspects of generalizations (G1-G3). It is also worth mentioning that (Wang et al., 2018) only applies TR-format for weight compression in a single deep net, whereas ours incorporates a more general tensor network framework into the deep MTL context. The authors of (Long et al., 2017 ) lately proposed multilinear relationship network (MRN) which incorporates tensor normal priors over the parameter tensors of the task-specific layers. However, like methods (Zhang et al., 2014; Ouyang et al., 2014; Chu et al., 2015) , MRN follows the architecture where all the lower layers are shared, which is also not tailored for the extended MTL paradigm, and may harm the transferability if tasks are not that tightly correlated. In addition, the relatedness of tasks is captured by the covariance structures over features, classes and tasks. Constantly updating these covariance matrices (via SVD in (Long et al., 2017) ) becomes computationally prohibitive for large scale networks. Compared to these non-latent-subspace methods, TRMTL is highly compact and needs much fewer parameters, which is obviously advantageous in tasks with small sample size. The detailed specification of network architecture and factorized TRRL representation of the experiments on MNIST dataset are recorded in Table 6 . In Table 7 , our TRMTL achieves the best results and is robust to small perturbation of C for pattern selection, since both '410' and '420' patterns obtain similarly good performance.
a distributed latent-space based knowledge-sharing framework for deep multi-task learning
{"src": "scitldr-AIC_train", "paper_id": "HygUBaEFDB"}