Most contemporary multi-task learning methods assume linear models. This setting is considered shallow in the era of deep learning. In this paper, we present a new deep multi-task representation learning framework that learns cross-task sharing structure at every layer in a deep network. Our approach is based on generalising the matrix factorisation techniques explicitly or implicitly used by many conventional MTL algorithms to tensor factorisation, to realise automatic learning of end-to-end knowledge sharing in deep networks. This is in contrast to existing deep learning approaches that need a user-defined multi-task sharing strategy. Our approach applies to both homogeneous and heterogeneous MTL. Experiments demonstrate the efficacy of our deep multi-task representation learning in terms of both higher accuracy and fewer design choices.
Our MTL approach is a , in that DNN weights are dynamically generated given some side information -- in the case of MTL, given the task identity. In a related example of speaker-adaptive speech recognition there may be several clusters in the data (e.g., gender, acoustic conditions), and each speaker's model could be a linear combination of these latent task clusters' models. They model each speaker @math 's weight matrix @math as a sum of @math base models @math , i.e., @math . The difference between speakers tasks comes from @math and the base models are shared. An advantage of this is that, when new data come, one can choose to re-train @math parameters only, and keep @math fixed. This will significantly reduce the number of parameters to learn, and consequently the required training data. Beyond this, @cite_0 show that it is possible to train another neural network to those @math values from some abstract metadata. Thus a model for an task can be generated on-the-fly with training instances given an abstract description of the task. The techniques developed here are compatible with both these ideas of generating models with minimal or no effort.
@cite_0
1963826206
The model for three-mode factor analysis is discussed in terms of newer applications of mathematical processes including a type of matrix process termed the Kronecker product and the definition of combination variables. Three methods of analysis to a type of extension of principal components analysis are discussed. Methods II and III are applicable to analysis of data collected for a large sample of individuals. An extension of the model is described in which allowance is made for unique variance for each combination variable when the data are collected for a large sample of individuals.