Deep learning yields great results across many fields,
 from speech recognition, image classification, to translation.
 But for each problem, getting a deep model to work well involves
 research into the architecture and a long period of tuning.

 We present a single model that yields good results on a number
 of problems spanning multiple domains. In particular, this single model
 is trained concurrently on ImageNet, multiple translation tasks,
 image captioning (COCO dataset), a speech recognition corpus,
 and an English parsing task. 

 Our model architecture incorporates building blocks from multiple
 domains. It contains convolutional layers, an attention mechanism,
 and sparsely-gated layers.

 Each of these computational blocks is crucial for a subset of
 the tasks we train on. Interestingly, even if a block is not
 crucial for a task, we observe that adding it never hurts performance
 and in most cases improves it on all tasks.

 We also show that tasks with less data benefit largely from joint
 training with other tasks, while performance on large tasks degrades
 only slightly if at all. Recent successes of deep neural networks have spanned many domains, from computer vision BID16 to speech recognition BID7 and many other tasks. Convolutional networks excel at tasks related to vision, while recurrent neural networks have proven successful at natural language processing tasks, e.g., at machine translation BID31 BID2 . But in each case, the network was designed and tuned specifically for the problem at hand. This limits the impact of deep learning, as this effort needs to be repeated for each new task. It is also very different from the general nature of the human brain, which is able to learn many different tasks and benefit from transfer learning. The natural question arises:Can we create a unified deep learning model to solve tasks across multiple domains?The question about multi-task models has been studied in many papers in the deep learning literature. Natural language processing models have been shown to benefit from a multi-task approach a long time ago BID5 , and recently multi-task machine translation models (MinhThang Luong, 2015) have even been shown to exhibit zero-shot learning when trained on multiple languages (Melvin Johnson, 2016) . Speech recognition has also been shown to benefit from multi-task training BID27 , as have some vision problems, such as facial landmark detection BID36 . But all these models are trained on other tasks from the same domain: translation tasks are trained with other translation tasks, vision tasks with other vision tasks, speech tasks with other speech tasks. Multi-modal learning has been shown to improve learned representations in the unsupervised setting BID22 and when used as a-priori known unrelated tasks BID24 . But no competitive multi-task multi-modal model has been proposed, so the above question remains unanswered.In this work, we take a step toward positively answering the above question by introducing the MultiModel architecture, a single deep-learning model that can simultaneously learn multiple tasks from various domains. Concretely, we train the MultiModel simultaneously on the following 8 corpora:Code available at redacted. (1) WSJ speech corpus (Consortium et al., 1994 ), used for sentence-level speech recognition.(2) ImageNet dataset BID25 , used for image classification. (3) COCO image captioning dataset BID17 , used for image captioning. (4) WSJ parsing dataset BID18 , used for constituency parsing. These corpora were chosen as they are commonly used for machine learning the respective tasks: speech-to-text, image classification, captioning, parsing and translation. The model learns all of these tasks and achieves good performance: not state-of-the-art at present, but above many task-specific models studied in recent past (see the Section 3 for details). FIG0 illustrates some decodes taken directly from the model: it is clear that it can caption images, categorize them, translate to French and German and construct parse trees. While the MultiModel is only a first step and will be improved in the future, two key insights are crucial to making it work at all and are our main contributions.Small modality-specific sub-networks convert into a unified representation and back from it. To allow training on input data of widely different sizes and dimensions, such as images, sound waves and text, we need sub-networks to convert inputs into a joint representation space. We call these sub-networks modality nets as they are specific to each modality (images, speech, text) and define transformations between these external domains and a unified representation. We design modality nets to be computationally minimal, promoting heavy feature extraction and ensuring that the majority of computation is performed within the domain-agnostic body of the model. Since our model is auto-regressive, modality nets need to both convert the inputs into the unified representation and later convert from this representation into the output space. Two design decisions were important:• The unified representation is variable-size. While a fixed-size representation is tempting and easier to implement, it creates a bottleneck and limits the performance of the model. • Different tasks from the same domain share modality nets. We avoid creating a sub-network for every task, and prefer only to create one for every input modality. For example, all translation tasks share the same modality-net (and vocabulary), no matter for which language pair. This encourages generalization across tasks and allows to add new tasks on the fly.Computational blocks of different kinds are crucial for good results on various problems. The body of the MultiModel incorporates building blocks from mutiple domains. We use depthwiseseparable convolutions, an attention mechanism , and sparsely-gated mixture-of-experts layers. These blocks were introduced in papers that belonged to different domains and were not studied before on tasks from other domains. For example, separable convolutions were introduced in the Xception Figure 2 : The MultiModel, with modality-nets, an encoder, and an autoregressive decoder.architecture BID4 and were not applied to text or speech processing before. On the other hand, the sparsely-gated mixture-of-experts BID29 had been introduced for language processing tasks and has not been studied on image problems. We find that each of these mechanisms is indeed crucial for the domain it was introduced, e.g., attention is far more important for languagerelated tasks than for image-related ones. But, interestingly, adding these computational blocks never hurts performance , even on tasks they were not designed for. In fact we find that both attention and mixture-of-experts layers slightly improve performance of MultiModel on ImageNet, the task that needs them the least.
Large scale multi-task architecture solves ImageNet and translation together and shows transfer learning.
{"src": "scitldr-AIC_train", "paper_id": "HyKZyYlRZ"}