Convolutional neural networks (CNN) have become the most successful and popular approach in many vision-related domains. While CNNs are particularly well-suited for capturing a proper hierarchy of concepts from real-world images, they are limited to domains where data is abundant. Recent attempts have looked into mitigating this data scarcity problem by casting their original single-task problem into a new multi-task learning (MTL) problem. The main goal of this inductive transfer mechanism is to leverage domain-specific information from related tasks, in order to improve generalization on the main task. While recent results in the deep learning (DL) community have shown the promising potential of training task-specific CNNs in a soft parameter sharing framework, integrating the recent DL advances for improving knowledge sharing is still an open problem. In this paper, we propose the Deep Collaboration Network (DCNet), a novel approach for connecting task-specific CNNs in a MTL framework. We define connectivity in terms of two distinct non-linear transformation blocks. One aggregates task-specific features into global features, while the other merges back the global features with each task-specific network. Based on the observation that task relevance depends on depth, our transformation blocks use skip connections as suggested by residual network approaches, to more easily deactivate unrelated task-dependent features. To validate our approach, we employed facial landmark detection (FLD) datasets as they are readily amenable to MTL, given the number of tasks they include. Experimental results show that we can achieve up to 24.31% relative improvement in landmark failure rate over other state-of-the-art MTL approaches. We finally perform an ablation study showing that our approach effectively allows knowledge sharing, by leveraging domain-specific features at particular depths from tasks that we know are related. Over the past few years, convolutional neural networks (CNNs) have become the leading approach in many vision-related tasks BID12 . By creating a hierarchy of increasingly abstract concepts, they can transform complex high-dimensional input images into simple low-dimensional output features. Although CNNs are particularly well-suited for capturing a proper hierarchy of concepts from real-world images, successively training them requires large amount of data. Optimizing deep networks is tricky, not only because of problems like vanishing / exploding gradients BID8 or internal covariate shift BID9 , but also because they typically have many parameters to be learned (which can go up to 137 billions BID21 ). While previous works have looked at networks pre-trained on a large image-based dataset as a starting point for their gradient descent optimization, others have considered improving generalization by casting their original single-task problem into a new multi-task learning (MTL) problem (see BID31 for a review). As BID2 explained in his seminal work: "MTL improves generalization by leveraging the domain-specific information contained in the training signals of related tasks". Exploring new ways to efficiently gather more information from related tasks -the core contribution of our approach -can thus help a network to further improve upon its main task.The use of MTL goes back several years, but has recently proven its value in several domains. As a consequence, it has become a dominant field of machine learning BID30 . Although many early and influential works contributed to this field BID5 ), recent major advances in neural networks opened up opportunities for novel contributions in MTL. Works on grasping BID17 , pedestrian detection BID24 , natural language processing BID14 , face recognition BID26 BID27 and object detection BID16 have all shown that MTL has been finally adopted by the deep learning (DL) community as a way to mitigate the lack of data, and is thus growing in popularity.MTL strategies can be divided into two major categories: hard and soft parameter sharing. Hard parameter sharing is the earliest and most common strategy for performing MTL, which dates back to the original work of BID2 . Approaches in this category generally share the hidden layers between all tasks, while keeping separate outputs. Recent results in the DL community have shown that a central CNN with separate task-specific fully connected (FC) layers can successfully leverage domain-specific information BID18 BID17 BID27 . Although hard parameter sharing reduces the risk of over-fitting BID1 , shared layers are prone to be overwhelmed by features or contaminated by noise coming from particular noxious related tasks .Soft parameter sharing has been proposed as an alternative to alleviate this drawback, and has been growing in popularity as a potential successor. Approaches in this category separate all hidden layers into task-specific models, while providing a knowledge sharing mechanism. Each model can then learn task-specific features without interfering with others, while still sharing their knowledge. Recent works using one network per task have looked at regularizing the distance between taskspecific parameters with a 2 norm BID4 or a trace norm BID25 , training shared and private LSTM submodules , partitioning the hidden layers into subspaces BID19 and regularizing the FC layers with tensor normal priors BID15 . In the domain of continual learning, progressive network BID20 has also shown promising results for cross-domain sequential transfer learning, by employing lateral connections to previously learned networks. Although all these soft parameter approaches have shown promising potential, improving the knowledge sharing mechanism is still an open problem.In this paper, we thus present the deep collaboration network (DCNet), a novel approach for connecting task-specific networks in a soft parameter sharing MTL framework. We contribute with a novel knowledge sharing mechanism, dubbed the collaborative block, which implements connectivity in terms of two distinct non-linear transformations. One aggregates task-specific features into global features, and the other merges back the global features into each task-specific network. We demonstrate that our collaborative block can be dropped in any existing architectures as a whole, and can easily enable MTL for any approaches. We evaluated our method on the problem of facial landmark detection in a MTL framework and obtained better results in comparison to other approaches of the literature. We further assess the objectivity of our training framework by randomly varying the contribution of each related tasks, and finally give insights on how our collaborative block enables knowledge sharing with an ablation study on our DCNet.The content of our paper is organized as follows. We first describe in Section 2 works on MTL closely related to our approach. We also describe Facial landmark detection, our targeted application. Architectural details of our proposed Multi-Task approach and its motivation are spelled out in Section 3. We then present in Section 4 a number of comparative results on this Facial landmark detection problem for two CNN architectures, AlexNet and ResNet18, that have been adapted with various MTL frameworks including ours. It also contains discussions on an ablation study showing at which depth feature maps from other tasks are borrowed to improve the main task. We conclude our paper in Section 5.2 RELATED WORK 2.1 MULTI-TASK LEARNING Our proposed deep collaboration network (DCNet) is related to other existing approaches. The first one is the cross-stitch (CS) BID16 ) network, which connects task-specific networks through linear combinations of the spatial feature maps at specific layers. One drawback of CS is that they are limited to capturing linear dependencies only, something we address in our proposed approach by employing non-linearities when sharing feature maps. Indeed, non-linear combinations are usually able to learn richer relationships, as demonstrated in deep networks. Another related approach is tasks-constrained deep convolutional network (TCDCN) for facial landmarks detection . In it, the authors proposed an early-stopping criterion for removing auxiliary tasks before the network starts to over-fit to the detriment of the main task. One drawback of their approach is that their criterion has several hyper-parameters, which must all be selected manually. For instance, they define an hyper-parameter controlling the period length of the local window and a threshold that stops the task when the criterion exceeds it, all of which can be specified for each task independently. Unlike TCDCN, our approach has no hyper-parameters that depend on the tasks at hand, which greatly simplifies the training process. Our two transformation blocks consist of a series of batch normalization, ReLU, and convolutional layers shaped in a standard setting based on recent advances in residual network (see Sec. 3). This is particularly useful for computationally expensive deep networks, since integrating our proposed approach requires no additional hyper-parameter tuning experiments.Our proposed approach is also related to HyperFace BID18 . In this work, the authors proposed to fuse the intermediate layers of AlexNet and exploit the hierarchical nature of the features. Their goal was to allow low-level features containing better localization properties to help tasks such as landmark localization and pose detection, and allow more class-specific high-level features to help tasks like face detection and gender recognition. Although HyperFace uses a single shared CNN instead of task-specific CNNs and is not entirely related to our approach, the idea of feature fusion is also central in our work. Instead of fusing the features at intermediate layers of a single CNN , our approach aggregates same-level features of multiple CNNs, at different depth independently. Also, one drawback of HyperFace is that the proposed feature fusion is specific to AlexNet, while our method is not specific to any network. In fact, our approach takes into account the vast diversity of existing network architectures, since it can be added to any architecture without modification. In this paper, we proposed the deep collaboration network (DCNet), a novel approach for connecting task-specific networks in a multi-task learning setting. It implements feature connectivity and sharing through two distinct non-linear transformations inside a collaborative block, which also incorporates skip connection and residual mapping that are known for their good training behavior. The first transformation aggregates the task-specific feature maps into a global feature map representing unified knowledge, and the second one merges it back into each task-specific network. One key characteristic of our collaborative blocks is that they can be dropped in virtually any existing architectures, making them universal adapters to endow deep networks with multi-task learning capabilities.Our results on the MTFL, AFW and AFLW datasets showed that our DCNet outperformed several state-of-the-art approaches, including cross-stitch networks. Our additional ablation study, using ResNet18 as underlying network, confirmed our intuition that the task-specific networks exploited the added flexibility provided by our approach. Additionally, these task-specific networks successfully incorporated features having varying levels of abstraction. Evaluating our proposed approach on other MTL problems could be an interesting avenue for future works. For instance, the recurrent networks used to solve natural language processing problems could benefit from incorporating our novel method leveraging domain-information of related tasks.
We propose a novel approach for connecting task-specific networks in a multi-task learning setting based on recent residual network advances.
{"src": "scitldr-AIC_train", "paper_id": "r17Q6WWA-"}