In the context of deep learning, this article presents an original deep network, namely CentralNet, for the fusion of information coming from different sensors. This approach is designed to efficiently and automatically balance the trade-off between early and late fusion (i.e. between the fusion of low-level vs high-level information). More specifically, at each level of abstraction-the different levels of deep networks-uni-modal representations of the data are fed to a central neural network which combines them into a common embedding. In addition, a multi-objective regularization is also introduced, helping to both optimize the central network and the unimodal networks. Experiments on four multimodal datasets not only show state-of-the-art performance, but also demonstrate that CentralNet can actually choose the best possible fusion strategy for a given problem.
The proposed method borrows from both visions: it is a multilayer framework and uses a constrained multi-objective loss. It builds on existing deep convolutional neural networks designed to process each modality independently. We suggest connecting these networks, at different levels, using an additional central network dedicated to the projection of the features coming from the different modalities into the same common space. In addition, the global loss allows to back propagate some global constraints on each modality, coordinating their representations. The proposed approach aims at automatically identifying an optimal fusion strategy for a given task. The approach is multi-objective in the sense that it simultaneously tries to minimize per modality losses as well as the global loss defined on the joint space. This article is an extension of @cite_18 .
@cite_18
2953039737
This paper proposes a novel multimodal fusion approach, aiming to produce best possible decisions by integrating information coming from multiple media. While most of the past multimodal approaches either work by projecting the features of different modalities into the same space, or by coordinating the representations of each modality through the use of constraints, our approach borrows from both visions. More specifically, assuming each modality can be processed by a separated deep convolutional network, allowing to take decisions independently from each modality, we introduce a central network linking the modality specific networks. This central network not only provides a common feature embedding but also regularizes the modality specific networks through the use of multi-task learning. The proposed approach is validated on 4 different computer vision tasks on which it consistently improves the accuracy of existing multimodal fusion approaches.