Abstract Classification using multimodal data arises in many machine learning applications. It is crucial not only to model cross-modal relationship effectively but also to ensure robustness against loss of part of data or modalities. In this paper, we propose a novel deep learning-based multimodal fusion architecture for classification tasks, which guarantees compatibility with any kind of learning models, deals with cross-modal information carefully, and prevents performance degradation due to partial absence of data. We employ two datasets for multimodal classification tasks, build models based on our architecture and other state-of-the-art models, and analyze their performance on various situations. The results show that our architecture outperforms the other multimodal fusion architectures when some parts of data are not available.
Some researchers developed more sophisticated multimodal integration approaches that regulate the degree of contribution of each modality for classification tasks. Bharadwaj @cite_38 proposed a context switching-based person identification system, which prioritizes the multimodal data by measuring their quality and chooses an appropriate classifier. Goswami @cite_17 employed the pool adjacent violators algorithm for combining multiple classifiers, which calibrates the outputs of the classifiers with respect to their confidence values. Choi and Lee @cite_11 developed an activity recognition model, which controls the amount of information for each modality by measuring its reliability. These imply that considering significance of each modality data is beneficial to improve the classification performance.
@cite_38 @cite_11 @cite_17
308789809 2899475803 2607930008
Abstract Biometrics, the science of verifying the identity of individuals, is increasingly being used in several applications such as assisting law enforcement agencies to control crime and fraud. Existing techniques are unable to provide significant levels of accuracy in uncontrolled noisy environments. Further, scalability is another challenge due to variations in data distribution with changing conditions. This paper presents an adaptive context switching algorithm coupled with online learning to address both these challenges. The proposed framework, termed as QFuse , uses the quality of input images to dynamically select the best biometric matcher or fusion algorithm to verify the identity of an individual. The proposed algorithm continuously updates the selection process using online learning to address the scalability and accommodate the variations in data distribution. The results on the WVU multimodal database and a large real world multimodal database obtained from a law enforcement agency show the efficacy of the proposed framework. Human activity recognition using multimodal sensors is widely studied in recent days. In this paper, we propose an end-to-end deep learning model for activity recognition, which fuses features of multiple modalities based on their confidence scores that are automatically determined. The confidence scores efficiently regulate the level of contribution of each sensor. We conduct an experiment on the latest activity recognition dataset. The results confirm that our model outperforms existing methods. We submit the proposed model to the Sussex-Huawei Locomotion-Transportation (SHL) recognition challenge [23] with the team name "Yonsei-MCML." Classifier fusion is a well-studied problem in which decisions from multiple classifiers are combined at the score, rank, or decision level to obtain better results than a single classifier. Subsequently, various techniques for combining classifiers at each of these levels have been proposed in the literature. Many popular methods entail scaling and normalizing the scores obtained by each classifier to a common numerical range before combining the normalized scores using the sum rule or another classifier. In this research, we explore an alternative method to combine classifiers at the score level. The Pool Adjacent Violators (PAV) algorithm has traditionally been utilized to convert classifier match scores to confidence values that model posterior probabilities for test data. The PAV algorithm and other score normalization techniques have studied the same problem without being aware of each other. In this first ever study to combine the two, we propose the PAV algorithm for classifier fusion on publicly available NIST multi-modal biometrics score dataset. We observe that it provides several advantages over existing techniques and find that the interpretation learned by the PAV algorithm is more robust than the scaling learned by other popular normalization algorithms such as min-max. Moreover, the PAV algorithm enables the combined score to be interpreted as confidence and is able to further improve the results obtained by other approaches. We also observe that utilizing traditional normalization techniques first for individual classifiers and then normalizing the fused score using PAV offers a performance boost compared to only using the PAV algorithm.