Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Multisensory Systems <s> We present a new approach to modeling and processing multimedia data. This approach is based on graphical models that combine audio and video variables. We demonstrate it by developing a new algorithm for tracking a moving object in a cluttered, noisy scene using two microphones and a camera. Our model uses unobserved variables to describe the data in terms of the process that generates them. It is therefore able to capture and exploit the statistical structure of the audio and video data separately, as well as their mutual dependencies. Model parameters are learned from data via an EM algorithm, and automatic calibration is performed as part of this procedure. Tracking is done by Bayesian inference of the object location from data. We demonstrate successful performance on multimedia clips captured in real world scenarios using off-the-shelf equipment. <s> BIB001 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Multisensory Systems <s> Microphones and cameras have been extensively used to observe and detect human activity and to facilitate natural modes of interaction between humans and intelligent systems. Human brain processes the audio and video modal- ities, extracting complementary and robust information from them. Intelligent systems with audiovisual sensors should be capable of achieving similar goals. The audiovisual information fusion strategy is a key component in designing such systems. In this paper, we exclusively survey the fusion techniques used in various audiovisual information fusion tasks. The fusion strategy used tends to depend mainly on the model, probabi- listic or otherwise, used in the particular task to process sensory information to obtain higher level semantic informa- tion. The models themselves are task oriented. In this paper, we describe the fusion strategies and the corresponding models used in audiovisual tasks such as speech recognition, tracking, biometrics, affective state recognition, and meeting scene analysis. We also review the challenges and existing solutions and also unresolved or partially resolved issues in these fields. Specifically, we discuss established and upcoming work in hierarchical fusion strategies and cross-modal learning tech- niques, identifying these as critical areas of research in the future development of intelligent systems. <s> BIB002 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Multisensory Systems <s> The separation of speech signals measured at multiple microphones in noisy and reverberant environments using only the audio modality has limitations because there is generally insufficient information to fully discriminate the different sound sources. Humans mitigate this problem by exploiting the visual modality, which is insensitive to background noise and can provide contextual information about the audio scene. This advantage has inspired the creation of the new field of audiovisual (AV) speech source separation that targets exploiting visual modality alongside the microphone measurements in a machine. Success in this emerging field will expand the application of voice-based machine interfaces, such as Siri, the intelligent personal assistant on the iPhone and iPad, to much more realistic settings and thereby provide more natural human?machine interfaces. <s> BIB003 </s> Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects <s> A. Multisensory Systems <s> Smart rooms are used for a growing number of practical applications. They are often equipped with microphones and cameras allowing acoustic and visual tracking of persons. For that, the geometry of the sensors has to be calibrated. In this paper, a method is introduced that calibrates the microphone arrays by using the visual localization of a speaker at a small number of fixed positions. By matching the positions to the direction of arrival (DoA) estimates of the microphone arrays, their absolute position and orientation are derived. Data from a reverberant smart room is used to show that the proposed method can estimate the absolute geometry with about 0.1 m and 2 precision. The calibration is good enough for acoustic and multi modal tracking applications and eliminates the need for dedicated calibration measures by using the tracking data itself. <s> BIB004
Example II-A.1: Audiovisual Multimodality: Audiovisual multimodality is probably the most intuitive, since it uses two of our most informative senses. Most human verbal communication involves seeing the speaker  . Indeed, a large number of audiovisual applications involve human speech and vision. In such applications, it is usually the audio channel that conveys the information of interest. It is well known that audio and video convey complementary information. Audio has the advantage over video that it does not require line of sight. On the other hand, the visual modality is resistant to various factors that make audio and speech processing difficult, such as ambient noise, reverberations, and other acoustic disturbances. Perhaps the most striking evidence to the amount of caution that needs to be taken in the design and use of multimodal systems is the ''McGurk effect''  . In their seminal paper, McGurk and McDonald  Today, audiovisual multimodality is used for a broad range of applications BIB002 , BIB003 . Examples include: speech processing, including speech recognition, speech activity detection, speech enhancement, speaker extraction, and separation; scene analysis, including tracking a speaker within a group, biometrics and monitoring, for safety and security applications BIB001 ; human-machine interaction (HMI) BIB002 ; calibration [10, Sec. V.C], BIB004 ; and more. Example II-A.2: Human-Machine Interaction: A domain that is heavily inspired by natural multimodality is HMI. In HMI, an important task is to design modalities that will make HMI as natural, efficient, and intuitive as possible  . The idea is to combine multiple interaction modes based on audiovision, touch, smell, movement (e.g., gesture detection and user tracking), interpretation of human language commands, and other multisensory functions BIB002 ,  . The principal point that makes HMI stand out among other multimodal applications that we mention is that, in HMI, the modalities are often interactive (as their name implies). Unlike other multimodal applications that we mention, not one but two very different types of systems (human and machine) are ''observed'' by each other's sensors, and the goal of data fusion is not only to interpret each system's output, but also to actively convey information between these two systems. An added challenge is that this task should usually be accomplished in real time. An additional complicating factor that makes multimodal HMI stand out is due to the fact that the human user often plays an active part in the choice of modalities (from the available set) and in the way that they are used in practice. This implies that the design of the multimodal setup and data fusion procedure must rely not only on the theoretically and technologically optimal combination of data streams but also on the ability to predict and adapt to the subjective cognitive preferences of the individual user. We refer to  (and references therein) for further discussion of these aspects.