This paper presents a conceptually simple and effective Deep Audio-Visual Eembedding for dynamic saliency prediction dubbed DAVE". Several behavioral studies have shown a strong relation between auditory and visual cues for guiding gaze during scene free viewing. The existing video saliency models, however, only consider visual cues for predicting saliency over videos and neglect the auditory information that is ubiquitous in dynamic scenes. We propose a multimodal saliency model that utilizes audio and visual information for predicting saliency in videos. Our model consists of a two-stream encoder and a decoder. First, auditory and visual information are mapped into a feature space using 3D Convolutional Neural Networks (3D CNNs). Then, a decoder combines the features and maps them to a final saliency map. To train such model, data from various eye tracking datasets containing video and audio are pulled together. We further categorised videos into social', nature', and miscellaneous' classes to analyze the models over different content types. Several analyses show that our audio-visual model outperforms video-based models significantly over all scores; overall and over individual categories. Contextual analysis of the model performance over the location of sound source reveals that the audio-visual model behaves similar to humans in attending to the location of sound source. Our endeavour demonstrates that audio is an important signal that can boost video saliency prediction and help getting closer to human performance.
With respect to multimodal attention, some behavioral studies have investigated the role of audio-visual components during sensory development in infants @cite_49 . Richards @cite_38 studied attention engagement to compound auditory-visual stimuli and showed an increase in sustained attention development with age progression in infants. Burg al @cite_37 conducted experiments to understand the effect of audiovisual events on guiding attention and concluded that audiovisual synchrony guides attention in an exogenous manner in adults.
@cite_38 @cite_37 @cite_49
2128779052 1988712925 11812203
This study examined the effect of attention engagement to compound auditory-visual stimuli on the modification of the startle blink reflex in infants. Infants at 8, 14, 20, or 26 weeks of age were presented with interesting audiovisual stimuli. After stimulus onset, at delays defined by heart rate changes known to be associated with sustained attention or attention disengagement, blink reflexes were elicited by visual or auditory stimuli. Blink amplitude to either visual or auditory stimuli was enhanced when the infants were engaged in attention to the foreground auditory-visual stimuli relative to control trials with no foreground patterns. This enhancement of the blink amplitude increased from 8 to 26 weeks of age. In contrast to selective modality enhancement for single-modality foreground stimuli, these results show that these multimodal stimuli engage both visual and auditory attention systems in this age range. Descriptors: Infants, Attention, Heart rate, Blink reflex, Multimodal stimuli De volgorde waarin we 2 vrijwel tegelijkertijd weergegeven stipjes zien hangt af van de plek waarop onze aandacht gericht is: het stipje dat het dichtst bij de focus van de aandacht ligt wordt iets eerder waargenomen. In deze experimenten wordt dit effect gebruikt om aan te tonen dat een irrelevante stimulus onwillekeurig de aandacht trekt wanneer een geluid wordt weergegeven op het moment dat die stimulus van kleur verandert. Het bijzondere hierbij is dat het niet uitmaakt waar het geluid vandaan komt. The present invention is used to make an excellent recording of color pictures and writings by a simple unit and easy operation wherein each set of ball-pen system recording part is provided for each restored primary color signal at receiving part for a color recording apparatus which is composed of transmitting and receiving parts, and a ball-pen is held pressed on a recording paper with some pressure corresponding to the strength of said primary color signal related. The present invention is able to apply to a facsimile.