This paper presents a conceptually simple and effective Deep Audio-Visual Eembedding for dynamic saliency prediction dubbed DAVE". Several behavioral studies have shown a strong relation between auditory and visual cues for guiding gaze during scene free viewing. The existing video saliency models, however, only consider visual cues for predicting saliency over videos and neglect the auditory information that is ubiquitous in dynamic scenes. We propose a multimodal saliency model that utilizes audio and visual information for predicting saliency in videos. Our model consists of a two-stream encoder and a decoder. First, auditory and visual information are mapped into a feature space using 3D Convolutional Neural Networks (3D CNNs). Then, a decoder combines the features and maps them to a final saliency map. To train such model, data from various eye tracking datasets containing video and audio are pulled together. We further categorised videos into social', nature', and miscellaneous' classes to analyze the models over different content types. Several analyses show that our audio-visual model outperforms video-based models significantly over all scores; overall and over individual categories. Contextual analysis of the model performance over the location of sound source reveals that the audio-visual model behaves similar to humans in attending to the location of sound source. Our endeavour demonstrates that audio is an important signal that can boost video saliency prediction and help getting closer to human performance.
Auditory attention regards mechanisms of attention in the auditory system. A good example of such mechanisms is the famous cocktail party problem @cite_22 . The behavioral studies on the formation of auditory attention have been interested in the subject's response to an auditory stimulus, @cite_41 @cite_10 . From neurophysiological point of view, mechanisms of auditory attention and its influence on auditory cortical representations have been of interest, @cite_3 .
@cite_41 @cite_10 @cite_22 @cite_3
1979753320 1986001465 1991139021 2082183045
Abstract Groups of 32 children in each of three grades (kindergarten, second, and fourth) listened to a man's voice and a woman's voice, speaking words simultaneously. On one run through the 23 pairs of words, Ss were instructed to report what the man's voice was saying; on another, the words spoken by the woman. The stimulus words were systematically varied with respect to the number of syllables they contained. For cach S, the stimulus words were presented on one occasion binaurally (both words in both cars), on another occasion, dichotically (two different words in each ear). The number of correct reports of the word spoken by the desired voice increased with age; the number of intrusive errors (reports of words spoken by the unwanted voice) declined with age. At all age levels, scores were higher under dichotic presentation, but there was no change with age in the advantage derived from this presentation. Children at each age level made more errors on single-syllable than on multi-syllable words, but this difference was greater for older children. An analysis of errors revealed systematic differences among age groups in the nature of wrong guesses. Performanec improved through blocks of trials, at all age levels. Abstract In 2 experiments, the development of auditory selective attention in children was assessed. The participants, aged 5, 7, and 9 years, responded to target stimuli in the left ear for 1 series (ignoring the standard stimuli) and in the right ear for the other series. In Experiment 1, event-related potentials (ERPs) evoked by the auditory stimuli were recorded from frontal, central, and pariet al sites. The 9-year-olds showed a greater processing negativity (Nd) to the attended channel compared with the 5-year-olds. Both 7-and 9-year-olds showed significantly larger amplitudes for the P3 component of ERPs in the attended vs. ignored condition. Behaviorally, the 5-year-olds made fewer hits and more false alarms than did older children, and the 9-year-olds made significantly more false alarms than did the 7-year-olds. The results of Experiment 2 showed that the detriment in the performance of the 9-year-olds was a result of task parameters. The inability of 5-year-olds to attend selectively appears to ... This paper describes a number of objective experiments on recognition, concerning particularly the relation between the messages received by the two ears. Rather than use steady tones or clicks (frequency or time‐point signals) continuous speech is used, and the results interpreted in the main statistically. Two types of test are reported: (a) the behavior of a listener when presented with two speech signals simultaneously (statistical filtering problem) and (b) behavior when different speech signals are presented to his two ears. The neural correlates of how attended speech is internally represented are described, shedding light on the ‘cocktail party problem’.