Emotion evoked by an advertisement plays a key role in influencing brand recall and eventual consumer choices. Automatic ad affect recognition has several useful applications. However, the use of content-based feature representations does not give insights into how affect is modulated by aspects such as the ad scene setting, salient object attributes and their interactions. Neither do such approaches inform us on how humans prioritize visual information for ad understanding. Our work addresses these lacunae by decomposing video content into detected objects, coarse scene structure, object statistics and actively attended objects identified via eye-gaze. We measure the importance of each of these information channels by systematically incorporating related information into ad affect prediction models. Contrary to the popular notion that ad affect hinges on the narrative and the clever use of linguistic and social cues, we find that actively attended objects and the coarse scene structure better encode affective information as compared to individual scene objects or conspicuous background elements.
Human physiological signals have been widely employed for implicit tagging of multimedia content in recent years. Implicit tagging has been defined as using non-verbal spontaneous behavioral responses to find relevant tags for multimedia content @cite_45 . These responses include Electroencephalography (EEG) and Magnetoencephalography (MEG) along with physiological responses such as heart rate and body temperature @cite_28 @cite_4 @cite_40 @cite_3 @cite_20 and gazing behavior @cite_23 @cite_33 @cite_32 @cite_18 . Determining the affective dimensions of valence and arousal using these methods has been shown to be very effective. Affective attributes have also been shown to be useful for retrieval @cite_14 . These methods have an advantage of not requiring additional human effort in annotation, and are based on the analysis of spontaneous user reactions or behavioral signals. We employ eye gaze as a modality of interest in our experiments.
@cite_18 @cite_14 @cite_4 @cite_33 @cite_28 @cite_32 @cite_3 @cite_40 @cite_45 @cite_23 @cite_20
2148608964 2041347087 2164699598 1987074976
In this work, we aim to find neuro-physiological indicators to validate tags attached to video content. Subjects are shown a video and a tag and we aim to determine whether the shown tag was congruent with the presented video by detecting the occurrence of an N400 event-related potential. Tag validation could be used in conjunction with a vision-based recognition system as a feedback mechanism to improve the classification accuracy for multimedia indexing and retrieval. An advantage of using the EEG modality for tag validation is that it is a way of performing implicit tagging. This means it can be performed while the user is passively watching the video. Independent Component Analysis and repeated measures ANOVA are used for analysis. Our experimental results show a clear occurrence of the N400 and a significant difference in N400 activation between matching and non-matching tags. In this paper, we propose an approach for affective ranking of movie scenes based on the emotions that are actually felt by spectators. Such a ranking can be used for characterizing the affective, or emotional, content of video clips. The ranking can for instance help determine which video clip from a database elicits, for a given user, the most joy. This in turn will permit video indexing and retrieval based on affective criteria corresponding to a personalized user affective profile. A dataset of 64 different scenes from 8 movies was shown to eight participants. While watching, their physiological responses were recorded; namely, five peripheral physiological signals (GSR - galvanic skin resistance, EMG - electromyograms, blood pressure, respiration pattern, skin temperature) were acquired. After watching each scene, the participants were asked to self-assess their felt arousal and valence for that scene. In addition, movie scenes were analyzed in order to characterize each with various audio- and video-based features capturing the key elements of the events occurring within that scene. Arousal and valence levels were estimated by a linear combination of features from physiological signals, as well as by a linear combination of content-based audio and video features. We show that a correlation exists between arousal- and valence-based rankings provided by the spectator's self-assessments, and rankings obtained automatically from either physiological signals or audio-video features. This demonstrates the ability of using physiological responses of participants to characterize video scenes and to rank them according to their emotional content. This further shows that audio-visual features, either individually or combined, can fairly reliably be used to predict the spectator's felt emotion for a given scene. The results also confirm that participants exhibit different affective responses to movie scenes, which emphasizes the need for the emotional profiles to be user-dependant. This paper presents a user-independent emotion recognition method with the goal of recovering affective tags for videos using electroencephalogram (EEG), pupillary response and gaze distance. We first selected 20 video clips with extrinsic emotional content from movies and online resources. Then, EEG responses and eye gaze data were recorded from 24 participants while watching emotional video clips. Ground truth was defined based on the median arousal and valence scores given to clips in a preliminary study using an online questionnaire. Based on the participants' responses, three classes for each dimension were defined. The arousal classes were calm, medium aroused, and activated and the valence classes were unpleasant, neutral, and pleasant. One of the three affective labels of either valence or arousal was determined by classification of bodily responses. A one-participant-out cross validation was employed to investigate the classification performance in a user-independent approach. The best classification accuracies of 68.5 percent for three labels of valence and 76.4 percent for three labels of arousal were obtained using a modality fusion strategy and a support vector machine. The results over a population of 24 participants demonstrate that user-independent emotion recognition can outperform individual self-reports for arousal assessments and do not underperform for valence assessments. Tagging is the annotation of multimedia data with userspecified keywords known as tags, with the aim of facilitating fast and accurate data retrieval based on these tags. In contrast to this process, also referred to as explicit tagging, implicit human-centered tagging (IHCT) refers to exploiting the information on user's nonverbal reactions (e.g., facial expressions like smiles or head gestures like shakes) to multimedia data, with which he or she interacts, to assign new or improve the existing tags associated with the target data. Thus, implicit tagging allows that a data item gets tagged each time a user interacts with it based on the reactions of the user to the data (e.g., laughter when seeing a funny video), in contrast to explicit tagging paradigm in which a data item gets tagged only if a user is requested (or chooses) to associate tags with it. As nonverbal reactions to observed multimedia are displayed naturally and spontaneously, no purposeful explicit action (effort) is required from the user; hence, the resulting tagging process is said to be "implicit" and "human centered" (in contrast to being dictated by computer and being "computer-centered").