Every culture and language is unique. Our work expressly focuses on the uniqueness of culture and language in relation to human affect, specifically sentiment and emotion semantics, and how they manifest in social multimedia. We develop sets of sentiment- and emotion-polarized visual concepts by adapting semantic structures called adjective-noun pairs, originally introduced by (2013), but in a multilingual context. We propose a new language-dependent method for automatic discovery of these adjective-noun constructs. We show how this pipeline can be applied on a social multimedia platform for the creation of a large-scale multilingual visual sentiment concept ontology (MVSO). Unlike the flat structure in (2013), our unified ontology is organized hierarchically by multilingual clusters of visually detectable nouns and subclusters of emotionally biased versions of these nouns. In addition, we present an image-based prediction task to show how generalizable language-specific models are in a multilingual context. A new, publicly available dataset of >15.6K sentiment-biased visual concepts across 12 languages with language-specific detector banks, >7.36M images and their metadata is also released.
We address the general challenge of , aiming at both recognition and analysis of sentiment and emotions in visual data, but from a multilingual and cross-cultural perspective. Our work is closely related to Multimedia and Vision research that focus on visual aesthetics @cite_6 , interestingness @cite_15 , popularity @cite_19 , and creativity @cite_4 . Our work also relates to research in Cognitive and Social Psychology, especially emotion and culture research @cite_40 @cite_38 @cite_8 , but also neuroaesthetics @cite_39 , visual preference @cite_39 @cite_5 , and social interaction @cite_2 .
@cite_38 @cite_4 @cite_8 @cite_6 @cite_39 @cite_19 @cite_40 @cite_2 @cite_5 @cite_15
2041003138 2604737966 2094637479 2347880541
The increasing adoption of social media provides unprecedented opportunities to gain insight into human nature at vastly broader scales. Regarding the study of population-wide sentiment, prior research commonly focuses on text-based analyses and ignores a treasure trove of sentiment-laden content: images. In this paper, we make methodological and computational contributions by introducing the Smile Index as a formalized measure of societ al happiness. Detecting smiles in 9 million geo-located tweets over 16 months, we validate our Smile Index against both text-based techniques and self-reported happiness. We further make observational contributions by applying our metric to explore temporal trends in sentiment, relate public mood to societ al events, and predict economic indicators. Reflecting upon the innate, language-independent aspects of facial expressions, we recommend future improvements and applications to enable robust, global-level analyses. We conclude with implications for researchers studying and facilitating the expression of collective emotion through socio-technical systems. Visual sentiment analysis, which studies the emotional response of humans on visual stimuli such as images and videos, has been an interesting and challenging problem. It tries to understand the high-level content of visual data. The success of current models can be attributed to the development of robust algorithms from computer vision. Most of the existing models try to solve the problem by proposing either robust features or more complex models. In particular, visual features from the whole image or video are the main proposed inputs. Little attention has been paid to local areas, which we believe is pretty relevant to human's emotional response to the whole image. In this work, we study the impact of local image regions on visual sentiment analysis. Our proposed model utilizes the recent studied attention mechanism to jointly discover the relevant local regions and build a sentiment classifier on top of these local regions. The experimental results suggest that 1) our model is capable of automatically discovering sentimental local regions of given images and 2) it outperforms existing state-of-the-art algorithms to visual sentiment analysis. The recent advancement of social media has given users a platform to socially engage and interact with a larger population. Millions of images and videos are being uploaded everyday by users on the web from different events and social gatherings. There is an increasing interest in designing systems capable of understanding human manifestations of emotional attributes and affective displays. As images and videos from social events generally contain multiple subjects, it is an essential step to study these groups of people. In this paper, we study the problem of happiness intensity analysis of a group of people in an image using facial expression analysis. A user perception study is conducted to understand various attributes, which affect a person’s perception of the happiness intensity of a group. We identify the challenges in developing an automatic mood analysis system and propose three models based on the attributes in the study. An ‘in the wild’ image-based database is collected. To validate the methods, both quantitative and qualitative experiments are performed and applied to the problem of shot selection, event summarisation and album creation. The experiments show that the global and local attributes defined in the paper provide useful information for theme expression analysis, with results close to human perception results. Psychological research results have confirmed that people can have different emotional reactions to different visual stimuli. Several papers have been published on the problem of visual emotion analysis. In particular, attempts have been made to analyze and predict people's emotional reaction towards images. To this end, different kinds of hand-tuned features are proposed. The results reported on several carefully selected and labeled small image data sets have confirmed the promise of such features. While the recent successes of many computer vision related tasks are due to the adoption of Convolutional Neural Networks (CNNs), visual emotion analysis has not achieved the same level of success. This may be primarily due to the unavailability of confidently labeled and relatively large image data sets for visual emotion analysis. In this work, we introduce a new data set, which started from 3+ million weakly labeled images of different emotions and ended up 30 times as large as the current largest publicly available visual emotion data set. We hope that this data set encourages further research on visual emotion analysis. We also perform extensive benchmarking analyses on this large data set using the state of the art methods including CNNs.