Emotion estimation in crowds: a survey <s> Emotion Estimation <s> In this paper, we propose a new image representation called Local Zernike Moments (LZM) for face recognition. In recent years, local image representations such as Gabor and Local Binary Patterns (LBP) have attracted great interest due to their success in handling difficulties of face recognition. In this study, we aim to develop an alternative representation to further improve the face recognition performance. We achieve this by utilizing Zernike Moments which have been successfully used as shape descriptors for character recognition. We modify global Zernike moments to obtain a local representation by computing the moments at every pixel of a face image by considering its local neighborhood, thus decomposing the image into a set of images, moment components, to capture the micro structure around each pixel. Our experiments on FERET face database reveal the superior performance of LZM over Gabor and LBP representations. <s> BIB001 </s> Emotion estimation in crowds: a survey <s> Emotion Estimation <s> A standard approach to describe an image for classification and retrieval purposes is to extract a set of local patch descriptors, encode them into a high dimensional vector and pool them into an image-level signature. The most common patch encoding strategy consists in quantizing the local descriptors into a finite set of prototypical elements. This leads to the popular Bag-of-Visual words representation. In this work, we propose to use the Fisher Kernel framework as an alternative patch encoding strategy: we describe patches by their deviation from an “universal” generative Gaussian mixture model. This representation, which we call Fisher vector has many advantages: it is efficient to compute, it leads to excellent results even with efficient linear classifiers, and it can be compressed with a minimal loss of accuracy using product quantization. We report experimental results on five standard datasets—PASCAL VOC 2007, Caltech 256, SUN 397, ILSVRC 2010 and ImageNet10K—with up to 9M images and 10K classes, showing that the FV framework is a state-of-the-art patch encoding technique. <s> BIB002 </s> Emotion estimation in crowds: a survey <s> Emotion Estimation <s> Perceiving crowd emotions and understand the situation is vital to control the situations in surveillance applications. This paper introduces the evolution of methods for crowd emotion perception based on bio-inspired probabilistic models. The emotions have been perceived both in an offline and online manner from the crowd. We focus on the perception of emotion from crowd behavior and dynamics. The paper explains few probabilistic algorithms and compares these for detection of emotion of crowds and proposes a probabilistic modelling approach which is trained on data to perceive the emotions of the crowd in an area under surveillance. Emotions are defined as evolving dynamic patterns arising due to interaction of people in an environment with their relationships to the past interaction patterns. Camera sensors are used to track the motion of the individuals within a crowd scenario under observation. The data mining techniques are used to distinguish between different behaviors and events into positive and negative emotions. The results have been evaluated using simulated data from a proposed office environment. <s> BIB003 </s> Emotion estimation in crowds: a survey <s> Emotion Estimation <s> Automatic affect analysis and understanding has become a well established research area in the last two decades. However, little attention has been paid to the analysis of the affect expressed in group settings, either in the form of affect expressed by the whole group collectively or affect expressed by each individual member of the group. This paper presents a framework which, in group settings automatically classifies the affect expressed by each individual group member along both arousal and valence dimensions. We first introduce a novel Volume Quantised Local Zernike Moments Fisher Vectors (vQLZM-FV) descriptor to represent the facial behaviours of individuals in the spatio-temporal domain and then propose a method to recognize the group membership of each individual (i.e., which group the individual in question is part of) by using their face and body behavioural cues. We conduct a set of experiments on a newly collected dataset that contains fourteen recordings of four groups, each consisting of four people watching affective movie stimuli. Our experimental results show that (1) the proposed vQLZM-FV outperforms the other feature representations in affect recognition, and (2) group membership can be recognized using the non-verbal face and body features, indicating that individuals influence each other's behaviours within a group setting. <s> BIB004 </s> Emotion estimation in crowds: a survey <s> Emotion Estimation <s> Estimation of emotions is an essential aspect in developing intelligent systems intended for crowded environments. However, emotion estimation in crowds remains a challenging problem due to the complexity in which human emotions are manifested and the capability of a system to perceive them in such conditions. This paper proposes a hierarchical Bayesian model to learn in unsupervised manner the behavior of individuals and of the crowd as a single entity, and explore the relation between behavior and emotions to infer emotional states. Information about the motion patterns of individuals are described using a self-organizing map, and a hierarchical Bayesian network builds probabilistic models to identify behaviors and infer the emotional state of individuals and the crowd. This model is trained and tested using data produced from simulated scenarios that resemble real-life environments. The conducted experiments tested the efficiency of our method to learn, detect and associate behaviors with emotional states yielding accuracy levels of 74% for individuals and 81% for the crowd, similar in performance with existing methods for pedestrian behavior detection but with novel concepts regarding the analysis of crowds. <s> BIB005
Focusing on the task of emotion estimation, the framework introduced in BIB004 addressed the recognition of individuals' membership and emotions within a group setting by means of multimodal analysis of facial and body expressions. Faces are represented by facial landmark trajectories and extended volume Quantised Local Zernike Moments (QLZM) BIB001 , and encoded into Fisher Vector (FV) BIB002 representations as input to a Gaussian Mixture Model (GMM) classifier to recognize emotions in arousal and valence dimensions. The framework was tested with a self-collected dataset of 3 groups of 4 individuals each, monitored while watching a movie. The proposed approach vQLZM-FV outperformed the compared methods, namely Facial landmarks, body HOG and body HOF. Although this approach focused on investigating the affective response of individuals while watching long-term videos, it is theoretically applicable to crowds under the assumptions that crowd members' face and body are visible for a long-enough interval and within an acceptable resolution. The authors of BIB003 summarize their previous work on three bio-inspired probabilistic algorithms for perception of emotions from crowd dynamics. The first algorithm starts by partitioning the environment using an Instantaneous Topological Map (ITM) and a Dynamic Bayesian Network (DBN) is employed to model conditional interactions occurred in each subregion. these interactions are then converted into super states using a Self-organizing map (SOM) and the occurrence of these super states (events) are encoded by a Gausian mixture model as positive or negative emotions. The second algorithm starts with the detection of events, collecting them over time to obtain behavioral patterns which are then clustered into classes by means of a DBN; the distribution of these classes are modeled using GMM, building one model for positive and one for negative emotions, to finally detect the emotional state by a likelihood ratio test. In the third algorithm the trajectory of single individuals are expressed as transitions (events) between subregions using a DBN and separated models are constructed for the event sequences labeled with a positive or negative emotion, to conclude with a log-likelihood test to determine the emotion according to the movement pattern exhibit by the individual. All three approaches are tested under a simulated scenario, showing the third algorithm to yield the highest emotion prediction accuracy according to the experiments conducted. The authors in BIB005 propose a hierarchical Bayesian model aiming to describe the crowd both at the microscopic and macroscopic level. This approach uses pedestrians trajectories to create a topological map by means of a self-organizing map (SOM), dividing the environment into zones. At the microscopic level, the pedestrians trajectories are described as a Markov process transitioning between zones, and behaviors are modeled according to the origin and destination; each behavior is assigned an emotional label (positive, neutral or negative) according to the time required to reach the estimated destination. At the macroscopic level, the crowd is described by a vector state counting the number of people in each zone at a given time, a second SOM clusters and reduce the dimension of the state vectors to describe the dynamics of the crowd as a Markov process; finally the emotion of the crowd is assigned to be the predominant one as displayed by single individuals. This method is validated by a simulated crowd under different levels of crowd density and multiple behaviors. A common test bed is essential to measure and compare performance among different methods. This section examines two types of datasets, those designed for affective states recognition and those intended for crowd analysis. The objective is to determine whether the reviewed datasets are suited to test computational models dealing with emotion estimation in crowds, for which a discussion is provided in section 4. The datasets considered in this section are not exhaustive but rather representative of the diversity available for such tasks. Both groups of affective and crowd analysis datasets are listed in tables 1 and 2 respectively.