In this paper we address the problem of multi-cue affect recognition in challenging environments such as child-robot interaction. Towards this goal we propose a method for automatic recognition of affect that leverages body expressions alongside facial expressions, as opposed to traditional methods that usually focus only on the latter. We evaluate our methods on a challenging child-robot interaction database of emotional expressions, as well as on a database of emotional expressions by actors, and show that the proposed method achieves significantly better results when compared with the facial expression baselines, can be trained both jointly and separately, and offers us computational models for both the individual modalities, as well as for the whole body emotion.
Why bodies? It is rather puzzling that given the massive interest in affective neuroscience in the last decade, it still seems to make sense to raise the question ‘Why bodies’ and to try to provide an answer to it, as is the goal of this article. There are now hundreds of articles on human emotion perception ranging from behavioural studies to brain imaging experiments. These experimental studies complement decades of reports on affective disorders in neurological patients and clinical studies of psychiatric populations. The most cursory glance at the literature on emotion in humans, now referred to by the umbrella term of social and affective neuroscience, shows that over 95 per cent of them have used faces as stimuli. Of the remaining 5 per cent, a few have used scenes or auditory information including human voices, music or environmental sounds. But by far the smallest number has looked into whole-body expressions. As a rough estimate, a search on PubMed today, 1 May 2009, yields 3521 hits for emotion × faces, 1003 hits for emotion × music and 339 hits for emotion × bodies. When looking in more detail, the body × emotion category in fact yields a majority of papers on well-being, nursing, sexual violence or organ donation. But the number of cognitive and affective neuroscience studies of emotional body perception as of today is lower than 20. @PARASPLIT Why then have whole bodies and bodily expressions not attracted the attention of researchers so far? The goal of this article is to contribute some elements for an answer to this question. I believe that there is something to learn from the historical neglect of bodies and bodily expressions. I will next address some historical misconceptions about whole-body perception, and in the process I intend not only to provide an impetus for this kind of work but also to contribute to a better understanding of the significance of the affective dimension of behaviour, mind and brain as seen from the vantage point of bodily communication. Subsequent sections discuss available evidence for the neurofunctional basis of facial and bodily expressions as well as neuropsychological and clinical studies of bodily expressions. Thanks to the decreasing cost of whole-body sensing technology and its increasing reliability, there is an increasing interest in, and understanding of, the role played by body expressions as a powerful affective communication channel. The aim of this survey is to review the literature on affective body expression perception and recognition. One issue is whether there are universal aspects to affect expression perception and recognition models or if they are affected by human factors such as culture. Next, we discuss the difference between form and movement information as studies have shown that they are governed by separate pathways in the brain. We also review psychological studies that have investigated bodily configurations to evaluate if specific features can be identified that contribute to the recognition of specific affective states. The survey then turns to automatic affect recognition systems using body expressions as at least one input modality. The survey ends by raising open questions on data collecting, labeling, modeling, and setting benchmarks for comparing automatic recognition systems. Body movements communicate affective expressions and, in recent years, computational models have been developed to recognize affective expressions from body movements or to generate movements for virtual agents or robots which convey affective expressions. This survey summarizes the state of the art on automatic recognition and generation of such movements. For both automatic recognition and generation, important aspects such as the movements analyzed, the affective state representation used, and the use of notation systems is discussed. The survey concludes with an outline of open problems and directions for future work. Automatic emotion recognition has become a trending research topic in the past decade. While works based on facial expressions or speech abound, recognizing affect from body gestures remains a less explored topic. We present a new comprehensive survey hoping to boost research in the field. We first introduce emotional body gestures as a component of what is commonly known as "body language" and comment general aspects as gender differences and culture dependence. We then define a complete framework for automatic emotional body gesture recognition. We introduce person detection and comment static and dynamic body pose estimation methods both in RGB and 3D. We then comment the recent literature related to representation learning and emotion recognition from images of emotionally expressive gestures. We also discuss multi-modal approaches that combine speech or face with body gestures for improved emotion recognition. While pre-processing methodologies (e.g. human detection and pose estimation) are nowadays mature technologies fully developed for robust large scale analysis, we show that for emotion recognition the quantity of labelled data is scarce, there is no agreement on clearly defined output spaces and the representations are shallow and largely based on naive geometrical representations.