Social relations are the foundation of human daily life. Developing techniques to analyze such relations from visual data bears great potential to build machines that better understand us and are capable of interacting with us at a social level. Previous investigations have remained partial due to the overwhelming diversity and complexity of the topic and consequently have only focused on a handful of social relations. In this paper, we argue that the domain-based theory from social psychology is a great starting point to systematically approach this problem. The theory provides coverage of all aspects of social relations and equally is concrete and predictive about the visual attributes and behaviors defining the relations included in each domain. We provide the first dataset built on this holistic conceptualization of social life that is composed of a hierarchical label space of social domains and social relations. We also contribute the first models to recognize such domains and relations and find superior performance for attribute based features. Beyond the encouraging performance of the attribute based approach, we also find interpretable features that are in accordance with the predictions from social psychology literature. Beyond our findings, we believe that our contributions more tightly interleave visual recognition and social psychology theory that has the potential to complement the theoretical work in the area with empirical and data-driven models of social life.
Social life endows various social appearances to people. Some research focus on urban tribes in daily life @cite_6 , social categories defined by Wikipedia @cite_0 @cite_1 , and popular groups such as Loli'', Syota'' and Goddess'' which are mostly derived from social networks @cite_32 . These fine-grained categorization uses body face positions and attributes such as age, face appearance, hair style, clothing style and so on. Occupation recognition studies @cite_29 @cite_14 not only use personal attributes but also leverage the contextual information in a semantic level, , a waiter is more likely to stand beside sitting consumers in a restaurant.
@cite_14 @cite_29 @cite_1 @cite_32 @cite_6 @cite_0
2134007045 2155171143 2043006422 1925450849 1993636005
In this paper, we investigate the problem of recognizing occupations of multiple people with arbitrary poses in a photo. Previous work utilizing single person's nearly frontal clothing information and fore background context preliminarily proves that occupation recognition is computationally feasible in computer vision. However, in practice, multiple people with arbitrary poses are common in a photo, and recognizing their occupations is even more challenging. We argue that with appropriately built visual attributes, co-occurrence, and spatial configuration model that is learned through structure SVM, we can recognize multiple people's occupations in a photo simultaneously. To evaluate our method's performance, we conduct extensive experiments on a new well-labeled occupation database with 14 representative occupations and over 7K images. Results on this database validate our method's effectiveness and show that occupation recognition is solvable in a more general case. Predicting human occupations in photos has great application potentials in intelligent services and systems. However, using traditional classification methods cannot reliably distinguish different occupations due to the complex relations between occupations and the low-level image features. In this paper, we investigate the human occupation prediction problem by modeling the appearances of human clothing as well as surrounding context. The human clothing, regarding its complex details and variant appearances, is described via part-based modeling on the automatically aligned patches of human body parts. The image patches are represented with semantic-level patterns such as clothes and haircut styles using methods based on sparse coding towards informative and noise-tolerant capacities. This description of human clothing is proved to be more effective than traditional methods. Different kinds of surrounding context are also investigated as a complementarity of human clothing features in the cases that the background information is available. Experiments are conducted on a well labeled image database that contains more than 5; 000 images from 20 representative occupation categories. The preliminary study shows the human occupation is reasonably predictable using the proposed clothing features and possible context. When people gather for a group photo, they are together for a social reason. Past work has shown that these social relationships affect how people position themselves in a group photograph. We propose classifying the type of group photo based on the spatial arrangement and the predicted attributes of the faces in the image. We propose a matching algorithm for finding images from a training set that have both similar arrangement of faces and attribute correspondence. We formulate the problem as a bipartite matching problem where the faces from each of the pair of images are nodes in the graph. Our work demonstrates that face arrangement, when combined with attribute (age and gender) correspondence, is a useful cue in capturing an approximate social essence of the group of people, and lets us understand why the group of people gathered for the photo. Human group, which indicates the people who share similar characteristics, is used to categorize humans into distinct populations or groups. In recent years, with the explosive growth of image, new concepts of human group are blooming in social networks . People in the same human group can be categorized by their facial and clothes appearance characteristics. In this work, we propose an approach to understanding the new concepts of human group with few positive samples. To this end, we construct visual models crossing two modalities related to human images and surrounding texts. Two convolutional neural networks based on face and upper body are constructed separately. Two different convolutional neural networks (CNNs) architectures are explored for visual pre-traing. To assist the human group recognition, we also merge global convolutional feature of the image. The surrounding texts are represented by semantical vectors and utilized as image labels. We transform words in the text into fixed length vectors by the skip-gram model. Then the texts corresponding to each image are converted into one feature vector by sparse coding and max pooling. Given a few positive samples of new concepts of human group, the visual model can be improved to understand the semantical meaning of the new label. The experimental results demonstrate the effectiveness of the proposed visual model and show the excellent learning capacity with few samples. Iljung S. Kwak1 iskwak@cs.ucsd.edu Ana C. Murillo2 acm@unizar.es Peter N. Belhumeur3 belhumeur@cs.columbia.edu David Kriegman1 kriegman@cs.ucsd.edu Serge Belongie1 sjb@cs.ucsd.edu 1 Dept. of Computer Science and Engineering University of California, San Diego, USA. 2 Dpt. Informatica e Ing. Sistemas Inst. Investigacion en Ingenieria de Aragon. University of Zaragoza, Spain. 3 Department of Computer Science Columbia University, USA.