While machine learning approaches to visual emotion recognition offer great promise, current methods consider training and testing models on small scale datasets covering limited visual emotion concepts. Our analysis identifies an important but long overlooked issue of existing visual emotion benchmarks in the form of dataset biases. We design a series of tests to show and measure how such dataset biases obstruct learning a generalizable emotion recognition model. Based on our analysis, we propose a webly supervised approach by leveraging a large quantity of stock image data. Our approach uses a simple yet effective curriculum guided training strategy for learning discriminative emotion features. We discover that the models learned using our large scale stock image dataset exhibit significantly better generalization ability than the existing datasets without the manual collection of even a single label. Moreover, visual representation learned using our approach holds a lot of promise across a variety of tasks on different image and video datasets.
 In this survey, we argue that using structured vocabularies is capital to the success of image annotation. We analyze literature on image annotation uses and user needs, and we stress the need for automatic annotation. We briefly expose the difficulties posed to machines for this task and how it relates to controlled vocabularies. We survey contributions in the field showing how structures are introduced. First we present studies that use unstructured vocabulary, focusing on those introducing links between categories or between features. Then we review work using structured vocabularies as an input and analyze how the structure is exploited. Learning a visual concept from a small number of positive examples is a significant challenge for machine learning algorithms. Current methods typically fail to find the appropriate level of generalization in a concept hierarchy for a given set of visual examples. Recent work in cognitive science on Bayesian models of generalization addresses this challenge, but prior results assumed that objects were perfectly recognized. We present an algorithm for learning visual concepts directly from images, using probabilistic predictions generated by visual classifiers as the input to a Bayesian generalization model. As no existing challenge data tests this paradigm, we collect and make available a new, large-scale dataset for visual concept learning using the ImageNet hierarchy as the source of possible concepts, with human annotators to provide ground truth labels as to whether a new image is an instance of each concept using a paradigm similar to that used in experiments studying word learning in children. We compare the performance of our system to several baseline algorithms, and show a significant advantage results from combining visual classifiers with the ability to identify an appropriate level of abstraction using Bayesian generalization. A semantically meaningful image hierarchy can ease the human effort in organizing thousands and millions of pictures (e.g., personal albums), and help to improve performance of end tasks such as image annotation and classification. Previous work has focused on using either low-level image features or textual tags to build image hierarchies, resulting in limited success in their general usage. In this paper, we propose a method to automatically discover the “semantivisual” image hierarchy by incorporating both image and tag information. This hierarchy encodes a general-to-specific image relationship. We pay particular attention to quantifying the effectiveness of the learned hierarchy, as well as comparing our method with others in the end-task applications. Our experiments show that humans find our semantivisual image hierarchy more effective than those solely based on texts or low-level visual features. And using the constructed image hierarchy as a knowledge ontology, our algorithm can perform challenging image classification and annotation tasks more accurately. The computational complexity of current visual categorization algorithms scales linearly at best with the number of categories. The goal of classifying simultaneously Ncat = 104 - 105 visual categories requires sub-linear classification costs. We explore algorithms for automatically building classification trees which have, in principle, logNcat complexity. We find that a greedy algorithm that recursively splits the set of categories into the two minimally confused subsets achieves 5-20 fold speedups at a small cost in classification performance. Our approach is independent of the specific classification algorithm used. A welcome by-product of our algorithm is a very reasonable taxonomy of the Caltech-256 dataset.   In image classification, visual separability between different object categories is highly uneven, and some categories are more difficult to distinguish than others. Such difficult categories demand more dedicated classifiers. However, existing deep convolutional neural networks (CNN) are trained as flat N-way classifiers, and few efforts have been made to leverage the hierarchical structure of categories. In this paper, we introduce hierarchical deep CNNs (HD-CNNs) by embedding deep CNNs into a category hierarchy. An HD-CNN separates easy classes using a coarse category classifier while distinguishing difficult classes using fine category classifiers. During HD-CNN training, component-wise pretraining is followed by global finetuning with a multinomial logistic loss regularized by a coarse category consistency term. In addition, conditional executions of fine category classifiers and layer parameter compression make HD-CNNs scalable for large-scale visual recognition. We achieve state-of-the-art results on both CIFAR100 and large-scale ImageNet 1000-class benchmark datasets. In our experiments, we build up three different HD-CNNs and they lower the top-1 error of the standard CNNs by 2.65 , 3.1 and 1.1 , respectively.  Supervised learning using deep convolutional neural network has shown its promise in large-scale image classification task. As a building block, it is now well positioned to be part of a larger system that tackles real-life multimedia tasks. An unresolved issue is that such model is trained on a static snapshot of data. Instead, this paper positions the training as a continuous learning process as new classes of data arrive. A system with such capability is useful in practical scenarios, as it gradually expands its capacity to predict increasing number of new classes. It is also our attempt to address the more fundamental issue: a good learning system must deal with new knowledge that it is exposed to, much as how human do. We developed a training algorithm that grows a network not only incrementally but also hierarchically. Classes are grouped according to similarities, and self-organized into levels. The newly added capacities are divided into component models that predict coarse-grained superclasses and those return final prediction within a superclass. Importantly, all models are cloned from existing ones and can be trained in parallel. These models inherit features from existing ones and thus further speed up the learning. Our experiment points out advantages of this approach, and also yields a few important open questions. Overview. Part I: The Nature of Emotion. P. Shaver, J. Schwartz, D. Kirson, and C. O'Connor, Emotion Knowledge: Further Exploration of a Prototype Approach. N. Frijda, The Laws of Emotion. Part II: The Role of Cognition. S. Schachter and J. Singer, Cognitive, Social, and Physiological Determinants of Emotional State. C. Smith and R. Lazarus, Appraisal Components, Core Relational Themes, and the Emotions. Part III: Culture and Socialization. H. Markus and S. Kitayama, The Cultural Construction of Self and Emotion: Implications for Social Behavior. Part IV: Gender Differences: Evolution, Socialization, and Stereotyping. D. Buss, R. Larsen, D. Westen, and J. Semmelroth, Sex Differences in Jealousy: Questioning the "Fitness" of the Model. M. Robinson, J. Johnson, and S. Shields, the Gender Heuristic and the Database: Factors Affecting the Perception of Gender-related Differences in the Experience and Display of Emotions. Part V: The Social Functions of Emotions. D. Keltner and J. Haidt, Social Functions of Emotions at Four Levels of Analysis. P. Salovey, C. Hsee, and J. Mayer, Emotional Intelligence and the Self-Regulation of Affect. Part VI: Emotion and Social Cognition. J. Forgas and G. Bower, Mood Effects on Person-Perception Judgements. H. Bless, G. Bohner, N. Schwarz, and F. Strack, Mood and Persuasion: A Cognitive Response Analysis. Part VII: Emotion's Effects on Others. J. Coyne, Depression and the Response of Others. V. Christophe and B. Rime, Exposure to the Social Sharing of Emotion: Emotional Impact, Listener Responses and Secondary Social Sharing. Part VIII: Emotion and Facial Expressions. P. Ekman, W. Friesen, and S. Ancoli, Facial Signs of Emotional Experience. A. Fridlund, Sociality of Solitary Smiling: Potentiation by an Implicit Audience. Part IX: Shame, Guilt, Envy, and Jealousy. J. Tagney, P. Wagner, C. Fletcher, and R. Gramzow, Shamed into Anger? The Relation of Shame and Guilt to Anger and Self-reported Aggression. R. Baumeister, A. Stillwell, and T. Heatherton, Guilt: An Interpersonal Approach. W. Parrott, The Emotional Experiences of Envy and Jealousy. Part X: The Nature of Emotion, Revisited: The Case of Anger. L. Berkowitz, On the Formation and Regulation of Anger and Aggression: A Cognitive-neoassociationistic Approach. J. Averill, Studies on Anger and Aggression: Implications for Theories of Emotion. In this paper we propose to use lexical semantic networks to extend the state-of-the-art object recognition techniques. We use the semantics of image labels to integrate prior knowledge about inter-class relationships into the visual appearance learning. We show how to build and train a semantic hierarchy of discriminative classifiers and how to use it to perform object detection. We evaluate how our approach influences the classification accuracy and speed on the Pascal VOC challenge 2006 dataset, a set of challenging real-world images. We also demonstrate additional features that become available to object recognition due to the extension with semantic inference tools- we can classify high-level categories, such as animals, and we can train part detectors, for example a window detector, by pure inference in the semantic network.  