Humans can easily recognize the importance of people in social event images, and they always focus on the most important individuals. However, learning to learn the relation between people in an image, and inferring the most important person based on this relation, remains undeveloped. In this work, we propose a deep imPOrtance relatIon NeTwork (POINT) that combines both relation modeling and feature learning. In particular, we infer two types of interaction modules: the person-person interaction module that learns the interaction between people and the event-person interaction module that learns to describe how a person is involved in the event occurring in an image. We then estimate the importance relations among people from both interactions and encode the relation feature from the importance relations. In this way, POINT automatically learns several types of relation features in parallel, and we aggregate these relation features and the person's feature to form the importance feature for important people classification. Extensive experimental results show that our method is effective for important people detection and verify the efficacy of learning to learn relations for important people detection.
Recently, the importance of generic object categories and persons has attracted increased attention and has been studied by several researchers @cite_16 @cite_6 @cite_19 @cite_0 @cite_7 @cite_10 @cite_4 @cite_17 . @cite_4 focused on studying the relative importance between a pair of faces, either in the same image or separate images, and developed a regression model for predicting the importance of faces. The authors designed customized features containing spatial and saliency information of faces for important face detection. In addition, @cite_18 trained an attention-based model with event recognition labels to assign attention importance scores to all detected individuals to measure how related they were to basketball game videos. More specifically , they proposed utilizing spatial and appearance features of persons including temporal information to infer the importance score of all detected persons. Recently, @cite_17 modeled all detected people in a hybrid interaction graph by organizing the interaction among persons sequentially and developed PersonRank, a graphical model to rank the persons by inferring the importance scores of persons from person-person interactions constructed on four types of features that have been pretrained for other tasks.
@cite_18 @cite_4 @cite_7 @cite_6 @cite_0 @cite_19 @cite_16 @cite_10 @cite_17
2963920562 2739491435 2012451022 2134557905
Always, some individuals in images are more important attractive than the others in some events such as presentation, basketball game or speech. However, it is challenging to ?nd important people among all individuals in an image directly based on their spatial or appearance information due to the existence of diverse variations of pose, action, appearance of persons an various changes of occasions. We overcome this challenge by constructing a multiple HyperInteraction Graph that treats each individual in an image as a node and inferring the most active node from the interactions estimated by using various types of cues. We model a pairwise interaction between people as an edge message communicated between nodes, resulting in a bidirectional pairwise-interaction graph. To enrich the person-person interaction estimation, we further introduce a unidirectional hyper-interaction graph that models the consensus of interactions between a focal person and any person in his her local region around. Finally, we modify the PageRank algorithm to infer the activeness of people on the multiple Hybrid-Interaction Graph (HIG), the union of the pairwise-interaction and hyper-interaction graphs, and we call our algorithm the PersonRank. In order to provide publicable datasets for evaluation, we have contributed a new dataset called Multi-scene Important People Image Dataset and gathered a NCAA Basketball Image Dataset from sports game sequences. We have demonstrated that the proposed PersonRank outperforms related methods clearly and substantially. Our code and datasets are available at https: weihonglee.github.io Projects PersonRank.htm. Tracking multiple persons in a monocular video of a crowded scene is a challenging task. Humans can master it even if they loose track of a person locally by re-identifying the same person based on their appearance. Care must be taken across long distances, as similar-looking persons need not be identical. In this work, we propose a novel graph-based formulation that links and clusters person hypotheses over time by solving an instance of a minimum cost lifted multicut problem. Our model generalizes previous works by introducing a mechanism for adding long-range attractive connections between nodes in the graph without modifying the original set of feasible solutions. This allows us to reward tracks that assign detections of similar appearance to the same person in a way that does not introduce implausible solutions. To effectively match hypotheses over longer temporal gaps we develop new deep architectures for re-identification of people. They combine holistic representations extracted with deep networks and body pose layout obtained with a state-of-the-art pose estimation model. We demonstrate the effectiveness of our formulation by reporting a new state-of-the-art for the MOT16 benchmark. The code and pre-trained models are publicly available. We present a new method to classify human activities by leveraging on the cues available from depth images alone. Towards this end, we propose a descriptor which couples depth and spatial information of the segmented body to describe a human pose. Unique poses (i.e. codewords) are then identified by a spatial-based clustering step. Given a video sequence of depth images, we segment humans from the depth images and represent these segmented bodies as a sequence of codewords. We exploit unique poses of an activity and the temporal ordering of these poses to learn subsequences of codewords which are strongly discriminative for the activity. Each discriminative subsequence acts as a classifier and we learn a boosted ensemble of discriminative subsequences to assign a confidence score for the activity label of the test sequence. Unlike existing methods which demand accurate tracking of 3D joint locations or couple depth with color image information as recognition cues, our method requires only the segmentation masks from depth images to recognize an activity. Experimental results on the publicly available Human Activity Dataset (which comprises 12 challenging activities) demonstrate the validity of our method, where we attain a precision recall of 78.1 75.4 when the person was not seen before in the training set, and 94.6 93.1 when the person was seen before. We assess the applicability of several popular learning methods for the problem of recognizing generic visual categories with invariance to pose, lighting, and surrounding clutter. A large dataset comprising stereo image pairs of 50 uniform-colored toys under 36 azimuths, 9 elevations, and 6 lighting conditions was collected (for a total of 194,400 individual images). The objects were 10 instances of 5 generic categories: four-legged animals, human figures, airplanes, trucks, and cars. Five instances of each category were used for training, and the other five for testing. Low-resolution grayscale images of the objects with various amounts of variability and surrounding clutter were used for training and testing. Nearest neighbor methods, support vector machines, and convolutional networks, operating on raw pixels or on PCA-derived features were tested. Test error rates for unseen object instances placed on uniform backgrounds were around 13 for SVM and 7 for convolutional nets. On a segmentation recognition task with highly cluttered images, SVM proved impractical, while convolutional nets yielded 16 7 error. A real-time version of the system was implemented that can detect and classify objects in natural scenes at around 10 frames per second.