In this work, we present a framework based on multi-stream convolutional neural networks (CNNs) for group activity recognition. Streams of CNNs are separately trained on different modalities and their predictions are fused at the end. Each stream has two branches to predict the group activity based on person and scene level representations. A new modality based on the human pose estimation is presented to add extra information to the model. We evaluate our method on the Volleyball and Collective Activity datasets. Experimental results show that the proposed framework is able to achieve state-of-the-art results when multiple or single frames are given as input to the model with 90.50 and 86.61 accuracy on Volleyball dataset, respectively, and 87.01 accuracy of multiple frames group activity on Collective Activity dataset.
There have been efforts to use probabilistic graphical models to tackle the group activity recognition problem. @cite_26 propose a graphical model with person-person and group-person factors and employ a two-stage inference mechanism to find the optimal graph structure and the best possible labels for the individual actions and collective activity. In @cite_15 , the authors explore the idea of tracking persons and predicting their activity as a group in a joint probabilistic framework. @cite_17 use a latent graph model for multi-target tracking, activity group localization and group activity recognition.
@cite_15 @cite_26 @cite_17
100367037 2047499569 2320007652
We present a coherent, discriminative framework for simultaneously tracking multiple people and estimating their collective activities. Instead of treating the two problems separately, our model is grounded in the intuition that a strong correlation exists between a person's motion, their activity, and the motion and activities of other nearby people. Instead of directly linking the solutions to these two problems, we introduce a hierarchy of activity types that creates a natural progression that leads from a specific person's motion to the activity of the group as a whole. Our model is capable of jointly tracking multiple people, recognizing individual activities (atomic activities), the interactions between pairs of people (interaction activities), and finally the behavior of groups of people (collective activities). We also propose an algorithm for solving this otherwise intractable joint inference problem by combining belief propagation with a version of the branch and bound algorithm equipped with integer programming. Experimental results on challenging video datasets demonstrate our theoretical claims and indicate that our model achieves the best collective activity classification results to date. In this paper, we go beyond recognizing the actions of individuals and focus on group activities. This is motivated from the observation that human actions are rarely performed in isolation; the contextual information of what other people in the scene are doing provides a useful cue for understanding high-level activities. We propose a novel framework for recognizing group activities which jointly captures the group activity, the individual person actions, and the interactions among them. Two types of contextual information, group-person interaction and person-person interaction, are explored in a latent variable framework. In particular, we propose three different approaches to model the person-person interaction. One approach is to explore the structures of person-person interaction. Differently from most of the previous latent structured models, which assume a predefined structure for the hidden layer, e.g., a tree structure, we treat the structure of the hidden layer as a latent variable and implicitly infer it during learning and inference. The second approach explores person-person interaction in the feature level. We introduce a new feature representation called the action context (AC) descriptor. The AC descriptor encodes information about not only the action of an individual person in the video, but also the behavior of other people nearby. The third approach combines the above two. Our experimental results demonstrate the benefit of using contextual information for disambiguating group activities. A latent graphical model integrating multi-target tracking, group discovery, and activity recognition is proposed.Performance of activity recognition improves when multi-target tracking and group clustering are incorporated.Group activities are better recognized based on the structured relations within the group and group-group compatibilities.Increasing the connectivity of different groups improves the overall performance.Incorporating activity information leads to robust group localization in the video. Beyond recognizing actions of individuals, activity group localization in videos aims to localize groups of persons in spatiotemporal spaces and recognize what activity the group performs. In this paper, we propose a latent graph model to simultaneously address the problem of multi-target tracking, group discovery and activity recognition. Our key insight is to exploit the contextual relations among people. We present them as a latent relational graph, which hierarchically encodes the association potentials between tracklets, intra-group interactions, correlations, and inter-group compatibilities. Our model is capable of propagating multiple evidences among different layers of the latent graph. Particularly, associated tracklets assist accurate group discovery, activity recognition can benefit from knowing the whole structured groups, and the group and activity information in turn provides strong cues for establishing coherent associations between tracklets. Experiments on five datasets demonstrate that our model achieves both significant improvements in activity group localization and competitive performance on activity recognition.