We propose a novel semi-supervised, Multi-Level Sequential Generative Adversarial Network (MLS-GAN) architecture for group activity recognition. In contrast to previous works which utilise manually annotated individual human action predictions, we allow the models to learn it's own internal representations to discover pertinent sub-activities that aid the final group activity recognition task. The generator is fed with person-level and scene-level features that are mapped temporally through LSTM networks. Action-based feature fusion is performed through novel gated fusion units that are able to consider long-term dependencies, exploring the relationships among all individual actions, to learn an intermediate representation or action code' for the current group activity. The network achieves its semi-supervised behaviour by allowing it to perform group action classification together with the adversarial real fake validation. We perform extensive evaluations on different architectural variants to demonstrate the importance of the proposed architecture. Furthermore, we show that utilising both person-level and scene-level features facilitates the group activity prediction better than using only person-level features. Our proposed architecture outperforms current state-of-the-art results for sports and pedestrian based classification tasks on Volleyball and Collective Activity datasets, showing it's flexible nature for effective learning of group activities.
Some early works @cite_6 @cite_5 @cite_29 @cite_28 on group activity recognition have addressed the group activity recognition task on surveillance and sports video datasets with probabilistic and discriminative models that utilise hand-crafted features. As these hand-crafted feature based methods always require feature engineering, attention has shifted towards deep network based methods due to their automatic feature learning capability.
@cite_28 @cite_5 @cite_29 @cite_6
2930494921 797104241 2171544105 2733111742
In this paper we present an approach for classifying the activity performed by a group of people in a video sequence. This problem of group activity recognition can be addressed by examining individual person actions and their relations. Temporal dynamics exist both at the level of individual person actions as well as at the level of group activity. Given a video sequence as input, methods can be developed to capture these dynamics at both person-level and group-level detail. We build a deep model to capture these dynamics based on LSTM (long short-term memory) models. In order to model both person-level and group-level dynamics, we present a 2-stage deep temporal model for the group activity recognition problem. In our approach, one LSTM model is designed to represent action dynamics of individual people in a video sequence and another LSTM model is designed to aggregate person-level information for group activity recognition. We collected a new dataset consisting of volleyball videos labeled with individual and group activities in order to evaluate our method. Experimental results on this new Volleyball Dataset and the standard benchmark Collective Activity Dataset demonstrate the efficacy of the proposed models. Activity recognition in video has become increasingly important due to its many applications ranging from in-home elder care, surveillance, human computer interaction to automatic sports commentary. To date, most approaches to video rely on fully supervised settings that require time consuming and error prone manual labeling. Moreover, existing supervised approaches are typically tailored for classification, not detection problems (the spatial and temporal support of the action has to be detected). Recently, weakly-supervised learning (WSL) approaches were able to learn discriminative classifiers while localizing the action in space and or time using weak labels. However, existing approaches for WSL provide coarse localization in terms of spatial regions or spatio-temporal volumes. Moreover, it is unclear how to extend current approaches to the multi-label case that is common in practical applications. This paper proposes a matrix completion approach to the problem of WSL for multi-label learning for video. Our approach localizes non-rectangular spatio-temporal discriminative regions that are inferred by clustering regions of common texture and motion features. We illustrate how our approach improves existing WSL and supervised learning techniques in three standard databases: Hollywood, UCF sports, and MSR-II. This paper addresses a new problem, that of multiscale activity recognition. Our goal is to detect and localize a wide range of activities, including individual actions and group activities, which may simultaneously co-occur in high-resolution video. The video resolution allows for digital zoom-in (or zoom-out) for examining fine details (or coarser scales), as needed for recognition. The key challenge is how to avoid running a multitude of detectors at all spatiotemporal scales, and yet arrive at a holistically consistent video interpretation. To this end, we use a three-layered AND-OR graph to jointly model group activities, individual actions, and participating objects. The AND-OR graph allows a principled formulation of efficient, cost-sensitive inference via an explore-exploit strategy. Our inference optimally schedules the following computational processes: 1) direct application of activity detectors --- called α process; 2) bottom-up inference based on detecting activity parts --- called β process; and 3) top-down inference based on detecting activity context --- called γ process. The scheduling iteratively maximizes the log-posteriors of the resulting parse graphs. For evaluation, we have compiled and benchmarked a new dataset of high-resolution videos of group and individual activities co-occurring in a courtyard of the UCLA campus. Human group activity recognition has drawn the attention of researchers worldwide because of the significant role it plays in many applications, including video surveillance and public security. Existing solutions for group activity recognition rely on human detection and tracking. To ensure high detection accuracy, current state-of-the-art tracking techniques require human supervision to identify objects of interest before automatic tracking can take place. This limitation has prevented existing approaches from being used in real-world applications. In scenarios when human supervision is unavailable, tracking algorithms could generate inaccurate trajectories and cause a decrease in performance for the existing group analysis methods. To address the aforementioned drawbacks, we investigate in this paper an end-to-end deep model, Differential Recurrent Convolutional Neural Networks (DRCNN). Our model consists of convolutional neural networks (CNN) and stacked differential long short-term memory (DLSTM) networks. It takes sequential raw video data as input and does not consider each group member as an individual object. Different from traditional non-end-to-end solutions which separate the steps of feature extraction and parameter learning, DRCNN utilizes a unified deep model to optimize the parameters of CNN and LSTM hand in hand. It thus has the potential of generating a more harmonious model. In addition, taking advantage of the semantic representation of CNN and the memory states of DLSTM, DRCNN has strong capabilities in understanding complex scene semantics and group dynamics. Extensive experimental studies indicate that the proposed technique can accomplish the task of fully automatic group activity recognition without sacrificing performance, and even outperforms the human-aided state-ofthe- art methods on two benchmark group activity datasets. To the best of our knowledge, this is the first end-to-end group activity recognition technique ever proposed.