Feature Encodings and Poolings for Action and Event Recognition: A Comprehensive Survey <s> Spatiotemporal Grids Encoding <s> We address recognition and localization of human actions in realistic scenarios. In contrast to the previous work studying human actions in controlled settings, here we train and test algorithms on real movies with substantial variation of actions in terms of subject appearance, motion, surrounding scenes, viewing angles and spatio-temporal extents. We introduce a new annotated human action dataset and use it to evaluate several existing methods. We in particular focus on boosted space-time window classifiers and introduce "keyframe priming" that combines discriminative models of human motion and shape within an action. Keyframe priming is shown to significantly improve the performance of action detection. We present detection results for the action class "drinking" evaluated on two episodes of the movie "Coffee and Cigarettes". <s> BIB001 </s> Feature Encodings and Poolings for Action and Event Recognition: A Comprehensive Survey <s> Spatiotemporal Grids Encoding <s> The aim of this paper is to address recognition of natural human actions in diverse and realistic video settings. This challenging but important subject has mostly been ignored in the past due to several problems one of which is the lack of realistic and annotated video datasets. Our first contribution is to address this limitation and to investigate the use of movie scripts for automatic annotation of human actions in videos. We evaluate alternative methods for action retrieval from scripts and show benefits of a text-based classifier. Using the retrieved action samples for visual learning, we next turn to the problem of action classification in video. We present a new method for video classification that builds upon and extends several recent ideas including local space-time features, space-time pyramids and multi-channel non-linear SVMs. The method is shown to improve state-of-the-art results on the standard KTH action dataset by achieving 91.8% accuracy. Given the inherent problem of noisy labels in automatic annotation, we particularly investigate and show high tolerance of our method to annotation errors in the training set. We finally apply the method to learning and classifying challenging action classes in movies and show promising results. <s> BIB002 </s> Feature Encodings and Poolings for Action and Event Recognition: A Comprehensive Survey <s> Spatiotemporal Grids Encoding <s> We propose semantic model vectors, an intermediate level semantic representation, as a basis for modeling and detecting complex events in unconstrained real-world videos, such as those from YouTube. The semantic model vectors are extracted using a set of discriminative semantic classifiers, each being an ensemble of SVM models trained from thousands of labeled web images, for a total of 280 generic concepts. Our study reveals that the proposed semantic model vectors representation outperforms-and is complementary to-other low-level visual descriptors for video event modeling. We hence present an end-to-end video event detection system, which combines semantic model vectors with other static or dynamic visual descriptors, extracted at the frame, segment, or full clip level. We perform a comprehensive empirical study on the 2010 TRECVID Multimedia Event Detection task (http://www.nist.gov/itl/iad/mig/med10.cfm), which validates the semantic model vectors representation not only as the best individual descriptor, outperforming state-of-the-art global and local static features as well as spatio-temporal HOG and HOF descriptors, but also as the most compact. We also study early and late feature fusion across the various approaches, leading to a 15% performance boost and an overall system performance of 0.46 mean average precision. In order to promote further research in this direction, we made our semantic model vectors for the TRECVID MED 2010 set publicly available for the community to use (http://www1.cs.columbia.edu/~mmerler/SMV.html). <s> BIB003 </s> Feature Encodings and Poolings for Action and Event Recognition: A Comprehensive Survey <s> Spatiotemporal Grids Encoding <s> T his paper present s a gener al surveillance event detection system that participates in the interactive Surveillance Event Detection ( i SED) task of TRECVID 2013 . In the proposed system, a set of spatio - temporal features including Space - Time Interest Points (STIP) and Dense Trajectories are extracted, and a sliding temporal window is empl oyed as the detection unit. Fisher Vector is used to encode lo w - level features as the representation of each sliding window. Both feature - level and decision - level fusion s are used to combine multiple features . In order to deal with the highly imbalanced nature of surveillance data, the system performs detections usin g the CascadeSVMs algorithm according to each specific event and camera view . Two different interactive environments are evaluated, one focuses on high throughput and the other includes related result expansion. In the primary run evaluations, o ur system r anks the top in 2 out of 7 event detection tasks. <s> BIB004 </s> Feature Encodings and Poolings for Action and Event Recognition: A Comprehensive Survey <s> Spatiotemporal Grids Encoding <s> Local spatio-temporal features and bag-of-features representations have become popular for action recognition. A recent trend is to use dense sampling for better performance. While many methods claimed to use dense feature sets, most of them are just denser than approaches based on sparse interest point detectors. In this paper, we explore sampling with high density on action recognition. We also investigate the impact of random sampling over dense grid for computational efficiency. We present a real-time action recognition system which integrates fast random sampling method with local spatio-temporal features extracted from a Local Part Model. A new method based on histogram intersection kernel is proposed to combine multiple channels of different descriptors. Our technique shows high accuracy on the simple KTH dataset, and achieves state-of-the-art on two very challenging real-world datasets, namely, 93% on KTH, 83.3% on UCF50 and 47.6% on HMDB51. <s> BIB005 </s> Feature Encodings and Poolings for Action and Event Recognition: A Comprehensive Survey <s> Spatiotemporal Grids Encoding <s> In this paper, we present two video - based event detection systems de veloped by City College of New York (CCNY) for the Surveillance Event Detection (SED) task of TRECVID 2014 . One is a generic event detection system that is applied to all the events of the SED task except CellToEar event . In this proposed system, the detection unit is differentiated by a sliding temporal window and a set of spatio - temporal features including STIP - HOG/H OF, DT - Trajectory, and DT - MBH. Fisher Vector is adopted to encode low - lev el features as the representation of each sliding window. Since the surveillance data is highly imbalanced, we chop the training data into balanced small chunks, and within each data chunk a random forest classifier is learned. In the testing phase, dec isi on - level fusion is applied to combine the prediction results by multiple random forests. The second system is specifically designed to the CellToEar event since it has distinct properties which are unsuited for traditional action based approaches but well compatible with the static gesture detections . <s> BIB006
Although spatial 2D encodings, such as spatial pyramid and JS tiling, could improve recognition accuracy by considering spatial layouts of interest patches, they are designed mainly for describing one image or one frame and thus not sufficient for describing spatiotemporal targets, such as actions and events. Laptev et al. BIB002 BIB001 proposed a spatiotemporal grids (or spatiotemporal bag-of-features) encoding approach for action recognition, which is an extension of the spatial pyramid encoding to spatiotemporal domain. The pipeline involves: (1) A K-dimension, e.g., K = 4000, visual vocabulary is constructed by clustering interest patches sampled from the training videos, such as HOG patches or HOF patches around detected interest points, with k-means algorithm. The authors concluded that the spatiotemporal grids give a significant gain over the standard bag-of-features methods for action recognition. Besides, the same spatiotemporal grids-based pipeline has been widely used and shown promising results by several research groups BIB004 BIB003 BIB006 BIB005 for multimedia event detection in TRECVID dataset.