A survey of datasets for visual tracking <s> Individual action/activity <s> With nearly one billion online videos viewed everyday, an emerging new frontier in computer vision research is recognition and search in video. While much effort has been devoted to the collection and annotation of large scalable static image datasets containing thousands of image categories, human action datasets lag far behind. Current action recognition databases contain on the order of ten different action categories collected under fairly controlled conditions. State-of-the-art performance on these datasets is now near ceiling and thus there is a need for the design and creation of new benchmarks. To address this issue we collected the largest action video database to-date with 51 action categories, which in total contain around 7,000 manually annotated clips extracted from a variety of sources ranging from digitized movies to YouTube. We use this database to evaluate the performance of two representative computer vision systems for action recognition and explore the robustness of these methods under various conditions such as camera motion, viewpoint, video quality and occlusion. <s> BIB001 </s> A survey of datasets for visual tracking <s> Individual action/activity <s> Highlights? Description of datasets for video-based human activity and action recognition. ? 68 datasets reported: 28 for heterogeneous and 40 for specific human actions. ? Useful data, such as web for dowloading, published works or ground truth, are provided. ? Datasets are compared and classified. Vision-based human action and activity recognition has an increasing importance among the computer vision community with applications to visual surveillance, video retrieval and human-computer interaction. In recent years, more and more datasets dedicated to human action and activity recognition have been created. The use of these datasets allows us to compare different recognition systems with the same input data. The survey introduced in this paper tries to cover the lack of a complete description of the most important public datasets for video-based human activity and action recognition and to guide researchers in the election of the most suitable dataset for benchmarking their algorithms. <s> BIB002 </s> A survey of datasets for visual tracking <s> Individual action/activity <s> Action recognition in unconstrained situations is a difficult task, suffering from massive intra-class variations. It is made even more challenging when complex 3D actions are projected down to the image plane, losing a great deal of information. The recent emergence of 3D data, both in broadcast content, and commercial depth sensors, provides the possibility to overcome this issue. This paper presents a new dataset, for benchmarking action recognition algorithms in natural environments, while making use of 3D information. The dataset contains around 650 video clips, across 14 classes. In addition, two state of the art action recognition algorithms are extended to make use of the 3D data, and five new interest point detection strategies are also proposed, that extend to the 3D data. Our evaluation compares all 4 feature descriptors, using 7 different types of interest point, over a variety of threshold levels, for the Hollywood3D dataset. We make the dataset including stereo video, estimated depth maps and all code required to reproduce the benchmark results, available to the wider community. <s> BIB003
The recognition of individual actions or activities has become a major focus of the computer vision community. Lots of datasets have thus been proposed for human action or activity recognition in video sequences (see BIB002 for a survey). These were more and more difficult to address concerning the visual tracking point of view, notably because actions or activity require long-term visual tracking of the gestures, the body and/or the head. The first datasets proposed on this topic were grayscale video sequences acquired in indoor [160] or outdoor [161] unconstrained environments (homogeneous background, people just moving their hands in a predefined way, only a small set of specific actions were addressed). Other datasets proposed scenarios for action recognition in colored outdoor environments with simple or cluttered backgrounds [162] [163] [164] , or with occlusions  . More complex situations were considered in [166] where video sequences were taken from two different Hollywood movies. Datasets on action/activity recognition thus seem to fall into one of the following two categories: either they consider only a very small set of actions or activities (such as "drinking" or "smoking"), or they correspond to real life actions/activities in daily environments. The Hollywood movies dataset clearly falls into the second one. In the HMDB51 dataset BIB001 , the number of actions increases considerably (51 actions or activities, such as "hand-waving", "drinking", "sword fighting", "diving", "running" or "kicking"), as well as the number of movies from which the dataset was constructed (around 7000 clips). The TRECVID workshop mainly focuses on video retrieval, but it also provides data sequences useful for human activity recognition [168] . Multi-modal datasets for action and activities recognition (grayscale, RGB, depth) are proposed in   BIB003 , or with motion capture data [154] . Multi-views is also considered in    , as well as both multimodal and multi-views in [175] . In all the last four cases, a fusion is necessary to include either multiple views or multiple modalities. Note that each modality has its own specificities: for example some of them are more noisy than others. Some datasets also propose video sequences in aerial views [115, 163,  . In such cases, we face the problem of similar appearance of the objects. Finally, some datasets propose to recognize specific daily life actions or activities [177], e.g., cooking [178, 179] , dancing [180] or playing sport [181] [182] [183] .