In this paper, we address the challenging problem of effi- cient temporal activity detection in untrimmed long videos. While most recent work has focused and advanced the de- tection accuracy, the inference time can take seconds to minutes in processing one video, which is computationally prohibitive for many applications with tight runtime con- straints. This motivates our proposed budget-aware frame- work, which learns to perform activity detection by intel- ligently selecting a small subset of frames according to a specified time or computational budget. We formulate this problem as a Markov decision process, and adopt a recurrent network to model a policy for the frame selec- tion. To train the network, we employ a recurrent policy gradient approach to approximate the gradient of the non- decomposable and non-differentiable objective defined in our problem. In the extensive experiments on two bench- mark datasets, we achieve competitive detection accuracy, and more importantly, our approach is able to substantially reduce computational time and detect multiple activities in only 348ms for each untrimmed long video of THUMOS14 and ActivityNet.
A majority of the existing approaches @cite_15 @cite_31 @cite_1 @cite_11 @cite_24 for temporal activity detection focus on extracting various features to represent sliding windows and subsequently classifying them with SVMs trained on the multiple features. Alternative proposal based methods can be used to generate action segments to replace the exhaustive sliding windows. A sparse learning based framework is presented in @cite_0 to produce segment proposals with high recall. @cite_13 introduce a proposal algorithm based on C3D and LSTM. @cite_27 propose a 3D-CNN based multi-stage framework with three different networks for proposal, classification and localization. A convolutional-de-convolutional network is further introduced in @cite_6 to boost the precision for temporal boundaries of proposal segments. These proposal based detection methods are by nature stage-wised, and therefore are not end-to-end trainable. R-C3D is recently developed in @cite_8 to save computation through sharing convolutional features between proposal and classification pipelines.
@cite_31 @cite_8 @cite_1 @cite_6 @cite_24 @cite_0 @cite_27 @cite_15 @cite_13 @cite_11
1594098193 2963247196 2773012335 2589615404
Current high-quality object detection approaches use the scheme of salience-based object proposal methods followed by post-classification using deep convolutional features. This spurred recent research in improving object proposal methods. However, domain agnostic proposal generation has the principal drawback that the proposals come unranked or with very weak ranking, making it hard to trade-off quality for running time. This raises the more fundamental question of whether high-quality proposal generation requires careful engineering or can be derived just from data alone. We demonstrate that learning-based proposal methods can effectively match the performance of hand-engineered methods while allowing for very efficient runtime-quality trade-offs. Using the multi-scale convolutional MultiBox (MSC-MultiBox) approach, we substantially advance the state-of-the-art on the ILSVRC 2014 detection challenge data set, with @math mAP for a single model and @math mAP for an ensemble of two models. MSC-Multibox significantly improves the proposal quality over its predecessor MultiBox method: AP increases from @math to @math for the ILSVRC detection challenge. Finally, we demonstrate improved bounding-box recall compared to Multiscale Combinatorial Grouping with less proposals on the Microsoft-COCO data set. We address the problem of activity detection in continuous, untrimmed video streams. This is a difficult task that requires extracting meaningful spatio-temporal features to capture activities, accurately localizing the start and end times of each activity. We introduce a new model, Region Convolutional 3D Network (R-C3D), which encodes the video streams using a three-dimensional fully convolutional network, then generates candidate temporal regions containing activities, and finally classifies selected regions into specific activities. Computation is saved due to the sharing of convolutional features between the proposal and the classification pipelines. The entire model is trained end-to-end with jointly optimized localization and classification losses. R-C3D is faster than existing methods (569 frames per second on a single Titan X Maxwell GPU) and achieves state-of-the-art results on THUMOS’14. We further demonstrate that our model is a general activity detection framework that does not rely on assumptions about particular dataset properties by evaluating our approach on ActivityNet and Charades. Our code is available at http: ai.bu.edu r-c3d In this paper, we focus on improving the proposal classification stage in the object detection task and present implicit negative sub-categorization and sink diversion to lift the performance by strengthening loss function in this stage. First, based on the observation that the “background” class is generally very diverse and thus challenging to be handled as a single indiscriminative class in existing state-of-the-art methods, we propose to divide the background category into multiple implicit sub-categories to explicitly differentiate diverse patterns within it. Second, since the ground truth class inevitably has low-value probability scores for certain images, we propose to add a “sink” class and divert the probabilities of wrong classes to this class when necessary, such that the ground truth label will still have a higher probability than other wrong classes even though it has low probability output. Additionally, we propose to use dilated convolution, which is widely used in the semantic segmentation task, for efficient and valuable context information extraction. Extensive experiments on PASCAL VOC 2007 and 2012 data sets show that our proposed methods based on faster R-CNN implementation can achieve state-of-the-art mAPs, i.e., 84.1 , 82.6 , respectively, and obtain 2.5 improvement on ILSVRC DET compared with that of ResNet. Object proposals have recently emerged as an essential cornerstone for object detection. The current state-of-the-art object detectors employ object proposals to detect objects within a modest set of candidate bounding box proposals instead of exhaustively searching across an image using the sliding window approach. However, achieving high recall and good localization with few proposals is still a challenging problem. The challenge becomes even more difficult in the context of autonomous driving, in which small objects, occlusion, shadows, and reflections usually occur. In this paper, we present a robust object proposals re-ranking algorithm that effectivity re-ranks candidates generated from a customized class-independent 3DOP (3D Object Proposals) method using a two-stream convolutional neural network (CNN). The goal is to ensure that those proposals that accurately cover the desired objects are amongst the few top-ranked candidates. The proposed algorithm, which we call DeepStereoOP, exploits not only RGB images as in the conventional CNN architecture, but also depth features including disparity map and distance to the ground. Experiments show that the proposed algorithm outperforms all existing object proposal algorithms on the challenging KITTI benchmark in terms of both recall and localization. Furthermore, the combination of DeepStereoOP and Fast R-CNN achieves one of the best detection results of all three KITTI object classes. HighlightsWe present a robust object proposals re-ranking algorithm for object detection in autonomous driving.Both RGB images and depth features are included in the proposed two-stream CNN architecture called DeepStereoOP.Initial object proposals are generated from a customized class-independent 3DOP method.Experiments show that the proposed algorithm outperforms all existing object proposals algorithms.The combination of DeepStereoOP and Fast R-CNN achieves one of the best detection results on KITTI benchmark.