Object detection in videos is an important task in computer vision for various applications such as object tracking, video summarization and video search. Although great progress has been made in improving the accuracy of object detection in recent years due to the rise of deep neural networks, the state-of-the-art algorithms are highly computationally intensive. In order to address this challenge, we make two important observations in the context of videos: (i) Objects often occupy only a small fraction of the area in each video frame, and (ii) There is a high likelihood of strong temporal correlation between consecutive frames. Based on these observations, we propose Pack and Detect (PaD), an approach to reduce the computational requirements of object detection in videos. In PaD, only selected video frames called anchor frames are processed at full size. In the frames that lie between anchor frames (inter-anchor frames), regions of interest (ROIs) are identified based on the detections in the previous frame. We propose an algorithm to pack the ROIs of each inter-anchor frame together into a reduced-size frame. The computational requirements of the detector are reduced due to the lower size of the input. In order to maintain the accuracy of object detection, the proposed algorithm expands the ROIs greedily to provide additional background around each object to the detector. PaD can use any underlying neural network architecture to process the full-size and reduced-size frames. Experiments using the ImageNet video object detection dataset indicate that PaD can potentially reduce the number of FLOPS required for a frame by 4×. This leads to an overall increase in throughput of 1.25× on a 2.1 GHz Intel Xeon server with a NVIDIA Titan X GPU at the cost of 1.1 drop in accuracy.
* Object Detection in Videos The temporal redundancy present in videos has been exploited before to improve the accuracy and speed of object detection. In @cite_33 @cite_23 @cite_45 @cite_6 , the aggregation of information from neighbouring frames is done at the bounding box level to improve accuracy. In @cite_33 , per-frame object detection is combined with multi-context suppression, motion-guided propagation and object tracking to improve detection accuracy. In @cite_45 @cite_6 , non-maximum suppression is done over bags of frames. In @cite_2 @cite_39 @cite_46 @cite_20 , integration of the CNN features across neighbouring frames is used to improve accuracy. In @cite_2 , a CNN is combined with a Long Short Term Memory (LSTM) to obtain temporal features for object detection. In @cite_39 @cite_38 @cite_20 , the features from neighbouring frames are aggregated together using optical flow information to improve feature quality. These methods @cite_33 @cite_23 @cite_45 @cite_6 @cite_2 pose large computation requirements, making them often unsuitable for real-time processing.
@cite_38 @cite_33 @cite_6 @cite_39 @cite_45 @cite_23 @cite_2 @cite_46 @cite_20
2964286567 2604445072 2777578098 2794682959
Extending state-of-the-art object detectors from image to video is challenging. The accuracy of detection suffers from degenerated object appearances in videos, e.g., motion blur, video defocus, rare poses, etc. Existing work attempts to exploit temporal information on box level, but such methods are not trained end-to-end. We present flow-guided feature aggregation, an accurate and end-to-end learning framework for video object detection. It leverages temporal coherence on feature level instead. It improves the per-frame features by aggregation of nearby features along the motion paths, and thus improves the video recognition accuracy. Our method significantly improves upon strong singleframe baselines in ImageNet VID [33], especially for more challenging fast moving objects. Our framework is principled, and on par with the best engineered systems winning the ImageNet VID challenges 2016, without additional bells-and-whistles. The code would be released. Extending state-of-the-art object detectors from image to video is challenging. The accuracy of detection suffers from degenerated object appearances in videos, e.g., motion blur, video defocus, rare poses, etc. Existing work attempts to exploit temporal information on box level, but such methods are not trained end-to-end. We present flow-guided feature aggregation, an accurate and end-to-end learning framework for video object detection. It leverages temporal coherence on feature level instead. It improves the per-frame features by aggregation of nearby features along the motion paths, and thus improves the video recognition accuracy. Our method significantly improves upon strong single-frame baselines in ImageNet VID, especially for more challenging fast moving objects. Our framework is principled, and on par with the best engineered systems winning the ImageNet VID challenges 2016, without additional bells-and-whistles. The proposed method, together with Deep Feature Flow, powered the winning entry of ImageNet VID challenges 2017. The code is available at this https URL. Video object detection is more challenging compared to image object detection. Previous works proved that applying object detector frame by frame is not only slow but also inaccurate. Visual clues get weakened by defocus and motion blur, causing failure on corresponding frames. Multi-frame feature fusion methods proved effective in improving the accuracy, but they dramatically sacrifice the speed. Feature propagation based methods proved effective in improving the speed, but they sacrifice the accuracy. So is it possible to improve speed and performance simultaneously? Inspired by how human utilize impression to recognize objects from blurry frames, we propose Impression Network that embodies a natural and efficient feature aggregation mechanism. In our framework, an impression feature is established by iteratively absorbing sparsely extracted frame features. The impression feature is propagated all the way down the video, helping enhance features of low-quality frames. This impression mechanism makes it possible to perform long-range multi-frame feature fusion among sparse keyframes with minimal overhead. It significantly improves per-frame detection baseline on ImageNet VID while being 3 times faster (20 fps). We hope Impression Network can provide a new perspective on video feature enhancement. Code will be made available. Temporal object detection has attracted significant attention, but most popular detection methods can not leverage the rich temporal information in video or robotic vision. Although many different algorithms have been developed for video detection task, real-time online approaches are frequently deficient. In this paper, based on attention mechanism and convolutional long short-term memory (ConvLSTM), we propose a temporal single-shot detector (TSSD) for robotic vision. Distinct from previous methods, we take aim at temporally integrating pyramidal feature hierarchy using ConvLSTM, and design a novel structure including a high-level temporal unit as well as a low-level one (HL-TU) for multi-scale feature maps. Moreover, we develop a creative temporal analysis unit, namely, attention-aware ConvLSTM (AC-LSTM), in which a temporal attention module is specially tailored for background suppression and scale suppression while ConvLSTM temporally integrates attention-aware features. An association loss is designed for temporal coherence. Finally, our method is evaluated on ImageNet VID dataset. Extensive comparisons on the detection capability confirm or validate the superiority of the proposed approach. Consequently, the developed TSSD is fairly faster and achieves an overall competitive performance in terms of mean average precision. As a temporal, real-time, and online detector, TSSD is applicable to robot's intelligent perception.