A Survey on Deep Learning Methods for Robot Vision <s> Spatiotemporal Vision <s> Human action recognition is an important yet challenging task. Human actions usually involve human-object interactions, highly articulated motions, high intra-class variations, and complicated temporal structures. The recently developed commodity depth sensors open up new possibilities of dealing with this problem by providing 3D depth data of the scene. This information not only facilitates a rather powerful human motion capturing technique, but also makes it possible to efficiently model human-object interactions and intra-class variations. In this paper, we propose to characterize the human actions with a novel actionlet ensemble model, which represents the interaction of a subset of human joints. The proposed model is robust to noise, invariant to translational and temporal misalignment, and capable of characterizing both the human motion and the human-object interactions. We evaluate the proposed approach on three challenging action recognition datasets captured by Kinect devices, a multiview action recognition dataset captured with Kinect device, and a dataset captured by a motion capture system. The experimental evaluations show that the proposed approach achieves superior performance to the state-of-the-art algorithms. <s> BIB001 </s> A Survey on Deep Learning Methods for Robot Vision <s> Spatiotemporal Vision <s> Deep neural networks have become the primary learning technique for object recognition. Videos, unlike still images, are temporally coherent which makes the application of deep networks non-trivial. Here, we investigate how motion can aid object recognition in short videos. Our approach is based on Long Short-Term Memory (LSTM) deep networks. Unlike previous applications of LSTMs, we implement each gate as a convolution. We show that convolutional-based LSTM models are capable of learning motion dependencies and are able to improve the recognition accuracy when more frames in a sequence are available. We evaluate our approach on the Washington RGBD Object dataset and on the Washington RGBD Scenes dataset. Our approach outperforms deep nets applied to still images and sets a new state-of-the-art in this domain. <s> BIB002 </s> A Survey on Deep Learning Methods for Robot Vision <s> Spatiotemporal Vision <s> Convolutional neural networks (CNNs) have been extensively applied for image recognition problems giving state-of-the-art results on recognition, detection, segmentation and retrieval. In this work we propose and evaluate several deep neural network architectures to combine image information across a video over longer time periods than previously attempted. We propose two methods capable of handling full length videos. The first method explores various convolutional temporal feature pooling architectures, examining the various design choices which need to be made when adapting a CNN for this task. The second proposed method explicitly models the video as an ordered sequence of frames. For this purpose we employ a recurrent neural network that uses Long Short-Term Memory (LSTM) cells which are connected to the output of the underlying CNN. Our best networks exhibit significant performance improvements over previously published results on the Sports 1 million dataset (73.1% vs. 60.9%) and the UCF-101 datasets with (88.6% vs. 88.0%) and without additional optical flow information (82.6% vs. 73.0%). <s> BIB003 </s> A Survey on Deep Learning Methods for Robot Vision <s> Spatiotemporal Vision <s> Recurrent Neural Networks (RNNs) have had considerable success in classifying and predicting sequences. We demonstrate that RNNs can be effectively used in order to encode sequences and provide effective representations. The methodology we use is based on Fisher Vectors, where the RNNs are the generative probabilistic models and the partial derivatives are computed using backpropagation. State of the art results are obtained in two central but distant tasks, which both rely on sequences: video action recognition and image annotation. We also show a surprising transfer learning result from the task of image annotation to the task of video action recognition. <s> BIB004 </s> A Survey on Deep Learning Methods for Robot Vision <s> Spatiotemporal Vision <s> We propose a simple, yet effective approach for spatiotemporal feature learning using deep 3-dimensional convolutional networks (3D ConvNets) trained on a large scale supervised video dataset. Our findings are three-fold: 1) 3D ConvNets are more suitable for spatiotemporal feature learning compared to 2D ConvNets, 2) A homogeneous architecture with small 3x3x3 convolution kernels in all layers is among the best performing architectures for 3D ConvNets, and 3) Our learned features, namely C3D (Convolutional 3D), with a simple linear classifier outperform state-of-the-art methods on 4 different benchmarks and are comparable with current best methods on the other 2 benchmarks. In addition, the features are compact: achieving 52.8% accuracy on UCF101 dataset with only 10 dimensions and also very efficient to compute due to the fast inference of ConvNets. Finally, they are conceptually very simple and easy to train and use. <s> BIB005 </s> A Survey on Deep Learning Methods for Robot Vision <s> Spatiotemporal Vision <s> Deep learning is a form of machine learning that enables computers to learn from experience and understand the world in terms of a hierarchy of concepts. Because the computer gathers knowledge from experience, there is no need for a human computer operator to formally specify all the knowledge that the computer needs. The hierarchy of concepts allows the computer to learn complicated concepts by building them out of simpler ones; a graph of these hierarchies would be many layers deep. This book introduces a broad range of topics in deep learning. The text offers mathematical and conceptual background, covering relevant concepts in linear algebra, probability theory and information theory, numerical computation, and machine learning. It describes deep learning techniques used by practitioners in industry, including deep feedforward networks, regularization, optimization algorithms, convolutional networks, sequence modeling, and practical methodology; and it surveys such applications as natural language processing, speech recognition, computer vision, online recommendation systems, bioinformatics, and videogames. Finally, the book offers research perspectives, covering such theoretical topics as linear factor models, autoencoders, representation learning, structured probabilistic models, Monte Carlo methods, the partition function, approximate inference, and deep generative models. Deep Learning can be used by undergraduate or graduate students planning careers in either industry or research, and by software engineers who want to begin using deep learning in their products or platforms. A website offers supplementary material for both readers and instructors. <s> BIB006 </s> A Survey on Deep Learning Methods for Robot Vision <s> Spatiotemporal Vision <s> This paper studies deep network architectures to address the problem of video classification. A multi-stream framework is proposed to fully utilize the rich multimodal information in videos. Specifically, we first train three Convolutional Neural Networks to model spatial, short-term motion and audio clues respectively. Long Short Term Memory networks are then adopted to explore long-term temporal dynamics. With the outputs of the individual streams on multiple classes, we propose to mine class relationships hidden in the data from the trained models. The automatically discovered relationships are then leveraged in the multi-stream multi-class fusion process as a prior, indicating which and how much information is needed from the remaining classes, to adaptively determine the optimal fusion weights for generating the final scores of each class. Our contributions are two-fold. First, the multi-stream framework is able to exploit multimodal features that are more comprehensive than those previously attempted. Second, our proposed fusion method not only learns the best weights of the multiple network streams for each class, but also takes class relationship into account, which is known as a helpful clue in multi-class visual classification tasks. Our framework produces significantly better results than the state of the arts on two popular benchmarks, 92.2% on UCF-101 (without using audio) and 84.9% on Columbia Consumer Videos. <s> BIB007 </s> A Survey on Deep Learning Methods for Robot Vision <s> Spatiotemporal Vision <s> Hand-crafted feature functions are usually designed based on the domain knowledge of a presumably controlled environment and often fail to generalize, as the statistics of real-world data cannot always be modeled correctly. Data-driven feature learning methods, on the other hand, have emerged as an alternative that often generalize better in uncontrolled environments. We present a simple, yet robust, 2-D convolutional neural network extended to a concatenated 3-D network that learns to extract features from the spatio-temporal domain of raw video data. The resulting network model is used for content-based recognition of videos. Relying on a 2-D convolutional neural network allows us to exploit a pretrained network as a descriptor that yielded the best results on the largest and challenging ILSVRC-2014 dataset. Experimental results on commonly used benchmarking video datasets demonstrate that our results are state-of-the-art in terms of accuracy and computational time without requiring any preprocessing (e.g., optic flow) or a priori knowledge on data capture (e.g., camera motion estimation), which makes it more general and flexible than other approaches. Our implementation is made available. <s> BIB008 </s> A Survey on Deep Learning Methods for Robot Vision <s> Spatiotemporal Vision <s> Temporal alignment is an important preprocessing procedure for human action recognition. The challenge of temporal alignment problem is the temporal scale difference between human actions as well as the variability of each subject. Metric learning is the central problem of temporal alignment. This paper presents a nonlinear time alignment method with deep autoencoder. The spatio-temporal features obtained from the neural network contain the metric information for feature comparison. The effectiveness of our method is verified with k-nearest neighbor (k-NN) classifier on MSR-Action 3D and MSR-Daily Activity 3D datasets. Experimental results illustrate that the proposed method achieves superior performance to other metric based techniques. <s> BIB009 </s> A Survey on Deep Learning Methods for Robot Vision <s> Spatiotemporal Vision <s> Anticipating the future actions of a human is a widely studied problem in robotics that requires spatio-temporal reasoning. In this work we propose a deep learning approach for anticipation in sensory-rich robotics applications. We introduce a sensory-fusion architecture which jointly learns to anticipate and fuse information from multiple sensory streams. Our architecture consists of Recurrent Neural Networks (RNNs) that use Long Short-Term Memory (LSTM) units to capture long temporal dependencies. We train our architecture in a sequence-to-sequence prediction manner, and it explicitly learns to predict the future given only a partial temporal context. We further introduce a novel loss layer for anticipation which prevents over-fitting and encourages early anticipation. We use our architecture to anticipate driving maneuvers several seconds before they happen on a natural driving data set of 1180 miles. The context for maneuver anticipation comes from multiple sensors installed on the vehicle. Our approach shows significant improvement over the state-of-the-art in maneuver anticipation by increasing the precision from 77.4% to 90.5% and recall from 71.2% to 87.4%. <s> BIB010 </s> A Survey on Deep Learning Methods for Robot Vision <s> Spatiotemporal Vision <s> The growth of robot-assisted minimally invasive surgery has led to sizable datasets of fixed-camera video and kinematic recordings of surgical subtasks. Segmentation of these trajectories into locally-similar contiguous sections can facilitate learning from demonstrations, skill assessment, and salvaging good segments from otherwise inconsistent demonstrations. Manual, or supervised, segmentation can be prone to error and impractical for large datasets. We present Transition State Clustering with Deep Learning (TSC-DL), a new unsupervised algorithm that leverages video and kinematic data for task-level segmentation, and finds regions of the visual feature space that correlate with transition events using features constructed from layers of pre-trained image classification Deep Convolutional Neural Networks (CNNs). We report results on three datasets comparing Deep Learning architectures (AlexNet and VGG), choice of convolutional layer, dimensionality reduction techniques, visual encoding, and the use of Scale Invariant Feature Transforms (SIFT). We find that the deep architectures extract features that result in up-to a 30.4% improvement in Silhouette Score (a measure of cluster tightness) over the traditional “shallow” features from SIFT. We also present cases where TSC-DL discovers human annotator omissions. Supplementary material, data and code is available at: http://berkeleyautomation.github.io/tsc-dl/ <s> BIB011 </s> A Survey on Deep Learning Methods for Robot Vision <s> Spatiotemporal Vision <s> Abstract Effective spatial-temporal representation of motion information is crucial to human action classification. In spite of the attempt of most existing methods capturing spatial-temporal structure and learning motion representations with deep neural networks, such representations are failing to model action at their full temporal extent. To address this problem, this paper proposes a global motion representation by using sequential low-rank tensor decomposition. Specifically, we model an action sequence as a third-order tensor with spatiotemporal structure. Then, by using low-rank tensor decomposition, partial motion of objects in global context were preserved which will be feeding into deep architecture to automatically learning global-term motion features. To simultaneously exploit static spatial features, short-term motion and global-term motion in the video, we describe a multi-stream framework with recurrent convolutional architectures which is end-to-end trainable. Gated Recurrent Unit (GRU) is used as our recurrent unit which have fewer parameters than Long Short-Term Memory (LSTM). Extensive experiments were conducted on two challenging dataset: HMDB51 and UCF101. Experimental results show that our method achieves state-of-the-art performance on the HMDB51 dataset, and is comparable to the state-of-the-art methods on the UCF101 dataset. <s> BIB012
The processing of video and other spatiotemporal sources of information is of paramount importance in robotics. Classical methods related to spatiotemporal vision require engineering subtasks such as motion detection, optical flow, local feature extraction, or image segmentation. Deep learning has the potential to change the current paradigm of spatiotemporal vision from hand-engineering complex subtasks to solving tasks as a whole. The increment in both video datasets and computational power will enable the analysis of spatiotemporal data at near real-time, but only in the future. Then, spatiotemporal vision will be a goal of future robot vision as well as computer vision research. The use of short video sequences for object recognition is an alternative for improving current image-based techniques. In BIB002 , a convolutional LSTM is proposed for processing short video sequences, those containing about four frames, for robotic perception. The system is shown to improve on the baseline (smoothing CNN results from individual image frames), and is able to run at 0.87 fps when using a GPU. Human action recognition is an area that can benefit greatly from using video sequences instead of isolated images. Reports of research dealing with recognition of human actions include BIB001 , BIB012 , BIB003 , BIB007 , BIB004 , BIB005 , BIB008 , BIB009 , and BIB010 , in which driver activity anticipation is performed. In various studies, information from multiple frames is integrated by using several CNNs for each frame, by using LSTM or by using Dynamic Time Warping. The C3D method BIB005 is able to run at 313 fps on a K40 GPU, processing videos with resolution 320x240. Thus, those kinds of methods have the potential for running in real time. However, current state-of-the-art methods are not real time, or their runtimes are not reported. The best performing methods on the UCF-101 dataset are reported in Table 11 . Note that both the accuracy and runtime of C3D BIB005 are lower than Wu et al. BIB007 , and so a tradeoff between accuracy and runtime is present. For selecting a method, it must be considered if real time is needed in the specific task to be solved. Transfer of video-based computer vision techniques to medical robotics is an active area of research, especially in surgeryrelated applications. The use of deep learning in this application area has already begun. For instance, in BIB011 , a system that is able to segment video and kinematic data for performing unsupervised trajectory segmentation of multi-modal surgical demonstrations is implemented. The system is able to segment video-kinematic descriptions of surgical demonstrations successfully corresponding to stages such as "position," "push needle," "pull needle," and "hand-off." The system is based on a switching linear dynamical system which considers both continuous and discrete states composed of kinematic and visual features. Visual features are extracted from the fifth convolutional layer of a VGG CNN and then reduced to 100 dimensions by using PCA. Clustering on the state space is applied by the system and it is able to learn transitions between clusters, enabling trajectory segmentation. The system uses a video source with 10 frames per second; however, the frame rate of the full method is not reported. Spatiotemporal vision applications need real-time processing capability to be useful in robotics. However, current investigations using deep learning are experimental, and the frame rates for most of the methods are not reported. Platforms with low computing capability (only CPU) are not able to run most of the methods at a useful frame rate. Availability of specialized hardware for computing CNNs BIB006 would enable the use of deep spatiotemporal vision applications in the future.