A Survey on Deep Learning Methods for Robot Vision <s> Object Detection and Categorization <s> The current state of the art solutions for object detection describe each class by a set of models trained on discovered sub-classes (so called "components"), with each model itself composed of collections of interrelated parts (deformable models). These detectors build upon the now classic Histogram of Oriented Gradients+linear SVM combo. In this paper we revisit some of the core assumptions in HOG+SVM and show that by properly designing the feature pooling, feature selection, preprocessing, and training methods, it is possible to reach top quality, at least for pedestrian detections, using a single rigid component. Abstract We provide experiments for a large design space, that give insights into the design of classifiers, as well as relevant information for practitioners. Our best detector is fully feed-forward, has a single unified architecture, uses only histograms of oriented gradients and colour information in monocular static images, and improves over 23 other methods on the INRIA, ETH and Caltech-USA datasets, reducing the average miss-rate over HOG+SVM by more than 30%. <s> BIB001 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Detection and Categorization <s> Even with the advent of more sophisticated, data-hungry methods, boosted decision trees remain extraordinarily successful for fast rigid object detection, achieving top accuracy on numerous datasets. While effective, most boosted detectors use decision trees with orthogonal (single feature) splits, and the topology of the resulting decision boundary may not be well matched to the natural topology of the data. Given highly correlated data, decision trees with oblique (multiple feature) splits can be effective. Use of oblique splits, however, comes at considerable computational expense. Inspired by recent work on discriminative decorrelation of HOG features, we instead propose an efficient feature transform that removes correlations in local neighborhoods. The result is an overcomplete but locally decorrelated representation ideally suited for use with orthogonal decision trees. In fact, orthogonal trees with our locally decorrelated features outperform oblique trees trained over the original features at a fraction of the computational cost. The overall improvement in accuracy is dramatic: on the Caltech Pedestrian Dataset, we reduce false positives nearly tenfold over the previous state-of-the-art. <s> BIB002 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Detection and Categorization <s> Multi-resolution image features may be approximated via extrapolation from nearby scales, rather than being computed explicitly. This fundamental insight allows us to design object detection algorithms that are as accurate, and considerably faster, than the state-of-the-art. The computational bottleneck of many modern detectors is the computation of features at every scale of a finely-sampled image pyramid. Our key insight is that one may compute finely sampled feature pyramids at a fraction of the cost, without sacrificing performance: for a broad family of features we find that features computed at octave-spaced scale intervals are sufficient to approximate features on a finely-sampled pyramid. Extrapolation is inexpensive as compared to direct feature computation. As a result, our approximation yields considerable speedups with negligible loss in detection accuracy. We modify three diverse visual recognition systems to use fast feature pyramids and show results on both pedestrian detection (measured on the Caltech, INRIA, TUD-Brussels and ETH data sets) and general object detection (measured on the PASCAL VOC). The approach is general and is widely applicable to vision algorithms requiring fine-grained multi-scale analysis. Our approximation is valid for images with broad spectra (most natural images) and fails for images with narrow band-pass spectra (e.g., periodic textures). <s> BIB003 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Detection and Categorization <s> Object recognition and pose estimation from RGB-D images are important tasks for manipulation robots which can be learned from examples. Creating and annotating datasets for learning is expensive, however. We address this problem with transfer learning from deep convolutional neural networks (CNN) that are pre-trained for image categorization and provide a rich, semantically meaningful feature set. We incorporate depth information, which the CNN was not trained with, by rendering objects from a canonical perspective and colorizing the depth channel according to distance from the object center. We evaluate our approach on the Washington RGB-D Objects dataset, where we find that the generated feature set naturally separates classes and instances well and retains pose manifolds. We outperform state-of-the-art on a number of subtasks and show that our approach can yield superior results when only little training data is available. <s> BIB004 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Detection and Categorization <s> The ability to visually recognize objects is a fundamental skill for robotics systems. Indeed, a large variety of tasks involving manipulation, navigation or interaction with other agents, deeply depends on the accurate understanding of the visual scene. Yet, at the time being, robots are lacking good visual perceptual systems, which often become the main bottleneck preventing the use of autonomous agents for real-world applications. ::: Lately in computer vision, systems that learn suitable visual representations and based on multi-layer deep convolutional networks are showing remarkable performance in tasks such as large-scale visual recognition and image retrieval. To this regard, it is natural to ask whether such remarkable performance would generalize also to the robotic setting. ::: In this paper we investigate such possibility, while taking further steps in developing a computational vision system to be embedded on a robotic platform, the iCub humanoid robot. In particular, we release a new dataset ({\sc iCubWorld28}) that we use as a benchmark to address the question: {\it how many objects can iCub recognize?} Our study is developed in a learning framework which reflects the typical visual experience of a humanoid robot like the iCub. Experiments shed interesting insights on the strength and weaknesses of current computer vision approaches applied in real robotic settings. <s> BIB005 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Detection and Categorization <s> Pedestrian detection is a popular research topic due to its paramount importance for a number of applications, especially in the fields of automotive, surveillance and robotics. Despite the significant improvements, pedestrian detection is still an open challenge that calls for more and more accurate algorithms. In the last few years, deep learning and in particular Convolutional Neural Networks emerged as the state of the art in terms of accuracy for a number of computer vision tasks such as image classification, object detection and segmentation, often outperforming the previous gold standards by a large margin. In this paper, we propose a pedestrian detection system based on deep learning, adapting a general-purpose convolutional network to the task at hand. By thoroughly analyzing and optimizing each step of the detection pipeline we propose an architecture that outperforms traditional methods, achieving a task accuracy close to that of state-of-the-art approaches, while requiring a low computational time. Finally, we tested the system on an NVIDIA Jetson TK1, a 192-core platform that is envisioned to be a forerunner computational brain of future self-driving cars. HighlightsA deep convolutional neural network for pedestrian detection is proposed.The proposed method improves the accuracy performance with respect to handcrafted features.Our method detects pedestrians with a good accuracy and has a reasonable computational complexity. <s> BIB006 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Detection and Categorization <s> In this paper we study the use of convolutional neural networks (convnets) for the task of pedestrian detection. Despite their recent diverse successes, convnets historically underperform compared to other pedestrian detectors. We deliberately omit explicitly modelling the problem into the network (e.g. parts or occlusion modelling) and show that we can reach competitive performance without bells and whistles. In a wide range of experiments we analyse small and big convnets, their architectural choices, parameters, and the influence of different training data, including pretraining on surrogate tasks. We present the best convnet detectors on the Caltech and KITTI dataset. On Caltech our convnets reach top performance both for the Caltech1x and Caltech10x training setup. Using additional data at training time our strongest convnet model is competitive even to detectors that use additional data (optical flow) at test time. <s> BIB007 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Detection and Categorization <s> In this paper, we explore various aspects of fusing LIDAR and color imagery for pedestrian detection in the context of convolutional neural networks (CNNs), which have recently become state-of-art for many vision problems. We incorporate LIDAR by up-sampling the point cloud to a dense depth map and then extracting three features representing different aspects of the 3D scene. We then use those features as extra image channels. Specifically, we leverage recent work on HHA [9] (horizontal disparity, height above ground, and angle) representations, adapting the code to work on up-sampled LIDAR rather than Microsoft Kinect depth maps. We show, for the first time, that such a representation is applicable to up-sampled LIDAR data, despite its sparsity. Since CNNs learn a deep hierarchy of feature representations, we then explore the question: At what level of representation should we fuse this additional information with the original RGB image channels? We use the KITTI pedestrian detection dataset for our exploration. We first replicate the finding that region-CNNs (R-CNNs) [8] can outperform the original proposal mechanism using only RGB images, but only if fine-tuning is employed. Then, we show that: 1) using HHA features and RGB images performs better than RGB-only, even without any fine-tuning using large RGB web data, 2) fusing RGB and HHA achieves the strongest results if done late, but, under a parameter or computational budget, is best done at the early to middle layers of the hierarchical representation, which tend to represent midlevel features rather than low (e.g. edges) or high (e.g. object class decision) level features, 3) some of the less successful methods have the most parameters, indicating that increased classification accuracy is not simply a function of increased capacity in the neural network. <s> BIB008 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Detection and Categorization <s> The use of identical robots in the RoboCup Standard Platform League (SPL) made software development the key aspect to achieve good results in competitions. In particular, the visual detection process is crucial for extracting information about the environment. In this paper, we present a novel approach for object detection and classification based on Convolutional Neural Networks (CNN). The approach is designed to be used by NAO robots and is made of two stages: image region segmentation, for reducing the search space, and Deep Learning, for validation. The proposed method can be easily extended to deal with different objects and adapted to be used in other RoboCup leagues. Quantitative experiments have been conducted on a data set of annotated images captured in real conditions from NAO robots in action. The used data set is made available for the community. <s> BIB009 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Detection and Categorization <s> Pedestrian detection is a well-studied problem. Even though many datasets contain challenging case studies, the performances of new methods are often only reported on cases of reasonable difficulty. In particular, the issue of small scale pedestrian detection is seldom considered. In this paper, we focus on the detection of small scale pedestrians, i.e., those that are at far distance from the camera. We show that classical features used for pedestrian detection are not well suited for our case of study. Instead, we propose a convolutional neural network based method to learn the features with an end-to-end approach. Experiments on the Caltech Pedestrian Detection Benchmark showed that we outperformed existing methods by more than 10% in terms of log-average miss rate. <s> BIB010 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Detection and Categorization <s> A unified deep neural network, denoted the multi-scale CNN (MS-CNN), is proposed for fast multi-scale object detection. The MS-CNN consists of a proposal sub-network and a detection sub-network. In the proposal sub-network, detection is performed at multiple output layers, so that receptive fields match objects of different scales. These complementary scale-specific detectors are combined to produce a strong multi-scale object detector. The unified network is learned end-to-end, by optimizing a multi-task loss. Feature upsampling by deconvolution is also explored, as an alternative to input upsampling, to reduce the memory and computation costs. State-of-the-art object detection performance, at up to 15 fps, is reported on datasets, such as KITTI and Caltech, containing a substantial number of small objects. <s> BIB011 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Detection and Categorization <s> Although the Region-based Convolutional Neural Network (R-CNN) families have shown promising results for object detection, they still face great challenges for task-specific detection, e.g., pedestrian detection, the current difficulties of which mainly lie in the large scale variations of pedestrians and insufficient discriminative power of pedestrian features. To overcome these difficulties, we propose a novel Scale-Adaptive Deconvolutional Regression (SADR) network in this paper. Specifically, the proposed network can effectively detect pedestrians of various scales by flexibly choosing which feature layer to regress object locations according to the height of pedestrians, thus improving the detection accuracy significantly. Furthermore, considering CNN can abstract different semantic-level features from different layers, we fuse features from multiple layers to provide both local characteristics and global semantic information of the object for final pedestrian classification, which improves the discriminative power of pedestrian features and boosts the detection performance further. Extensive experiments have verified the effectiveness of our proposed approach, which achieves the state-of-the-art log-average miss rate (MR) of 6.94% on the revised Caltech [1] and a competitive result on KITTI. <s> BIB012 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Detection and Categorization <s> In this paper, we investigate two new strategies to detect objects accurately and efficiently using deep convolutional neural network: 1) scale-dependent pooling and 2) layerwise cascaded rejection classifiers. The scale-dependent pooling (SDP) improves detection accuracy by exploiting appropriate convolutional features depending on the scale of candidate object proposals. The cascaded rejection classifiers (CRC) effectively utilize convolutional features and eliminate negative object proposals in a cascaded manner, which greatly speeds up the detection while maintaining high accuracy. In combination of the two, our method achieves significantly better accuracy compared to other state-of-the-arts in three challenging datasets, PASCAL object detection challenge, KITTI object detection benchmark and newly collected Inner-city dataset, while being more efficient. <s> BIB013 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Detection and Categorization <s> In RoboCup soccer, ball localization is an important and challenging task, especially since the last change of the rule which allows 50% of the ball’s surface to be of any color or pattern while the rest must remain white. Multi-color balls have changing color histograms and patterns in dependence of the current orientation and movement. This paper presents a neural approach using a convolutional neural network (CNN) to localize the ball in various scenes. CNNs were used in several image recognition tasks, particularly because of their capability to learn invariances in images. In this work we use CNNs to locate a ball by training two output layers, representing the x- and y-coordinates, with normal distributions fitted around the ball. Therefore the network not only locates the ball’s position but also provides an estimation of the noise. The architecture processes the whole image in full size, no sliding-window approach is used. <s> BIB014 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Detection and Categorization <s> State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet [1] and Fast R-CNN [2] have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features—using the recently popular terminology of neural networks with ’attention’ mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model [3] , our detection system has a frame rate of 5 fps ( including all steps ) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available. <s> BIB015 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Detection and Categorization <s> Most of the recent successful methods in accurate object detection and localization used some variants of R-CNN style two stage Convolutional Neural Networks (CNN) where plausible regions were proposed in the first stage then followed by a second stage for decision refinement. Despite the simplicity of training and the efficiency in deployment, the single stage detection methods have not been as competitive when evaluated in benchmarks consider mAP for high IoU thresholds. In this paper, we proposed a novel single stage end-to-end trainable object detection network to overcome this limitation. We achieved this by introducing Recurrent Rolling Convolution (RRC) architecture over multi-scale feature maps to construct object classifiers and bounding box regressors which are deep in context. We evaluated our method in the challenging KITTI dataset which measures methods under IoU threshold of 0.7. We showed that with RRC, a single reduced VGG-16 based model already significantly outperformed all the previously published results. At the time this paper was written our models ranked the first in KITTI car detection (the hard level), the first in cyclist detection and the second in pedestrian detection. These results were not reached by the previous single stage methods. The code is publicly available. <s> BIB016 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Detection and Categorization <s> In Convolutional Neural Network (CNN)-based object detection methods, region proposal becomes a bottleneck when objects exhibit significant scale variation, occlusion or truncation. In addition, these methods mainly focus on 2D object detection and cannot estimate detailed properties of objects. In this paper, we propose subcategory-aware CNNs for object detection. We introduce a novel region proposal network that uses subcategory information to guide the proposal generating process, and a new detection network for joint detection and subcategory classification. By using subcategories related to object pose, we achieve state of-the-art performance on both detection and pose estimation on commonly used benchmarks. <s> BIB017 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Detection and Categorization <s> We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0%, respectively, which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully connected layers we employed a recently developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry. <s> BIB018 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Detection and Categorization <s> The main goal of this paper is to analyze the general problem of using Convolutional Neural Networks (CNNs) in robots with limited computational capabilities, and to propose general design guidelines for their use. In addition, two different CNN based NAO robot detectors that are able to run in real-time while playing soccer are proposed. One of the detectors is based on the XNOR-Net and the other on the SqueezeNet. Each detector is able to process a robot object-proposal in ~1ms, with an average number of 1.5 proposals per frame obtained by the upper camera of the NAO. The obtained detection rate is ~97%. <s> BIB019
Object detection and categorization is a fundamental ability in robotics. It enables a robot to execute tasks that require interaction with object instances in the real-world. Deep learning is already being used for general-purpose object detection and categorization, as well as for pedestrian detection, and for detecting objects in robotics soccer. State-of-the-art methods used for object detection and categorization are based on generating object proposals, and then classifying them using a DNN, enabling systems to detect thousands of different object categories. As will be shown, one of the main challenges for the application of DNNs for object detection and characterization in robotics is real-time operation. It must be stressed that obtaining the required object proposals for feeding the DNNs is not real-time in the general case, and that, on the other hand, general-purpose object detection and categorization DNN based methods are not able to run in real-time in most robotics platforms. These challenges are addressed by using task-dependent methods for generating few, fast, and high quality proposals for a limited number of possible object categories. These methods are based on using other information sources for segmenting the objects (depth information, motion, color, etc.), and/or by using object specific, non general-purpose, weak detectors, for generating the required proposals. Also, smaller DNN architectures can be used when dealing with a limited number of object categories. It is worth mentioning that most of the reported studies do not indicate the frame rate needed for full object detection/categorization, or they show frame rates that are far from being real-time. In generic object detection methods, computation of proposals using methods like Selective Search or EdgeBoxes takes most of the time BIB015 . Systems like Faster R-CNN that compute proposals using CNNs are able to obtain higher frame rates, but require the use of high-end GPUs for working. The use of task-specific knowledge based detectors on depth BIB004 [34] BIB008 , motion BIB005 , color segmentation BIB009 , or weak object detectors BIB006 can be useful for generating fewer, faster proposals, which is the key for achieving high frame rates on CPU-based systems. Finally, methods based on FCNs cannot achieve high frame rates on robotic platforms with low processing capabilities (no GPU available) because they process images with larger resolutions than normal CNNs. Then, FCNs cannot be used trivially for real time robotics on these kinds of platforms. First, we will analyze the use of complementary information sources for segmenting the objects. proposals by using tabletop segmentation or edge/gradient information. Also, they generate colorized images from depth for use in a standard CNN pipeline for object recognition. These studies use CNNs and custom depth-based proposals, and then their speed is limited by the CNN model. For instance, in  , a system is described that runs at 405 fps on a Titan X GPU, 67 fps on a Tegra X1 GPU, and 62 fps on a Core i7 6700K CPU  . It must be noted that, while TitanX GPU and Core i7 CPU processors are designed for desktop computers, Tegra X1 is a mobile GPU processor for embedded systems aimed at low power consumption, and so it can be used on robotic platforms. Second, we will present two applications that use specific detectors for generating the object proposals: pedestrian detection, and detection of objects in robotics soccer. Detecting pedestrians in real time with high reliability is an important ability in many robotics applications, especially for autonomous driving. Large differences in illumination and variable, cluttered backgrounds outdoors are hard issues to be addressed. Methods that use CNN-based methods for pedestrian detection include BIB007 , BIB001 , BIB006 , BIB010 , BIB016 , BIB011 , BIB017 , BIB012 , BIB013 . Person detectors, such as LDCF BIB002 or ACF BIB003 , are used for generating pedestrian proposals in some methods. As indicated in BIB006 , the system based on AlexNet BIB018 requires only 3 ms for processing a region, and proposal computation runs at 2 fps when using LDCF, and at 21.7 fps when using ACF in an NVIDIA GTX980 GPU, with images of size 640x480. The lightweight pedestrian detector is able to run at 2.4 fps in an NVIDIA Jetson TK1, and beats classical methods by a large margin BIB006 . A second approach is to use a FCN for detecting parts of persons  . This method is tested on a NVIDIA Titan X GPU, and is able to run at 4 fps when processing images of size 300x300. KITTY pedestrians is a benchmark used by state-of-the-art methods. Current leading methods whose algorithms are described (many of the best performing methods do not describe their algorithms) are shown in Table 7 . Note that, as the objective of the benchmark is to evaluate accuracy, most of the methods cannot run in real-time (~30 fps). This is an aspect that needs further research. BIB019 , and ball detection BIB014 . State-of-the-art soccer player detection BIB019 uses a model based on SqueezeNet using only 2 convolutional layers, an extended fire layer and a fully-connected layer, while processing regions of size 28x28. This method requires only 1 ms on an Nao robot for processing a region, using only an Intel Atom CPU. Another application is ball detection, which is addressed in BIB014 . This method uses an architecture based on 3 conv layers and 2 fully-connected layers. However, its runtime inside a Nao robot is not reported. Currently, CNNs have been used for solving very specific tasks in robotics soccer successfully, and it is expected that new algorithms (and hardware) will enable the use of deep learning in more global, high-level tasks.