A Survey on Deep Learning Methods for Robot Vision <s> Object Grasping and Manipulation <s> We consider the problem of detecting robotic grasps in an RGB-D view of a scene containing objects. In this work, we apply a deep learning approach to solve this problem, which avoids time-consuming hand-design of features. This presents two main challenges. First, we need to evaluate a huge number of candidate grasps. In order to make detection fast and robust, we present a two-step cascaded system with two deep networks, where the top detections from the first are re-evaluated by the second. The first network has fewer features, is faster to run, and can effectively prune out unlikely candidate grasps. The second, with more features, is slower but has to run only on the top few detections. Second, we need to handle multimodal inputs effectively, for which we present a method that applies structured regularization on the weights based on multimodal group regularization. We show that our method improves performance on an RGBD robotic grasping dataset, and can be used to successfully execute grasps on two different robotic platforms. <s> BIB001 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Grasping and Manipulation <s> Grasp an object from a stack of objects in real-time is still a challenge in robotics. This requires the robot to have the ability of both fast object discovery and grasp detection: a target object should be picked out from the stack first and then a proper grasp configuration is applied to grasp the object. In this paper, we propose a shared convolutional neural network (CNN) which can simultaneously implement these two tasks in real-time. The processing speed of the model is about 100 frames per second on a GPU which largely satisfies the requirement. Meanwhile, we also establish a labeled RGBD dataset which contains scenes of stacked objects for robotic grasping. At last, we demonstrate the implementation of our shared CNN model on a real robotic platform and show that the robot can accurately discover a target object from the stack and successfully grasp it. <s> BIB002 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Grasping and Manipulation <s> Current model free learning-based robot grasping approaches exploit human-labeled datasets for training the models. However, there are two problems with such a methodology: (a) since each object can be grasped in multiple ways, manually labeling grasp locations is not a trivial task; (b) human labeling is biased by semantics. While there have been attempts to train robots using trial-and-error experiments, the amount of data used in such experiments remains substantially low and hence makes the learner prone to over-fitting. In this paper, we take the leap of increasing the available training data to 40 times more than prior work, leading to a dataset size of 50K data points collected over 700 hours of robot grasping attempts. This allows us to train a Convolutional Neural Network (CNN) for the task of predicting grasp locations without severe overfitting. In our formulation, we recast the regression problem to an 18-way binary classification over image patches. We also present a multi-stage learning approach where a CNN trained in one stage is used to collect hard negatives in subsequent stages. Our experiments clearly show the benefit of using large-scale datasets (and multi-stage training) for the task of grasping. We also compare to several baselines and show state-of-the-art performance on generalization to unseen objects for grasping. <s> BIB003 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Grasping and Manipulation <s> Reinforcement learning provides a powerful and flexible framework for automated acquisition of robotic motion skills. However, applying reinforcement learning requires a sufficiently detailed representation of the state, including the configuration of task-relevant objects. We present an approach that automates state-space construction by learning a state representation directly from camera images. Our method uses a deep spatial autoencoder to acquire a set of feature points that describe the environment for the current task, such as the positions of objects, and then learns a motion skill with these feature points using an efficient reinforcement learning method based on local linear models. The resulting controller reacts continuously to the learned feature points, allowing the robot to dynamically manipulate objects in the world with closed-loop control. We demonstrate our method with a PR2 robot on tasks that include pushing a free-standing toy block, picking up a bag of rice using a spatula, and hanging a loop of rope on a hook at various positions. In each task, our method automatically learns to track task-relevant objects and manipulate their configuration with the robot's arm. <s> BIB004 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Grasping and Manipulation <s> Robots which interact with the physical world will benefit from a fine-grained tactile understanding of objects and surfaces. Additionally, for certain tasks, robots may need to know the haptic properties of an object before touching it. To enable better tactile understanding for robots, we propose a method of classifying surfaces with haptic adjectives (e.g., compressible or smooth) from both visual and physical interaction data. Humans typically combine visual predictions and feedback from physical interactions to accurately predict haptic properties and interact with the world. Inspired by this cognitive pattern, we propose and explore a purely visual haptic prediction model. Purely visual models enable a robot to "feel" without physical interaction. Furthermore, we demonstrate that using both visual and physical interaction signals together yields more accurate haptic classification. Our models take advantage of recent advances in deep neural networks by employing a unified approach to learning features for physical interaction and visual observations. Even though we employ little domain specific knowledge, our model still achieves better results than methods based on hand-designed features. <s> BIB005 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Grasping and Manipulation <s> Present image based visual servoing approaches rely on extracting hand crafted visual features from an image. Choosing the right set of features is important as it directly affects the performance of any approach. Motivated by recent breakthroughs in performance of data driven methods on recognition and localization tasks, we aim to learn visual feature representations suitable for servoing tasks in unstructured and unknown environments. In this paper, we present an end-to-end learning based approach for visual servoing in diverse scenes where the knowledge of camera parameters and scene geometry is not available a priori. This is achieved by training a convolutional neural network over color images with synchronised camera poses. Through experiments performed in simulation and on a quadrotor, we demonstrate the efficacy and robustness of our approach for a wide range of camera poses in both indoor as well as outdoor environments. <s> BIB006 </s> A Survey on Deep Learning Methods for Robot Vision <s> Object Grasping and Manipulation <s> We describe a learning-based approach to hand-eye coordination for robotic grasping from monocular images. To learn hand-eye coordination for grasping, we trained a large convolutional neural network to predict the probability that task-space motion of the gripper will result in successful grasps, using only monocular camera images and independently of camera calibration or the current robot pose. This requires the network to observe the spatial relationship between the gripper and objects in the scene, thus learning hand-eye coordination. We then use this network to servo the gripper in real time to achieve successful grasps. To train our network, we collected over 800,000 grasp attempts over the course of two months, using between 6 and 14 robotic manipulators at any given time, with differences in camera placement and hardware. Our experimental evaluation demonstrates that our method achieves effective real-time control, can successfully grasp novel objects, and corrects mistakes by continuous servoing. <s> BIB007
The ability to grasp objects is of paramount importance for autonomous robots. Classical algorithms for grasping and manipulating objects require detecting both the gripper and the object in the image, and using graph-based or Jacobian-based algorithms for computing the movements for the gripper. Deep learning has generated a paradigm change since tasks like grasping, manipulation and visual servoing can be solved directly, without the need of hand-crafting intermediate processes like detecting features, detecting objects, detecting the gripper, or performing graph-based optimizations. As problems are solved as a whole from raw data, the performance limit imposed by inaccuracies of intermediate processes can be surpassed, and, therefore, novel approaches for solving tasks related to grasping, manipulation, and visual servoing are being developed. Also, the new algorithms are able to be generalized to novel objects and situations. The performance of this new generation of methods is currently limited by the size of the training datasets and the computing capabilities, which becomes relevant when using hardwarelimited robotic platforms. Grasping objects directly, without the need of detecting specific objects, is a difficult problem because it must generalize over different kinds of objects, including those not available when training the system. This problem has recently been addressed successfully by using deep learning. Studies that are based on selecting grasp locations include BIB001 ,  , BIB002 and BIB003 , which uses self-supervision during 700 hours of trial and error in a Baxter robot (see Figure 4) . That work has been extended for learning eye-hand coordination in BIB007 , which is the state-of-the-art. The latter system is trained without supervision by using over 800,000 grasp attempts collected over the course of two months, using between 6 and 14 manipulators at any given time. The architecture is based on a concatenation of two inputs: an image (size 512x512), and a motor command. The image network has 7 convolutional layers, and its output is concatenated with the motor command. The concatenation layer is further processed by an additional 8 convolutional layers and two fully-connected layers. The system achieves effective real-time control; the robot can grasp novel objects successfully and correct mistakes by continuous servoing, and lowers the failure rate from 35% (hand designed) to 20%. However, learned eye-hand coordination depends on the particular robot used for training the system. The full computing hardware used in this work is not fully described. It must be noted that these kinds of solutions are able to run successfully in real-time using GPUs. Systems able to infer manipulation trajectories for object-task pairs using DNNs have been developed recently. This task is difficult because generalization over different objects and manipulators is needed. They include Robobarista  , and BIB004 . These systems are able to learn from manipulation demonstrations and generalize over new objects. In Robobarista  , the input of the system is composed of a point cloud, a trajectory, and a natural language space, which is mapped to a joint space. Once trained, the trajectory generating the best similarity to a new point cloud and natural language instruction is selected. Computational hardware (GPU) and runtime are not reported in this work. In BIB004 the system is able to learn new tasks by using image-based reinforcement learning. That system is composed by an encoder and a decoder. The encoder is formed by 3 convolutional layers and 2 fully-connected layers. Computational hardware and runtime are not reported in this work. Visual servoing systems have also benefited from the use of deep learning techniques. In Saxena et al. BIB006 , a system that is able to perform image-based visual servoing by feeding raw data into convolutional networks is presented. The system does not require 3D geometry of the scene or intrinsic parameters from the camera. The CNN takes a pair of monocular images as input, representing the current and desired pose, and processes them by using 4 convolutional layers and a fully-connected layer. The system is trained for computing the transform in 3D space (position and orientation) needed for moving the camera from its current pose to the desired one. The estimated relative motion is used for generating linear and angular speed commands for moving the camera to the desired pose. The system is trained on synthetic data, and tested both in simulated environments and using a real quadrotor. The system takes 20 msec for processing each frame when using an NVIDIA Titan X GPU, and 65 msec when using a Quadro M2000 GPU. A wifi connection was used for transferring data between the drone and a host PC. Multimodal data delivers valuable information for object grasping. In BIB005 , a haptic object classifier that is fed by visual and haptic data is proposed. The outputs of a visual CNN and a haptic CNN are fused into a fusion layer, followed by a loss function for haptic adjectives. The visual CNN is based on the (5a) layer of GoogleNet, while the haptic CNN is based on 3 onedimensional convolutional layers followed by a fully-connected layer. Examples of haptic adjectives are "absorbent," "bumpy," and "compressible". The system, after being trained, is able to estimate haptic adjectives of objects by using only visual data, and is shown to improve over systems based on classical features. Computing hardware and runtime of the method is not reported. In summary, deep learning has enabled the generation of novel approaches to solving tasks related to object grasping, object manipulation, and visual servoing. Real-time processing is needed for hand-eye coordination and visual servoing to work, which imposes constraints on hardware capabilities. However, real-time processing is not needed for open-loop object grasping detection. Direct methods that do not require detecting objects to work are robust, but also dependent on the robot used for collecting data. GPUs are needed for deep learning based methods that require closed loop control, which limits its usefulness when working with hardware-limited robotic platforms. Grasping and manipulation algorithms will continue to benefit from the use of deep reinforcement learning methods in the near future.