Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> End-to-end Grasp Detection <s> Object detection performance, as measured on the canonical PASCAL VOC dataset, has plateaued in the last few years. The best-performing methods are complex ensemble systems that typically combine multiple low-level image features with high-level context. In this paper, we propose a simple and scalable detection algorithm that improves mean average precision (mAP) by more than 30% relative to the previous best result on VOC 2012 -- achieving a mAP of 53.3%. Our approach combines two key insights: (1) one can apply high-capacity convolutional neural networks (CNNs) to bottom-up region proposals in order to localize and segment objects and (2) when labeled training data is scarce, supervised pre-training for an auxiliary task, followed by domain-specific fine-tuning, yields a significant performance boost. Since we combine region proposals with CNNs, we call our method R-CNN: Regions with CNN features. We also present experiments that provide insight into what the network learns, revealing a rich hierarchy of image features. Source code for the complete system is available at http://www.cs.berkeley.edu/~rbg/rcnn. <s> BIB001 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> End-to-end Grasp Detection <s> Recently, a number of grasp detection methods have been proposed that can be used to localize robotic grasp configurations directly from sensor data without estimating object pose. The underlying idea is to treat grasp perception analogously to object detection in computer vision. These methods take as input a noisy and partially occluded RGBD image or point cloud and produce as output pose estimates of viable grasps, without assuming a known CAD model of the object. Although these methods generalize grasp knowledge to new objects well, they have not yet been demonstrated to be reliable enough for wide use. Many grasp detection methods achieve grasp success rates (grasp successes as a fraction of the total number of grasp attempts) between 75% and 95% for novel objects presented in isolation or in light clutter. Not only are these success rates too low for practical grasping applications, but the light clutter scenarios that are evaluated often do not reflect the realities of real world grasping. This paper proposes a number of innovations that together result in a significant improvement in grasp detection performance. The specific improvement in performance due to each of our contributions is quantitatively measured either in simulation or on robotic hardware. Ultimately, we report a series of robotic experiments that average a 93% end-to-end grasp success rate for novel objects presented in dense clutter. <s> BIB002 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> End-to-end Grasp Detection <s> In this paper, we explore deep reinforcement learning algorithms for vision-based robotic grasping. Model-free deep reinforcement learning (RL) has been successfully applied to a range of challenging environments, but the proliferation of algorithms makes it difficult to discern which particular approach would be best suited for a rich, diverse task like grasping. To answer this question, we propose a simulated benchmark for robotic grasping that emphasizes off-policy learning and generalization to unseen objects. Off-policy learning enables utilization of grasping data over a wide variety of objects, and diversity is important to enable the method to generalize to new objects that were not seen during training. We evaluate the benchmark tasks against a variety of Q-function estimation methods, a method previously proposed for robotic grasping with deep neural network models, and a novel approach based on a combination of Monte Carlo return estimation and an off-policy correction. Our results indicate that several simple methods provide a surprisingly strong competitor to popular algorithms such as double Q-learning, and our analysis of stability sheds light on the relative tradeoffs between the algorithms. <s> BIB003 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> End-to-end Grasp Detection <s> This paper presents a robotic pick-and-place system that is capable of grasping and recognizing both known and novel objects in cluttered environments. The key new feature of the system is that it handles a wide range of object categories without needing any task-specific training data for novel objects. To achieve this, it first uses a category-agnostic affordance prediction algorithm to select and execute among four different grasping primitive behaviors. It then recognizes picked objects with a cross-domain image classification framework that matches observed images to product images. Since product images are readily available for a wide range of objects (e.g., from the web), the system works out-of-the-box for novel objects without requiring any additional training data. Exhaustive experimental results demonstrate that our multi-affordance grasping achieves high success rates for a wide variety of objects in clutter, and our recognition algorithm achieves high accuracy for both known and novel grasped objects. The approach was part of the MIT-Princeton Team system that took 1st place in the stowing task at the 2017 Amazon Robotics Challenge. All code, datasets, and pre-trained models are available online at http://arc.cs.princeton.edu <s> BIB004
In this section, the grasp positions are detected in an end-toend manner, which means the localization of the target object is skipped and the grasp positions are recovered directly from the input image, as shown in Fig. 3 . Typical functional flowchart of end-to-end grasp detection is illustrated in Fig. 29 . This kind of methods can be divided into two-stage and onestage methods. The former method first estimates candidate grasp positions and then selects the most likely one, while the latter method regress grasp positions directly. In the two-stage method, the sliding window strategy is commonly used for detecting 2D robotic grasps. Lenz et al.  presented a two-step cascaded system with two deep networks, where the top detection results from the first are re-evaluated by the second, as shown in Fig. 30 . The first network has fewer features, is faster to run, and can effectively prune out unlikely candidate grasps. The second, with more features, is slower but has to run only on the top few detections. Even though they achieved a high accuracy, the iterative scanning makes the process very slow. ten Pas et al. BIB002 proposed a method for generating grasp hypotheses on any visible surface without requiring a precise segmentation of the target object. They also proposed a new grasp descriptor that incorporates surface normals and multiple views. However, multiple objects may be treated as a single atomic object since instance-level segmentation is not conducted. Since a uniform network would perform better than the two-cascaded system  , more and more one-stage methods have been proposed. Redmon and Angelova [Redmon and Angelova, 2015] proposed a larger neural network, which performs a single-stage regression to obtain graspable bounding boxes without using standard sliding The system obtains an RGB-D image from a Kinect mounted on the robot, and searches over a large space of possible grasps, for which some candidates are shown. For each of these, it extracts a set of raw features corresponding to the color and depth images and surface normals, then uses these as inputs to a deep network which scores each rectangle. Finally, the top-ranked rectangle is selected and the corresponding grasp is executed using the parameters of the detected rectangle and the surface normal at its center. Red and green lines correspond to gripper plates, blue in RGB-D features indicates masked-out pixels. (Courtesy of  window or region proposal techniques. Guo et al. BIB001 presented a shared convolutional neural network to conduct object discovery and grasp detection. Pinto and Gupta BIB003 proposed a method to predict grasp locations via trial and error, as shown in Fig. 31 . They trained a CNN-based classifier to estimate the grasp likelihood for different grasp directions given an input image patch. Chu et al.  introduced a network composed of a grasp region proposal component and a robotic grasp detection component. Park et al.  proposed an accurate robotic grasp detection algorithm using fully convolutional neural networks with highresolution images to recover the five poses (x, y, α, w, h) for manipulation. Zeng et al. BIB004 presented a system capable of picking and recognizing novel objects with limited prior knowledge. This system first uses a categoryagnostic affordable prediction algorithm to select among four different grasping primitive behaviors, and then recognizes the grasped objects by matching them to their product images. This method took 1st place in the stowing task of the Amazon Robotics Challenge 2017. Anyway, this method is not designed to pick a desired object for which the state of the target is needed. BIB003 Challenges For the end-to-end grasp detection methods, the computed grasp point may not be the globally optimal one, as only part of the object is available in the image. Meanwhile, they only considered the geometric information, while other important factors like material and weight are missing.