Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Comparisons <s> Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks by themselves, trained end-to-end, pixels-to-pixels, exceed the state-of-the-art in semantic segmentation. Our key insight is to build “fully convolutional” networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning. We define and detail the space of fully convolutional networks, explain their application to spatially dense prediction tasks, and draw connections to prior models. We adapt contemporary classification networks (AlexNet [20], the VGG net [31], and GoogLeNet [32]) into fully convolutional networks and transfer their learned representations by fine-tuning [3] to the segmentation task. We then define a skip architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations. Our fully convolutional network achieves state-of-the-art segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012), NYUDv2, and SIFT Flow, while inference takes less than one fifth of a second for a typical image. <s> BIB001 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Comparisons <s> To reduce data collection time for deep learning of robust robotic grasp plans, we explore training from a synthetic dataset of 6.7 million point clouds, grasps, and analytic grasp metrics generated from thousands of 3D models from Dex-Net 1.0 in randomized poses on a table. We use the resulting dataset, Dex-Net 2.0, to train a Grasp Quality Convolutional Neural Network (GQ-CNN) model that rapidly predicts the probability of success of grasps from depth images, where grasps are specified as the planar position, angle, and depth of a gripper relative to an RGB-D sensor. Experiments with over 1,000 trials on an ABB YuMi comparing grasp planning methods on singulated objects suggest that a GQ-CNN trained with only synthetic data from Dex-Net 2.0 can be used to plan grasps in 0.8sec with a success rate of 93% on eight known objects with adversarial geometry and is 3x faster than registering point clouds to a precomputed dataset of objects and indexing grasps. The Dex-Net 2.0 grasp planner also has the highest success rate on a dataset of 10 novel rigid objects and achieves 99% precision (one false positive out of 69 grasps classified as robust) on a dataset of 40 novel household objects, some of which are articulated or deformable. Code, datasets, videos, and supplementary material are available at http://berkeleyautomation.github.io/dex-net . <s> BIB002 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Comparisons <s> In this work we address the task of semantic image segmentation with Deep Learning and make three main contributions that are experimentally shown to have substantial practical merit. First , we highlight convolution with upsampled filters, or ‘atrous convolution’, as a powerful tool in dense prediction tasks. Atrous convolution allows us to explicitly control the resolution at which feature responses are computed within Deep Convolutional Neural Networks. It also allows us to effectively enlarge the field of view of filters to incorporate larger context without increasing the number of parameters or the amount of computation. Second , we propose atrous spatial pyramid pooling (ASPP) to robustly segment objects at multiple scales. ASPP probes an incoming convolutional feature layer with filters at multiple sampling rates and effective fields-of-views, thus capturing objects as well as image context at multiple scales. Third , we improve the localization of object boundaries by combining methods from DCNNs and probabilistic graphical models. The commonly deployed combination of max-pooling and downsampling in DCNNs achieves invariance but has a toll on localization accuracy. We overcome this by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF), which is shown both qualitatively and quantitatively to improve localization performance. Our proposed “DeepLab” system sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 79.7 percent mIOU in the test set, and advances the results on three other datasets: PASCAL-Context, PASCAL-Person-Part, and Cityscapes. All of our code is made publicly available online. <s> BIB003 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Comparisons <s> This paper presents a robotic pick-and-place system that is capable of grasping and recognizing both known and novel objects in cluttered environments. The key new feature of the system is that it handles a wide range of object categories without needing any task-specific training data for novel objects. To achieve this, it first uses a category-agnostic affordance prediction algorithm to select and execute among four different grasping primitive behaviors. It then recognizes picked objects with a cross-domain image classification framework that matches observed images to product images. Since product images are readily available for a wide range of objects (e.g., from the web), the system works out-of-the-box for novel objects without requiring any additional training data. Exhaustive experimental results demonstrate that our multi-affordance grasping achieves high success rates for a wide variety of objects in clutter, and our recognition algorithm achieves high accuracy for both known and novel grasped objects. The approach was part of the MIT-Princeton Team system that took 1st place in the stowing task at the 2017 Amazon Robotics Challenge. All code, datasets, and pre-trained models are available online at http://arc.cs.princeton.edu <s> BIB004
Comparisons of mAP of some typical object detection networks on COCO dataset are presented in Table 6 .  Here we simply give the performance results of some typical segmentation networks on the PASCAL VOC-2012 dataset in Table 8 . Among all these methods, DeepLab BIB003 has the best performance. DeepLab BIB003 79.70 Dilation BIB004 75.30 CRFasRNN [Zheng et al., 2015] 74.70 ParseNet 69.80 FCN-8s BIB001 67.20 Bayesian SegNet  60.50 The accuracy of different methods is the core element to be measured, and other properties such as speed and robustness to occlusions are also introduced here. Hodaň et al.  utilized the VSD metric, where the misalignment tolerance τ is set to 20 mm and the correctness threshold θ is set to 0.3. They compared many algorithms and the representative methods are presented in Table 10 . Among them, Buch-16 [Buch et al., 2016] is the correspondence-based method, Hodaň  is the template-based method, and Drost-10 [Drost et al., 2010] , Brachmann-14 [Brachmann et al., 2014] and Vidal-18 [Vidal et al., 2018] are voting-based methods. It was concluded that method of Vidal-18 [Vidal et al., 2018] which is based on the point-pair features outperformed the other methods. However, comparisons with regression-based methods were not conducted. Wang et al.  proposed a generic framework for estimating the 6D poses of a set of known objects from RGB-D images and conducted comparisons with regression-based methods using the ADD-S metric on YCBvideo dataset. The results are shown in Table 11 . Besides, they also conducted experiments on the LineMOD dataset using the ADD metric, and the results are shown in Table 12 . From the tables, we can see that DenseFusion achieved the highest accuracy comparing with other regression-based methods. This is because DenseFusion employed a novel local feature fusion scheme using both RGB and depth images, whereas other methods only utilized single source data.  The performance of various grasp detection methods on DexNet2.0 BIB002 are listed in Table 14 . In this table, IGQ represents the image-based grasp quality metric, where a set of force closure grasp candidates are firstly sampled BIB003 and ranked. REG, ML-RF, ML-SVM and GQ-L-Adv represent the point cloud registration method  , Random Forestbased machine learning method, Support Vector Machinebased machine learning method, and the Grasp Quality CNN model-based method, respectively. 