Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Analytical methods <s> The synthesis of force-closure grasps on three-dimensional (3-D) objects is a fundamental issue in robotic grasping and dextrous manipulation. In this paper, a numerical force-closure test is developed based on the concept of Q distance. With some mild and realistic assumptions, the proposed test criterion is differentiable almost everywhere and its derivative can be calculated exactly. On this basis, we present an algorithm for planning force-closure grasps, which is implemented by applying descent search to the proposed numerical test in the grasp configuration space. The algorithm is generally applicable to planning optimal force-closure grasps on 3-D objects with curved surfaces and with arbitrary number of contact points. The effectiveness and efficiency of the algorithm are demonstrated by using simulation examples. <s> BIB001 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Analytical methods <s> In this paper, we present a novel algorithm for computing three-finger force-closure grasps of 2D and 3D objects. In the case of a robot hand with three hard fingers and point contact with friction, new necessary and sufficient conditions for 2D and 3D equilibrium and force-closure grasps have been deduced, and a corresponding algorithm for computing force-closure grasps has been developed. Based on geometrical analysis, the algorithm is simple and only needs a few algebraic calculations. Finally, the algorithm has been implemented and its effectivity has been demonstrated by two examples. <s> BIB002 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Analytical methods <s> Automatic grasp planning for robotic hands is a difficult problem because of the huge number of possible hand configurations. However, humans simplify the problem by choosing an appropriate prehensile posture appropriate for the object and task to be performed. By modeling an object as a set of shape primitives, such as spheres, cylinders, cones and boxes, we can use a set of rules to generate a set of grasp starting positions and pregrasp shapes that can then be tested on the object model. Each grasp is tested and evaluated within our grasping simulator "GraspIt!", and the best grasps are presented to the user. The simulator can also plan grasps in a complex environment involving obstacles and the reachability constraints of a robot arm. <s> BIB003 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Analytical methods <s> We propose a new and efficient approach to compute task oriented quality measures for dextrous grasps. Tasks can be specified as a single wrench to be applied, as a rough direction in form of a wrench cone, or as a complex wrench polytope. Based on the linear matrix inequality formalism to treat the friction cone constraints we formulate respective convex optimization problems, whose solutions give the maximal applicable wrench in the task direction together with the needed contact forces. Numerical experiments show that application to complex grasps with many contacts is possible. <s> BIB004 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Analytical methods <s> To reduce data collection time for deep learning of robust robotic grasp plans, we explore training from a synthetic dataset of 6.7 million point clouds, grasps, and analytic grasp metrics generated from thousands of 3D models from Dex-Net 1.0 in randomized poses on a table. We use the resulting dataset, Dex-Net 2.0, to train a Grasp Quality Convolutional Neural Network (GQ-CNN) model that rapidly predicts the probability of success of grasps from depth images, where grasps are specified as the planar position, angle, and depth of a gripper relative to an RGB-D sensor. Experiments with over 1,000 trials on an ABB YuMi comparing grasp planning methods on singulated objects suggest that a GQ-CNN trained with only synthetic data from Dex-Net 2.0 can be used to plan grasps in 0.8sec with a success rate of 93% on eight known objects with adversarial geometry and is 3x faster than registering point clouds to a precomputed dataset of objects and indexing grasps. The Dex-Net 2.0 grasp planner also has the highest success rate on a dataset of 10 novel rigid objects and achieves 99% precision (one false positive out of 69 grasps classified as robust) on a dataset of 40 novel household objects, some of which are articulated or deformable. Code, datasets, videos, and supplementary material are available at http://berkeleyautomation.github.io/dex-net . <s> BIB005 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Analytical methods <s> In this paper, we explore deep reinforcement learning algorithms for vision-based robotic grasping. Model-free deep reinforcement learning (RL) has been successfully applied to a range of challenging environments, but the proliferation of algorithms makes it difficult to discern which particular approach would be best suited for a rich, diverse task like grasping. To answer this question, we propose a simulated benchmark for robotic grasping that emphasizes off-policy learning and generalization to unseen objects. Off-policy learning enables utilization of grasping data over a wide variety of objects, and diversity is important to enable the method to generalize to new objects that were not seen during training. We evaluate the benchmark tasks against a variety of Q-function estimation methods, a method previously proposed for robotic grasping with deep neural network models, and a novel approach based on a combination of Monte Carlo return estimation and an off-policy correction. Our results indicate that several simple methods provide a surprisingly strong competitor to popular algorithms such as double Q-learning, and our analysis of stability sheds light on the relative tradeoffs between the algorithms. <s> BIB006 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Analytical methods <s> Tool manipulation is vital for facilitating robots to complete challenging task goals. It requires reasoning about the desired effect of the task and thus properly grasping and manipulating the tool to achieve the task. Task-agnostic grasping optimizes for grasp robustness while ignoring crucial task-specific constraints. In this paper, we propose the Task-Oriented Grasping Network (TOG-Net) to jointly optimize both task-oriented grasping of a tool and the manipulation policy for that tool. The training process of the model is based on large-scale simulated self-supervision with procedurally generated tool objects. We perform both simulated and real-world experiments on two tool-based manipulation tasks: sweeping and hammering. Our model achieves overall 71.1% task success rate for sweeping and 80.0% task success rate for hammering. Supplementary material is available at: this http URL <s> BIB007 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Analytical methods <s> We describe a learning-based approach to hand-eye coordination for robotic grasping from monocular images. To learn hand-eye coordination for grasping, we trained a large convolutional neural network to predict the probability that task-space motion of the gripper will result in successful grasps, using only monocular camera images and independently of camera calibration or the current robot pose. This requires the network to observe the spatial relationship between the gripper and objects in the scene, thus learning hand-eye coordination. We then use this network to servo the gripper in real time to achieve successful grasps. To train our network, we collected over 800,000 grasp attempts over the course of two months, using between 6 and 14 robotic manipulators at any given time, with differences in camera placement and hardware. Our experimental evaluation demonstrates that our method achieves effective real-time control, can successfully grasp novel objects, and corrects mistakes by continuous servoing. <s> BIB008 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Analytical methods <s> This paper presents a robotic pick-and-place system that is capable of grasping and recognizing both known and novel objects in cluttered environments. The key new feature of the system is that it handles a wide range of object categories without needing any task-specific training data for novel objects. To achieve this, it first uses a category-agnostic affordance prediction algorithm to select and execute among four different grasping primitive behaviors. It then recognizes picked objects with a cross-domain image classification framework that matches observed images to product images. Since product images are readily available for a wide range of objects (e.g., from the web), the system works out-of-the-box for novel objects without requiring any additional training data. Exhaustive experimental results demonstrate that our multi-affordance grasping achieves high success rates for a wide variety of objects in clutter, and our recognition algorithm achieves high accuracy for both known and novel grasped objects. The approach was part of the MIT-Princeton Team system that took 1st place in the stowing task at the 2017 Amazon Robotics Challenge. All code, datasets, and pre-trained models are available online at http://arc.cs.princeton.edu <s> BIB009 </s> Vision-based Robotic Grasping from Object Localization, Pose Estimation, Grasp Detection to Motion Planning: A Review <s> Analytical methods <s> We present a new approach to transfer grasp configurations from prior example objects to novel objects. We assume the novel and example objects have the same topology and similar shapes. We perform 3D segmentation on these objects using geometric and semantic shape characteristics. We compute a grasp space for each part of the example object using active learning. We build bijective contact mapping between these model parts and compute the corresponding grasps for novel objects. Finally, we assemble the individual parts and use local replanning to adjust grasp configurations while maintaining its stability and physical constraints. Our approach is general, can handle all kind of objects represented using mesh or point cloud and a variety of robotic hands. <s> BIB010
Analytical methods consider kinematics and dynamics formulation in determining grasps  . Forceclosure and task compatibility are two main conditions in completing the grasping tasks. The force-closure is satisfied to afford a stable grasp, which only affords simple picking tasks. Whereas, task compatible grasps support specific tasks and optimally accomplish the grasping task. Typical functional flow-chart of analytical grasp detection methods is illustrated in Fig. 24 . There exist many force-closure grasp synthesis methods for 3D objects. Among them, the polyhedral objects are firstly dealt with, as they are composed of a finite number of flat faces. The force-closure condition is reduced into the test of the angles between the faces normals  or using the linear model to derive analytical formulation for grasp characterization BIB006   . These methods did not consider the issue of selecting a grasping facet and conduct exhaustive searches. To handle the commonly used objects which usually have more complicated shapes, methods placing no restrictions on the object model by observing different contact points are proposed BIB002  . These methods try to find contact points on a 3D object surface to ensure force-closure and compute the optimal grasp by minimizing an objective energy function according to a predefined grasp quality criterion [Mirtich and Canny, 1994; BIB001 . However, searching the grasp solution space is a complex problem which is quite timeconsuming. Some heuristical techniques were then proposed to reduce the search space by generating a set of grasp candidates according to a predefined procedure  BIB007 , or by defining a set of rules to generate the starting positions BIB003 . These methods can only produce one successful grasp and are thus suitable for dealing with rather simple tasks such as picking and placing. The choice of an optimal grasp is usually determined by the specific task to perform, and this leads to the research on task-oriented grasps. To do that, task-related grasp quality measures based on the capabilities to generate wrenches BIB008 or a linear matrix inequality formalism BIB004 have been proposed. Prats et al. BIB006 proposed to use hand-preshapes during the early grasp planning stages where the task was considered. These methods still face the problem of precisely representing the task and are computationally unaffordable. Meanwhile, these methods could neither be adapted for new tasks nor for new objects. To solve these problems, Song et al. BIB009 proposed a general approach for planning grasps on 3D objects based on hand-object geometric fitting. They built a contact score map on a 3D object's voxelization and applied this score map and a hand's kinematic parameters to find a set of target contacts on the object surface. Although this method works well for cases in which a complete 3D shape is provided in the image, it will fail when only partial view of the shape is available. Empirical methods aiming at similar objects In most cases, the target objects are not totally the same with the objects in the existing database. If an object comes from a class that is involved in the database, it is regarded as a similar object. After the localization of the target object, correspondences-based methods can be utilized to transfer the grasp points from the similar full 3D object to the current partial view object. These methods learn the grasp by observing the object without estimating its 6D pose, since the current target object is not totally the same with the objects in the database. Typical functional flow-chart of empirical methods aiming at similar objects is illustrated in Fig. 25 . Different kinds of methods are utilized to find the correspondences based on taxonomy, segmentation, and so on. Andrew et al. BIB003 proposed a taxonomybased approach, which classified objects into categories that should be grasped by each canonical grasp. Vahrenkamp et al. [Vahrenkamp et al., 2016 ] presented a part-based grasp planning approach to generate grasps that are applicable to multiple familiar objects, as shown in Fig. 26 . The object models are segmented according to their shape and volumetric information, and the objet parts are labeled with semantic and grasping information. A grasp transferability measure is proposed to evaluate how successful planned grasps are applied to novel object instances of the same object category. Tian et al. BIB010 proposed a method to transfer grasp configurations from prior example objects to novel objets, which assumes that the novel and example objects have the same topology and similar shapes. They perform 3D segmentation on the objects considering geometric and semantic shape characteristics, compute a grasp space for each part of the example object using active learning, and build bijective contact mapping between the model parts and the corresponding grasps for novel objects. Empirical methods aiming at unknown objects Above empirical methods of robotic grasping are performed based on the premise that certain prior knowledge, such as object geometry, physics models, or force analytic, are known. The grasp database usually covers a limited amount of objects, and empirical methods will face difficulties in dealing with unknown object. Whereas, the grasp experience learned before could provide the quality measurements when dealing with unknown objects. Typical functional flow-chart of empirical methods aiming at unknown objects is illustrated in Fig. 27 . Mahler et al. BIB005 proposed a deep learning-based method to plan robust grasps with synthetic point clouds and analytic grasping metrics, as shown in Fig. 28 . They first segment the current points of interests from the depth image, and multiple candidate grasp points are generated. The grasp qualities are then measured and the one with the highest quality will be selected as the final grasp point. Their database have more than 50k grasps, and the grasp quality measurement network achieved relatively satisfactory performance. 