Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> This paper presents fast and simple algorithms for directly constructing force-closure grasps based on the shape of the grasped object. The synthesis of force-closure grasps finds in dependent regions of contact for the fingertips, such that the motion of the grasped object is totally constrained. A force- closure grasp implies equilibrium grasps exist. In the reverse direction, we show that most nonmarginal equilibrium grasps are force-closure grasps. <s> BIB001 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> The authors address the problem of planning optimal grasps. Two general optimality criteria that consider the total finger force and the maximum finger force are introduced and discussed. Their formalization using various metrics on a space of generalized forces is detailed. The geometric interpretation of the two criteria leads to an efficient planning algorithm. An example of its use in a robotic environment equipped with two-jaw and three-jaw is described. > <s> BIB002 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> This article presents a survey of the existing computational algorithms meant for achieving four important properties in autonomous multifingered robotic hands. The four properties are: dexterity, equilibrium, stability, and dynamic behavior The multifingered robotic hands must be controlled so as to possess these properties and hence be able to autonomously perform complex tasks in a way similar to human hands. Existing algorithms to achieve dexterity primarily involve solving an unconstrained linear programming problem where an objective function can be chosen to represent one or more of the currently known dexterity measures. Algorithms to achieve equilibrium also constitute solving a linear program ming problem wherein the positivity, friction, and joint torque constraints of all fingers are accounted for while optimizing the internal grasping forces. Stability algorithms aim at achiev ing positive definite grasp impedance matrices by solving for the required fingertip impedances. This problem reduces to a nonlinear programming problem. Dynamic behavior algorithms determine fingertip impedances, which, when achieved, lead to a desired dynamic behavior. This problem, too, becomes a linear programming problem. If a robotic hand has to acquire any or all of these proper ties, the corresponding algorithms should become integral parts of the hand control system. These algorithms are collectively referred to in this article as robot grasp synthesis algorithms. <s> BIB003 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> A scheme for learning to grasp objects using visual information is presented. The learning problem is divided into two separate subproblems: choosing grasping points and predicting the quality of a given grasp. For each grasp we store location parameters that code the locations of the grasping points, quality parameters that are relevant features for the assessment of grasp quality, and the associated grade. The location parameters, using a special coding which is not object specific, are used to locate grasping points on new target objects. A function from the quality parameters to the grade is learned from examples. Grasp quality for novel situations can be generalized and estimated using the learned function. An experimental setup using an AdeptOne manipulator was developed to test this scheme. The system had demonstrated an ability to grasp a relatively wide variety of objects, and its performance had significantly improved with practice following a small number of trials. The knowledge learned for a set of objects was successfully generalized to new objects. <s> BIB004 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> Automatic grasp planning for robotic hands is a difficult problem because of the huge number of possible hand configurations. However, humans simplify the problem by choosing an appropriate prehensile posture appropriate for the object and task to be performed. By modeling an object as a set of shape primitives, such as spheres, cylinders, cones and boxes, we can use a set of rules to generate a set of grasp starting positions and pregrasp shapes that can then be tested on the object model. Each grasp is tested and evaluated within our grasping simulator "GraspIt!", and the best grasps are presented to the user. The simulator can also plan grasps in a complex environment involving obstacles and the reachability constraints of a robot arm. <s> BIB005 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> Many methods for generating and analyzing grasps have been developed in the recent years. They gave insight and comprehension of grasping with robot hands but many of them are rather complicated to implement and of high computational complexity. In this paper we study if the basic quality criterion for grasps, the force-closure property, is in principle easy or difficult to reach. We show that it is not necessary to generate optimal grasps, due to a certain quality measure, for real robot grasping tasks where an average quality grasp is acceptable. We present statistical data that confirm our opinion that a randomized grasp generation algorithm is fast and suitable for the planning of robot grasping tasks. <s> BIB006 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> Finding appropriate stable grasps for a hand (either robotic or human) on an arbitrary object has proved to be a challenging and difficult problem. The space of grasping parameters coupled with the degrees-of-freedom and geometry of the object to be grasped creates a high-dimensional, non-smooth manifold. Traditional search methods applied to this manifold are typically not powerful enough to find appropriate stable grasping solutions, let alone optimal grasps. We address this issue in this paper, which attempts to find optimal grasps of objects using a grasping simulator. Our unique approach to the problem involves a combination of numerical methods to recover parts of the grasp quality surface with any robotic hand, and contemporary machine learning methods to interpolate that surface, in order to find the optimal grasp. <s> BIB007 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> Manipulation skills are a key issue for a humanoid robot. Here, we are interested in a vision-based grasping system able to deal with previously unknown objects in real time and in an intelligent man- ner. Starting from a number of feasible candidate grasps, we focus on the problem of predicting their reliability using the knowledge acquired in previous grasping experiences. A set of visual features which take into account physical properties that can aect the stability and reliability of a grasp are dened. A humanoid robot obtains its grasping experience by repeating a large number of grasping actions on dieren t objects. An experimental protocol is established in order to classify grasps according to their reliability. Two prediction/classication strategies are dened which allow the robot to predict the outcome of a grasp only analizing its visual features. The results indicate that these strategies are adequate to predict the realibility of a grasp and to generalize to dieren t objects. <s> BIB008 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> Current approaches to visual object class detection mainly focus on the recognition of basic level categories, such as cars, motorbikes, mugs and bottles. Although these approaches have demonstrated impressive performance in terms of recognition, their restriction to these categories seems inadequate in the context of embodied, cognitive agents. Here, distinguishing objects according to functional aspects based on object affordances is important in order to enable manipulation of and interaction between physical objects and cognitive agent. ::: ::: In this paper, we propose a system for the detection of functional object classes, based on a representation of visually distinct hints on object affordances (affordance cues). It spans the complete range from tutordriven acquisition of affordance cues, learning of corresponding object models, and detecting novel instances of functional object classes in real images. <s> BIB009 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> Planning realizable and stable grasps on 3D objects is crucial for many robotics applications, but grasp planners often ignore the relative sizes of the robotic hand and the object being grasped or do not account for physical joint and positioning limitations. We present a grasp planner that can consider the full range of parameters of a real hand and an arbitrary object, including physical and material properties as well as environmental obstacles and forces, and produce an output grasp that can be immediately executed. We do this by decomposing a 3D model into a superquadric 'decomposition tree' which we use to prune the intractably large space of possible grasps into a subspace that is likely to contain many good grasps. This subspace can be sampled and evaluated in GraspIt!, our 3D grasping simulator, to find a set of highly stable grasps, all of which are physically realizable. We show grasp results on various models using a Barrett hand. <s> BIB010 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> Affordances encode relationships between actions, objects, and effects. They play an important role on basic cognitive capabilities such as prediction and planning. We address the problem of learning affordances through the interaction of a robot with the environment, a key step to understand the world properties and develop social skills. We present a general model for learning object affordances using Bayesian networks integrated within a general developmental architecture for social robots. Since learning is based on a probabilistic model, the approach is able to deal with uncertainty, redundancy, and irrelevant information. We demonstrate successful learning in the real world by having an humanoid robot interacting with objects. We illustrate the benefits of the acquired knowledge in imitation games. <s> BIB011 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> We consider the problem of grasping novel objects, specifically objects that are being seen for the first time through vision. Grasping a previously unknown object, one for which a 3-d model is not available, is a challenging problem. Furthermore, even if given a model, one still has to decide where to grasp the object. We present a learning algorithm that neither requires nor tries to build a 3-d model of the object. Given two (or more) images of an object, our algorithm attempts to identify a few points in each image corresponding to good locations at which to grasp the object. This sparse set of points is then triangulated to obtain a 3-d location at which to attempt a grasp. This is in contrast to standard dense stereo, which tries to triangulate every single point in an image (and often fails to return a good 3-d model). Our algorithm for identifying grasp locations from an image is trained by means of supervised learning, using synthetic images for the training set. We demonstrate this approach on two robotic manipulation platforms. Our algorithm successfully grasps a wide variety of objects, such as plates, tape rolls, jugs, cellphones, keys, screwdrivers, staplers, a thick coil of wire, a strangely shaped power horn and others, none of which were seen in the training set. We also apply our method to the task of unloading items from dishwashers. <s> BIB012 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> We consider the problem of grasping novel objects in cluttered environments. If a full 3-d model of the scene were available, one could use the model to estimate the stability and robustness of different grasps (formalized as form/force-closure, etc); in practice, however, a robot facing a novel object will usually be able to perceive only the front (visible) faces of the object. In this paper, we propose an approach to grasping that estimates the stability of different grasps, given only noisy estimates of the shape of visible portions of an object, such as that obtained from a depth sensor. By combining this with a kinematic description of a robot arm and hand, our algorithm is able to compute a specific positioning of the robot's fingers so as to grasp an object. ::: ::: We test our algorithm on two robots (with very different arms/manipulators, including one with a multifingered hand). We report results on the task of grasping objects of significantly different shapes and appearances than ones in the training set, both in highly cluttered and in uncluttered environments. We also apply our algorithm to the problem of unloading items from a dishwasher. <s> BIB013 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> In this paper we focus on the concept of low-dimensional posture subspaces for artificial hands. We begin by discussing the applicability of a hand configuration subspace to the problem of automated grasp synthesis; our results show that low-dimensional optimization can be instrumental in deriving effective pre-grasp shapes for a number of complex robotic hands. We then show that the computational advantages of using a reduced dimensionality framework enable it to serve as an interface between the human and automated components of an interactive grasping system. We present an on-line grasp planner that allows a human operator to perform dexterous grasping tasks using an artificial hand. In order to achieve the computational rates required for effective user interaction, grasp planning is performed in a hand posture subspace of highly reduced dimensionality. The system also uses real-time input provided by the operator, further simplifying the search for stable grasps to the point where solutions can be found at interactive rates. We demonstrate our approach on a number of different hand models and target objects, in both real and virtual environments. <s> BIB014 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> This paper addresses the issue of learning and representing object grasp affordances, i.e. object-gripper relative configurations that lead to successful grasps. The purpose of grasp affordances is to organize and store the whole knowledge that an agent has about the grasping of an object, in order to facilitate reasoning on grasping solutions and their achievability. The affordance representation consists in a continuous probability density function defined on the 6D gripper pose space - 3D position and orientation -, within an object-relative reference frame. Grasp affordances are initially learned from various sources, e.g. from imitation or from visual cues, leading to grasp hypothesis densities. Grasp densities are attached to a learned 3D visual object model, and pose estimation of the visual model allows a robotic agent to execute samples from a grasp hypothesis density under various object poses. Grasp outcomes are used to learn grasp empirical densities, i.e. grasps that have been confirmed through experience. We show the result of learning grasp hypothesis densities from both imitation and visual cues, and present grasp empirical densities learned from physical experience by a robot. <s> BIB015 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> In our recent work [1], [2], we proposed Point Feature Histograms (PFH) as robust multi-dimensional features which describe the local geometry around a point p for 3D point cloud datasets. In this paper, we modify their mathematical expressions and perform a rigorous analysis on their robustness and complexity for the problem of 3D registration for overlapping point cloud views. More concretely, we present several optimizations that reduce their computation times drastically by either caching previously computed values or by revising their theoretical formulations. The latter results in a new type of local features, called Fast Point Feature Histograms (FPFH), which retain most of the discriminative power of the PFH. Moreover, we propose an algorithm for the online computation of FPFH features for realtime applications. To validate our results we demonstrate their efficiency for 3D registration and propose a new sample consensus based method for bringing two datasets into the convergence basin of a local non-linear optimizer: SAC-IA (SAmple Consensus Initial Alignment). <s> BIB016 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> We consider the problem of grasping novel objects and its application to cleaning a desk. A recent successful approach applies machine learning to learn one grasp point in an image and a point cloud. Although those methods are able to generalize to novel objects, they yield suboptimal results because they rely on motion planner for finger placements. In this paper, we extend their method to accommodate grasps with multiple contacts. This approach works well for many human-made objects because it models the way we grasp objects. To further improve the grasping, we also use a method that learns the ranking between candidates. The experiments show that our method is highly effective compared to a state-of-the-art competitor. <s> BIB017 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> This paper presents work on vision based robotic grasping. The proposed method adopts a learning framework where prototypical grasping points are learnt from several examples and then used on novel objects. For representation purposes, we apply the concept of shape context and for learning we use a supervised learning approach in which the classifier is trained with labelled synthetic images. We evaluate and compare the performance of linear and non-linear classifiers. Our results show that a combination of a descriptor based on shape context with a non-linear classification algorithm leads to a stable detection of grasping points for a variety of objects. <s> BIB018 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> Passive stereo vision is widely used as a range sensing technology in robots, but suffers from dropouts: areas of low texture where stereo matching fails. By supplementing a stereo system with a strong texture projector, dropouts can be eliminated or reduced. This paper develops a practical stereo projector system, first by finding good patterns to project in the ideal case, then by analyzing the effects of system blur and phase noise on these patterns, and finally by designing a compact projector that is capable of good performance out to 3m in indoor scenes. The system has been implemented and has excellent depth precision and resolution, especially in the range out to 1.5m. <s> BIB019 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> Robotic grasping in unstructured environments requires the ability to select grasps for unknown objects and execute them while dealing with uncertainty due to sensor noise or calibration errors. In this work, we propose a simple but robust approach to grasp selection for unknown objects, and a reactive adjustment approach to deal with uncertainty in object location and shape. The grasp selection method uses 3D sensor data directly to determine a ranked set of grasps for objects in a scene, using heuristics based on both the overall shape of the object and its local features. The reactive grasping approach uses tactile feedback from fingertip sensors to execute a compliant robust grasp. We present experimental results to validate our approach by grasping a wide range of unknown objects. Our results show that reactive grasping can correct for a fair amount of uncertainty in the measured position or shape of the objects, and that our grasp selection approach is successful in grasping objects with a variety of shapes. <s> BIB020 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> We present a novel vision-based grasp point detection algorithm that can reliably detect the corners of a piece of cloth, using only geometric cues that are robust to variation in texture. Furthermore, we demonstrate the effectiveness of our algorithm in the context of folding a towel using a general-purpose two-armed mobile robotic platform without the use of specialized end-effectors or tools. The robot begins by picking up a randomly dropped towel from a table, goes through a sequence of vision-based re-grasps and manipulations—partially in the air, partially on the table—and finally stacks the folded towel in a target location. The reliability and robustness of our algorithm enables for the first time a robot with general purpose manipulators to reliably and fully-autonomously fold previously unseen towels, demonstrating success on all 50 out of 50 single-towel trials as well as on a pile of 5 towels. <s> BIB021 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> This overview presents computational algorithms for generating 3D object grasps with autonomous multi-fingered robotic hands. Robotic grasping has been an active research subject for decades, and a great deal of effort has been spent on grasp synthesis algorithms. Existing papers focus on reviewing the mechanics of grasping and the finger-object contact interactions Bicchi and Kumar (2000) [12] or robot hand design and their control Al-Gallaf et al. (1993) [70]. Robot grasp synthesis algorithms have been reviewed in Shimoga (1996) [71], but since then an important progress has been made toward applying learning techniques to the grasping problem. This overview focuses on analytical as well as empirical grasp synthesis approaches. <s> BIB022 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> In this paper, we present a novel grasp selection algorithm to enable a robot with a two-fingered end-effector to autonomously grasp unknown objects. Our approach requires as input only the raw depth data obtained from a single frame of a 3D sensor. Additionally, our approach uses no explicit models of the objects and does not require a training phase. We use the grasping capability to demonstrate the application of a robot as an autonomous checkout clerk. To perform this task, the robot must identify how to grasp an object, locate the barcode on the object and read the numeric code. We evaluate our grasping algorithm in experiments where the robot was required to autonomously grasp unknown objects. The robot achieved a success of 91.6%in grasping novel objects. We performed two sets of experiments to evaluate the checkout robot application. In the first set, the objects were placed in many orientations in front of the robot one at a time. In the second set, the objects were placed several at a time with varying amounts of clutter. The robot was able to autonomously grasp and scan the objects in 49/50 of the single-object trials and 46/50 of the cluttered trials. <s> BIB023 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> In this paper we report on a recent public experiment that shows two robots making pancakes using web instructions. In the experiment, the robots retrieve instructions for making pancakes from the World Wide Web and generate robot action plans from the instructions. This task is jointly performed by two autonomous robots: The first robot opens and closes cupboards and drawers, takes a pancake mix from the refrigerator, and hands it to the robot B. The second robot cooks and flips the pancakes, and then delivers them back to the first robot. While the robot plans in the scenario are all percept-guided, they are also limited in different ways and rely on manually implemented sub-plans for parts of the task. We will thus discuss the potential of the underlying technologies as well as the research challenges raised by the experiment. <s> BIB024 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> We present a novel and simple experimental method called physical human interactive guidance to study human-planned grasping. Instead of studying how the human uses his/her own biological hand or how a human teleoperates a robot hand in a grasping task, the method involves a human interacting physically with a robot arm and hand, carefully moving and guiding the robot into the grasping pose, while the robot's configuration is recorded. Analysis of the grasps from this simple method has produced two interesting results. First, the grasps produced by this method perform better than grasps generated through a state-of-the-art automated grasp planner. Second, this method when combined with a detailed statistical analysis using a variety of grasp measures (physics-based heuristics considered critical for a good grasp) offered insights into how the human grasping method is similar or different from automated grasping synthesis techniques. Specifically, data from the physical human interactive guidance method showed that the human-planned grasping method provides grasps that are similar to grasps from a state-of-the-art automated grasp planner, but differed in one key aspect. The robot wrists were aligned with the object's principal axes in the human-planned grasps (termed low skewness in this paper), while the automated grasps used arbitrary wrist orientation. Preliminary tests show that grasps with low skewness were significantly more robust than grasps with high skewness (77-93%). We conclude with a detailed discussion of how the physical human interactive guidance method relates to existing methods to extract the human principles for physical interaction. <s> BIB025 </s> Data-Driven Grasp Synthesis - A Survey <s> VI. DISCUSSION AND CONCLUSION <s> Grasp quality metrics which analyze the contact wrench space are commonly used to synthesize and analyze preplanned grasps. Preplanned grasping approaches rely on the robustness of stored solutions. Analyzing the robustness of such solutions for large databases of preplanned grasps is a limiting factor for the applicability of data driven approaches to grasping. In this work, we will focus on the stability of the widely used grasp wrench space epsilon quality metric over a large range of poses in simulation. We examine a large number of grasps from the Columbia Grasp Database for the Barrett hand. We find that in most cases the grasp with the most robust force closure with respect to pose error for a particular object is not the grasp with the highest epsilon quality. We demonstrate that grasps can be reranked by an estimate of the stability of their epsilon quality. We find that the grasps ranked best by this method are successful more often in physical experiments than grasps ranked best by the epsilon quality. <s> BIB026
In the previous sections, we have discussed a number of different approaches towards synthesizing robotic grasps in a data-driven manner. We classified them according to the prior knowledge they assume to have about the object that the robot is supposed to grasp. In this section, we will attempt to order the most cited articles 1 into a rough time line and thereby identify recent trends in the field. Up to the year 2000, the field of robotic grasping was clearly dominated by analytic approaches BIB001  BIB002 BIB003 . Apart from e.g. Kamon et al. BIB004 , data-driven grasp synthesis started to become popular with the availability of GraspIt!  in 2004. Many highly cited approaches have been developed, analyzed and evaluated in this or other simulators BIB005 BIB007 BIB010 BIB006 BIB014  . These approaches differ in how grasp candidates are sampled from the infinite space of possibilities. For grasp ranking BIB022 they rely on metrics such as proposed by Ferrari and Canny BIB002 . Developing and evaluating approaches in simulation is attractive because the environment and its attributes can be completely controlled. A large number of experiments can be efficiently performed without having access to expensive robotics hardware that would also add a lot of complexity to the evaluation process. However, it is not clear if the simulated environment resembles the real world well enough to be able to transfer methods easily. Only recently, several articles BIB025 BIB026 have analyzed this question and come to the conclusion that the classic metrics are not good predictors for grasp success in the real world. For this reason, several researchers BIB008 BIB011 BIB015 proposed to let the robot learn how to grasp by experience that is gathered during grasp execution. Although, collecting examples is extremely time-consuming, the problem of transferring the learned model to the real robot is non-existant. A crucial question is how the object to be grasped is represented and how the experience is generalized to novel objects. Saxena et al. BIB012 pushed machine learning approaches for data-driven grasp synthesis even further. A simple logistic regressor was trained on large amounts of synthetic labeled training data to predict good grasping points in a monocular image. The authors demonstrated their method in a household scenario in which a robot emptied a dishwasher. None of the classical principles based on analytic formulations were used. This paper spawned a lot of research BIB013 BIB009 BIB017 BIB018 in which essentially one question is addressed: What are the object features that are sufficiently discriminative to infer a suitable grasp configuration? From 2009, there were further developments in the area of 3D sensing. Projected Texture Stereo was proposed by Konolige BIB019 . This technology is built into the sensor head of the PR2 [127] , a robot that is available to comparatively many robotics research labs and running on the OpenSource middle-ware ROS  . In 2010, Microsoft released the Kinect  , a highly accurate depth sensing device based on the technology developed by PrimeSense [130] . Due to its low price and simple usage, it became a ubiquitous device within the robotics community. Although the importance of 3D data for grasping has been previously recognized, many new approaches were proposed that operate on real world 3D data. They are either heuristics that map structures in this data to grasp configurations directly BIB020 BIB023 or they try to detect and recognize objects and estimate their pose BIB016  . Furthermore, we have recently seen an increasing amount of robots fulfilling very specific tasks such as towel folding BIB021 or preparing pancakes BIB024 . In these scenarios, grasping is embedded into a sequence of different manipulation actions.