Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> This paper describes a machine learning approach for visual object detection which is capable of processing images extremely rapidly and achieving high detection rates. This work is distinguished by three key contributions. The first is the introduction of a new image representation called the "integral image" which allows the features used by our detector to be computed very quickly. The second is a learning algorithm, based on AdaBoost, which selects a small number of critical visual features from a larger set and yields extremely efficient classifiers. The third contribution is a method for combining increasingly more complex classifiers in a "cascade" which allows background regions of the image to be quickly discarded while spending more computation on promising object-like regions. The cascade can be viewed as an object specific focus-of-attention mechanism which unlike previous approaches provides statistical guarantees that discarded regions are unlikely to contain the object of interest. In the domain of face detection the system yields detection rates comparable to the best previous systems. Used in real-time applications, the detector runs at 15 frames per second without resorting to image differencing or skin color detection. <s> BIB001 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> Finding appropriate stable grasps for a hand (either robotic or human) on an arbitrary object has proved to be a challenging and difficult problem. The space of grasping parameters coupled with the degrees-of-freedom and geometry of the object to be grasped creates a high-dimensional, non-smooth manifold. Traditional search methods applied to this manifold are typically not powerful enough to find appropriate stable grasping solutions, let alone optimal grasps. We address this issue in this paper, which attempts to find optimal grasps of objects using a grasping simulator. Our unique approach to the problem involves a combination of numerical methods to recover parts of the grasp quality surface with any robotic hand, and contemporary machine learning methods to interpolate that surface, in order to find the optimal grasp. <s> BIB002 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> Humanoid robots must be capable of interacting with the world using their hands. A variety of humanlike robot hands have been constructed, but it remains difficult to control these hands in a dexterous way. One challenge is grasp synthesis, where we wish to place the hand and control its shape to successfully grasp a given object. In this paper, we present a datadriven approach to grasp synthesis that treats grasping as a shape matching problem. We begin with a database of grasp examples. Given a model of a new object to be grasped (the query), shape features of the object are compared to shape features of hand poses in these examples in order to identify candidate grasps. For effective retrieval, we develop a novel shape matching algorithm that can accommodate the sparse shape information associated with hand pose and that considers relative placements of contact points and normals, which are important for grasp function. We illustrate our approach with examples using a model of the human hand <s> BIB003 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> Current approaches to visual object class detection mainly focus on the recognition of basic level categories, such as cars, motorbikes, mugs and bottles. Although these approaches have demonstrated impressive performance in terms of recognition, their restriction to these categories seems inadequate in the context of embodied, cognitive agents. Here, distinguishing objects according to functional aspects based on object affordances is important in order to enable manipulation of and interaction between physical objects and cognitive agent. ::: ::: In this paper, we propose a system for the detection of functional object classes, based on a representation of visually distinct hints on object affordances (affordance cues). It spans the complete range from tutordriven acquisition of affordance cues, learning of corresponding object models, and detecting novel instances of functional object classes in real images. <s> BIB004 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> This paper presents an efficient method to decide robust grasps given new objects using example-based learning. A robust grasp is a stable grasp, suitable for object manipulation. Adaptability to object manipulation is ensured by imitating the human choice of the object grasping component, its handle. Stability is obtained by computing contact points, ensuring force-closure property, on that handle. <s> BIB005 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> Grasping is a central issue of various robot applications, especially when unknown objects have to be manipulated by the system. In earlier work, we have shown the efficiency of 3D object shape approximation by box primitives for the purpose of grasping. A point cloud was approximated by box primitives [1]. In this paper, we present a continuation of these ideas and focus on the box representation itself. On the number of grasp hypotheses from box face normals, we apply heuristic selection integrating task, orientation and shape issues. Finally, an off-line trained neural network is applied to chose a final best hypothesis as the final grasp. We motivate how boxes as one of the simplest representations can be applied in a more sophisticated manner to generate task-dependent grasps. <s> BIB006 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> We consider the problem of grasping novel objects, specifically objects that are being seen for the first time through vision. Grasping a previously unknown object, one for which a 3-d model is not available, is a challenging problem. Furthermore, even if given a model, one still has to decide where to grasp the object. We present a learning algorithm that neither requires nor tries to build a 3-d model of the object. Given two (or more) images of an object, our algorithm attempts to identify a few points in each image corresponding to good locations at which to grasp the object. This sparse set of points is then triangulated to obtain a 3-d location at which to attempt a grasp. This is in contrast to standard dense stereo, which tries to triangulate every single point in an image (and often fails to return a good 3-d model). Our algorithm for identifying grasp locations from an image is trained by means of supervised learning, using synthetic images for the training set. We demonstrate this approach on two robotic manipulation platforms. Our algorithm successfully grasps a wide variety of objects, such as plates, tape rolls, jugs, cellphones, keys, screwdrivers, staplers, a thick coil of wire, a strangely shaped power horn and others, none of which were seen in the training set. We also apply our method to the task of unloading items from dishwashers. <s> BIB007 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> We consider the problem of grasping novel objects in cluttered environments. If a full 3-d model of the scene were available, one could use the model to estimate the stability and robustness of different grasps (formalized as form/force-closure, etc); in practice, however, a robot facing a novel object will usually be able to perceive only the front (visible) faces of the object. In this paper, we propose an approach to grasping that estimates the stability of different grasps, given only noisy estimates of the shape of visible portions of an object, such as that obtained from a depth sensor. By combining this with a kinematic description of a robot arm and hand, our algorithm is able to compute a specific positioning of the robot's fingers so as to grasp an object. ::: ::: We test our algorithm on two robots (with very different arms/manipulators, including one with a multifingered hand). We report results on the task of grasping objects of significantly different shapes and appearances than ones in the training set, both in highly cluttered and in uncluttered environments. We also apply our algorithm to the problem of unloading items from a dishwasher. <s> BIB008 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> This paper introduces a new approach to establish a good grasp for a novel object quickly. A comprehensive knowledge base for grasping is learned that takes into account the geometrical and physical knowledge of grasping. To automate the learning process as much as possible, learning happens in a virtual environment. We used the GraspIt! simulation environment with the Barrett hand for this work. As only approximate features of objects are used for training the grasping knowledge base (GKB), the knowledge gained is rather robust to object uncertainty. Based on the guidance of the GKB, a suitable grasp for a novel object can be found quickly. The newly gained grasping information of the new object can also be feedback to the GKB so that the knowledge base continues to improve as it is exposed to more grasping cases. The GKB serves as the ldquoexperiencerdquo of the robotic gripper to make grasping more and more skillful. We implemented the approach and tested it on This paper introduces a new approach to establish a good grasp for a novel object quickly. A comprehensive knowledge base for grasping is learned that takes into account the geometrical and physical knowledge of grasping. To automate the learning process as much as possible, learning happens in a virtual environment. We used the GraspIt! [T. Miller and P.K. Allen, 2000] simulation environment with the Barrett hand for this work. As only approximate features of objects are used for training the grasping knowledge base (GKB), the knowledge gained is rather robust to object uncertainty. Based on the guidance of the GKB, a suitable grasp for a novel object can be found quickly. The newly gained grasping information of the new object can also be feedback to the GKB so that the knowledge base continues to improve as it is exposed to more grasping cases. The GKB serves as the ldquoexperiencerdquo of the robotic gripper to make grasping more and more skillful. We implemented the approach and tested it on a wide variety of objects. The results show the effectiveness of this approach to achieve quick and good grasps of novel objects.a wide variety of objects. The results show the effectiveness of this approach to achieve quick and good grasps of novel objects. <s> BIB009 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> We consider the problem of grasping novel objects and its application to cleaning a desk. A recent successful approach applies machine learning to learn one grasp point in an image and a point cloud. Although those methods are able to generalize to novel objects, they yield suboptimal results because they rely on motion planner for finger placements. In this paper, we extend their method to accommodate grasps with multiple contacts. This approach works well for many human-made objects because it models the way we grasp objects. To further improve the grasping, we also use a method that learns the ranking between candidates. The experiments show that our method is highly effective compared to a state-of-the-art competitor. <s> BIB010 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> This paper presents work on vision based robotic grasping. The proposed method adopts a learning framework where prototypical grasping points are learnt from several examples and then used on novel objects. For representation purposes, we apply the concept of shape context and for learning we use a supervised learning approach in which the classifier is trained with labelled synthetic images. We evaluate and compare the performance of linear and non-linear classifiers. Our results show that a combination of a descriptor based on shape context with a non-linear classification algorithm leads to a stable detection of grasping points for a variety of objects. <s> BIB011 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> We consider the task of grasping novel objects and cleaning fairly cluttered tables with many novel objects. Recent successful approaches employ machine learning algorithms to identify points on the scene that the robot should grasp. In this paper, we show that the task can be significantly simplified by using segmentation, especially with depth information. A supervised localization method is employed to select graspable segments. We also propose a shape completion and grasp planner method which takes partial 3D information and plans the most stable grasping strategy. Extensive experiments on our robot demonstrate the effectiveness of our approach. <s> BIB012 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> We present a novel approach to measuring similarity between shapes and exploit it for object recognition. In our framework, the measurement of similarity is preceded by: (1) solving for correspondences between points on the two shapes; (2) using the correspondences to estimate an aligning transform. In order to solve the correspondence problem, we attach a descriptor, the shape context, to each point. The shape context at a reference point captures the distribution of the remaining points relative to it, thus offering a globally discriminative characterization. Corresponding points on two similar shapes will have similar shape contexts, enabling us to solve for correspondences as an optimal assignment problem. Given the point correspondences, we estimate the transformation that best aligns the two shapes; regularized thin-plate splines provide a flexible class of transformation maps for this purpose. The dissimilarity between the two shapes is computed as a sum of matching errors between corresponding points, together with a term measuring the magnitude of the aligning transform. We treat recognition in a nearest-neighbor classification framework as the problem of finding the stored prototype shape that is maximally similar to that in the image. Results are presented for silhouettes, trademarks, handwritten digits, and the COIL data set. <s> BIB013 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> A major challenge in modeling with BNs is learning the structure from both discrete and multivariate continuous data. A common approach in such situations is to discretize continuous data before structure learning. However efficient methods to discretize high-dimensional variables are largely lacking. This paper presents a novel method specifically aiming at discretization of high-dimensional, high-correlated data. The method consists of two integrated steps: non-linear dimensionality reduction using sparse Gaussian process latent variable models, and discretization by application of a mixture model. The model is fully probabilistic and capable to facilitate structure learning from discretized data, while at the same time retain the continuous representation. We evaluate the effectiveness of the method in the domain of robot grasping. Compared with traditional discretization schemes, our model excels both in task classification and prediction of hand grasp configurations. Further, being a fully probabilistic model it handles uncertainty in the data and can easily be integrated into other frameworks in a principled manner. <s> BIB014 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> Learning to grasp novel objects is an essential skill for robots operating in unstructured environments. We therefore propose a probabilistic approach for learning to grasp. In particular, we learn a function that predicts the success probability of grasps performed on surface points of a given object. Our approach is based on Markov Random Fields (MRF), and motivated by the fact that points that are geometrically close to each other tend to have similar grasp success probabilities. The MRF approach is successfully tested in simulation, and on a real robot using 3-D scans of various types of objects. The empirical results show a significant improvement over methods that do not utilize the smoothness assumption and classify each point separately from the others. <s> BIB015 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> This thesis introduces a new framework for data-driven grasping. We assume, with neuropsychological justification, that human grasping is fundamentally example based rather than rule based. This means that planning grasps for a novel object will usually reduce to identifying it as similar to known objects with known grasp affordances. Our framework is intended to allow robots to mimic this approach. ::: For robots to succeed in the real world, it is essential that they be able to grasp objects based on realistically available sensor data. However, most existing grasp planners assume that the robot has access to the full 3D geometry of all objects to be grasped, which is unscalable, or abandon 3D geometry entirely to plan grasps based on appearance, which is difficult to extend to dexterous hands. The core advantage of our data-driven framework is that it naturally allows grasps to be planned for partially sensed objects. We accomplish this by using the partial sensor data to find similar 3D models, which can be used as proxy geometries for grasp planning. ::: Along with the framework, we present a new set of shape descriptors suitable for matching partial sensor data to similar - but not identical - 3D models. This is in contrast to most previous descriptors for partial matching, which tend to rely on local feature correspondences that will often not exist in our problem setting. In a similar vein we also present new algorithms for aligning the pose and scale of partial sensor data to the best matching models, where no local correspondences may be assumed to exist. ::: Our grasp planner makes use of a grasp database, consisting of example grasps for a large number of 3D models. As no such database has previously existed, this thesis introduces the Columbia Grasp Database, a freely available collection of hundreds of thousands of grasps for thousands of 3D models using a variety of robotic hands. To construct this database we modified the Eigengrasp grasp planner, which uses a low dimensional control space to simplify the grasp search space. We also discuss some limitations of this planner and show how they can be addressed by model decomposition. ::: Our use of a database of 3D models annotated with precomputed grasps suggests the possibility of annotating the models with other forms of information as well. With this in mind, we show how to leverage noisy user data downloaded from the internet to suggest likely text tags for previously unlabeled 3D models. Although this work has not yet been applied to the problem of grasp planning, we demonstrate a content-based 3D model search engine that implements our automatic labeling algorithm. ::: The title of this thesis, “Data-Driven Grasping,” represents a vision of robotic grasping that is larger than our particular implementation. Instead, the major contribution of this thesis is bridging the worlds of robotics and 3D model similarity search. Content based search is an important area of research in its own right, but the combination of content based search and robotics is particularly exciting because it opens up the possibility of using the entire internet as a knowledge base for intelligent robots. <s> BIB016 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> In this article we describe an object perception system for autonomous robots performing everyday manipulation tasks in kitchen environments. The perception system gains its strengths by exploiting that the robots are to perform the same kinds of tasks with the same objects over and over again. It does so by learning the object representations necessary for the recognition and reconstruction in the context of pick-and-place tasks. The system employs a library of specialized perception routines that solve different, well-defined perceptual sub-tasks and can be combined into composite perceptual activities including the construction of an object model database, multimodal object classification, and object model reconstruction for grasping. We evaluate the effectiveness of our methods, and give examples of application scenarios using our personal robotic assistants acting in a human living environment. <s> BIB017 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> The paper starts by reviewing the challenges associated to grasp planning, and previous work on robot grasping. Our review emphasizes the importance of agents that generalize grasping strategies across objects, and that are able to transfer these strategies to novel objects. In the rest of the paper, we then devise a novel approach to the grasp transfer problem, where generalization is achieved by learning, from a set of grasp examples, a dictionary of object parts by which objects are often grasped. We detail the application of dimensionality reduction and unsupervised clustering algorithms to the end of identifying the size and shape of parts that often predict the application of a grasp. The learned dictionary allows our agent to grasp novel objects which share a part with previously seen objects, by matching the learned parts to the current view of the new object, and selecting the grasp associated to the best-fitting part. We present and discuss a proof-of-concept experiment in which a dictionary is learned from a set of synthetic grasp examples. While prior work in this area focused primarily on shape analysis (parts identified, e.g., through visual clustering, or salient structure analysis), the key aspect of this work is the emergence of parts from both object shape and grasp examples. As a result, parts intrinsically encode the intention of executing a grasp. <s> BIB018 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> We present a method for transferring grasps between objects of the same functional category. This transfer is intended to preserve the functionality of a grasp constructed for one of the objects, thus enabling the analogous action to be performed on a novel object for which no grasp has been specified. Manipulation knowledge is hence generalized from a single example to a class of objects with a significant amount of shape variability. The transfer is achieved through warping the surface geometry of the source object onto the target object, and along with it the contact points of a grasp. The warped contacts are locally replanned, if necessary, to ensure grasp stability, and a suitable grasp pose is computed. We present extensive results of experiments with a database of four-finger grasps, designed to systematically cover variations on grasping the mugs of the Princeton Shape Benchmark. <s> BIB019 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> In this paper, we present an approach towards autonomous grasping of objects according to their category and a given task. Recent advances in the field of object segmentation and categorization as well as task-based grasp inference have been leveraged by integrating them into one pipeline. This allows us to transfer task-specific grasp experience between objects of the same category. The effectiveness of the approach is demonstrated on the humanoid robot ARMAR-IIIa. <s> BIB020 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> In this paper we address the problem of grasp generation and grasp transfer between objects using categorical knowledge. The system is built upon an i) active scene segmentation module, able of generating object hypotheses and segmenting them from the background in real time, ii) object categorization system using integration of 2D and 3D cues, and iii) probabilistic grasp reasoning system. Individual object hypotheses are first generated, categorized and then used as the input to a grasp generation and transfer system that encodes task, object and action properties. The experimental evaluation compares individual 2D and 3D categorization approaches with the integrated system, and it demonstrates the usefulness of the categorization in task-based grasping and grasp transfer. <s> BIB021 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> We design an example based planning framework to generate semantic grasps, stable grasps that are functionally suitable for specific object manipulation tasks. We propose to use partial object geometry, tactile contacts, and hand kinematic data as proxies to encode semantic constraints, which are task-related constraints. We introduce a semantic affordance map, which relates local geometry to a set of predefined semantic grasps that are appropriate to different tasks. Using this map, the pose of a robotic hand can be estimated so that the hand is adjusted to achieve the ideal approach direction required by a particular task. A grasp planner is then used to generate a set of final grasps which have appropriate stability, tactile contacts, and hand kinematics along this approach direction. We show experiments planning semantic grasps on everyday objects and executing these grasps with a physical robot. <s> BIB022 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> This paper presents a novel approach to emptying a basket filled with a pile of objects. Form, size, position, orientation and constellation of the objects are unknown. Additional challenges are to localize the basket and treat it as an obstacle, and to cope with incomplete point cloud data. There are three key contributions. First, we introduce Height Accumulated Features (HAF) which provide an efficient way of calculating grasp related feature values. The second contribution is an extensible machine learning system for binary classification of grasp hypotheses based on raw point cloud data. Finally, a practical heuristic for selection of the most robust grasp hypothesis is introduced. We evaluate our system in experiments where a robot was required to autonomously empty a basket with unknown objects on a pile. Despite the challenging scenarios, our system succeeded each time. <s> BIB023 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> One of the basic skills for a robot autonomous grasping is to select the appropriate grasping point for an object. Several recent works have shown that it is possible to learn grasping points from different types of features extracted from a single image or from more complex 3D reconstructions. In the context of learning through experience, this is very convenient, since it does not require a full reconstruction of the object and implicitly incorporates kinematic constraints as the hand morphology. These learning strategies usually require a large set of labeled examples which can be expensive to obtain. In this paper, we address the problem of actively learning good grasping points to reduce the number of examples needed by the robot. The proposed algorithm computes the probability of successfully grasping an object at a given location represented by a feature vector. By autonomously exploring different feature values on different objects, the systems learn where to grasp each of the objects. The algorithm combines beta-binomial distributions and a non-parametric kernel approach to provide the full distribution for the probability of grasping. This information allows to perform an active exploration that efficiently learns good grasping points even among different objects. We tested our algorithm using a real humanoid robot that acquired the examples by experimenting directly on the objects and, therefore, it deals better with complex (anthropomorphic) hand-object interactions whose results are difficult to model, or predict. The results show a smooth generalization even in the presence of very few data as is often the case in learning through experience. <s> BIB024 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Discriminative Approaches <s> We present a real-world robotic agent that is capable of transferring grasping strategies across objects that share similar parts. The agent transfers grasps across objects by identifying, from examples provided by a teacher, parts by which objects are often grasped in a similar fashion. It then uses these parts to identify grasping points onto novel objects. We focus our report on the definition of a similarity measure that reflects whether the shapes of two parts resemble each other, and whether their associated grasps are applied near one another. We present an experiment in which our agent extracts five prototypical parts from thirty-two real-world grasp examples, and we demonstrate the applicability of the prototypical parts for grasping novel objects. <s> BIB025
First, there are approaches that learn a discriminative function to distinguish between good and bad grasp configurations. They mainly differ in what object features are used and thereby in the space over which objects are considered similar. Furthermore, they parameterize grasp candidates differently. Many of them only consider whether a specific part of the object is graspable or not. Others also learn multiple contact points or full grasp configurations. A flow-chart for the approaches discussed in the following is presented in Fig. 9 . 1) Based on 3D Data: El-Khoury and Sahbani BIB005 distinguishes between graspable and non-graspable parts of an object. A point cloud of an object is segmented into parts and each part then approximated by a superquadric (SQ). An artificial neural network (ANN) is used to classify whether or not the part is prehensile. The ANN is trained offline on Figure 9 : Typical functional flow-chart of a system that learns from labeled examples. In the offline learning phase a database is available consisting of a set of objects labeled with grasp configurations and their quality. Database entries are analyzed to extract relations between specific features and the grasps. The result is a learned model that given some features can predict grasp qualities. In the online phase, the scene is segmented and features are extracted from the scene. Given this, the model outputs a ranked set of promising grasp hypotheses. Unreachable grasps are filtered out and the best is executed. The following approaches use labeled training examples: BIB014 BIB003 BIB005 BIB006 BIB018 BIB025  BIB015 BIB004 BIB007 BIB008 BIB010  BIB019 BIB011 BIB020 BIB009 BIB016 BIB017 BIB021 BIB002 BIB022 human-labeled SQs. If one of the object parts is classified as prehensile, an n-fingered force-closure grasp is synthesized on this object part. Grasp experience is therefore only used to decide where to apply a grasp, not how the grasp should be configured. These steps are shown for two objects in Fig. 10 . Pelossof et al. BIB002 approximate an object with a a single SQ. Given this, their goal is to find a suitable grasp configuration for a Barrett hand consisting of the approach vector, wrist orientation and finger spread. A Support Vector Machine (SVM) is trained on data consisting of feature vectors containing the SQ parameters and a grasp configuration. They are labeled with a scalar estimating the grasp quality. This training data is shown in Fig. 11 . When feeding the SVM only with the shape parameters of the SQ, their algorithm searches efficiently through the grasp configuration space for parameters that maximize the grasp quality. Both of the aforementioned approaches are evaluated in simulation where the central assumption is that accurate and detailed 3D object models are available: an assumption not always valid. An SQ is an attractive 3D representation due to its low number of parameters and high shape variability. However, it remains unclear whether an SQ could equally well approximate object shape when given real-world sensory data that is noisy and incomplete. Hübner and Kragic BIB006 decompose a point cloud into a constellation of boxes. The simple geometry of a box reduces the number of potential grasps significantly. To decide which of the sides of the boxes provides a good grasp, an ANN is trained offline on synthetic data. The projection of the point cloud inside a box to its sides provides the input to the ANN. An example is shown in Fig. 12 . The training data consists of a set of these projections from different objects labeled with the grasp quality metrics. These are computed by GraspIt! while performing the according grasps. Boularias et al. BIB015 model an object as a Markov Random Field (MRF) in which the nodes are points in a point cloud and edges are spanned between the six nearest neighbors of a point. The features of a node describe the local point distribution Figure 13 : Filters used to compute the feature vector of image patches in BIB007 . (a) (b) Figure 14 : Labeled training data. 14a) One example for each of the eight object classes in training data in BIB007 along with their grasp labels (in yellow). 14b) Positive (red) and negative examples (blue) for grasping points BIB015 . around that node. A node in the MRF can carry either one of two labels: a good or a bad grasp location. The goal of the approach is to find the maximum a-posteriori labeling of point clouds for new objects. Very little training data is used which is shown in Fig. 14b . A handle serves as a positive example. The experiments show that this leads to a robust labeling of 3D object parts that are very similar to a handle. Although both approaches BIB006 BIB015 also rely on 3D models for learning, the authors show examples for real sensor data. It remains unclear how well the classifiers would generalize to a larger set of object categories and real sensor data. Fischinger and Vincze BIB023 propose a height-accumulated feature that is similar to Haar basis functions as successfully applied by e.g. Viola and Jones BIB001 for face detection. The values of the feature are computed based on the height of objects above e.g. the table plane. Positive and negative examples are used to train an SVM that distinguishes between good and bad grasping points. The authors demonstrate their approach for cleaning cluttered scenes. No object segmentation is required for the approach. 2) Based on 2D Data: There are number of experiencebased approaches that avoid the complexity of 3D data and mainly rely on 2D data to learn to discriminate between good and bad grasp locations. Saxena et al. BIB007 propose a system that infers a point at where to grasp an object directly as a function of its image. The authors apply machine learning to train a grasping point model on labeled synthetic images of a number of different objects. The classification is based on a feature vector containing local appearance cues regarding color, texture and edges of an image patch in several scales and of its neighboring patches. The filters used to compute some of the features are shown in Fig. 13 . Samples from the labeled training data are shown in Fig. 14a . The system was used successfully to pick up objects from a dishwasher after it has been additionally trained for this scenario. Instead of assuming the availability of a labeled data set, Figure 15 : Scene segmentation with (right) and without (left) depth information. BIB012 Montesano and Lopes BIB024 allow the robot to autonomously explore which features encode graspability. Similar to BIB007 , simple 2D filters are used that can be rapidly convolved with an image. Given features from a region, the robot can compute the posterior probability that a grasp applied to this location will be successful. It is modeled as a Beta distribution and estimated from the grasping trials executed by the robot and their outcome. Furthermore, the variance of the posterior can be used to guide exploration to regions that are predicted to have a high success rate but are still uncertain. Another example of a system involving 2D data and grasp experience is presented by Stark et al. BIB004 . Here, an object is represented by a composition of prehensile parts. These so called affordance cues are obtained by observing the interaction of a person with a specific object. Grasp hypotheses for new stimuli are inferred by matching features of that object against a codebook of learned affordance cues that are stored along with relative object position and scale. How to grasp the detected parts is not solved since hand orientation and finger configuration are not inferred from the affordance cues. Similar to Boularias et al. BIB015 , especially locally very discriminative structures like handles are well detected. 3) Integrating 2D and 3D Data: Although the above approaches have been demonstrated to work well in specific manipulation scenarios, inferring a full grasp configuration from 2D data alone is a highly under-constrained problem. Regions in the image may have very similar visual features but afford completely different grasps. The following approaches integrate multiple complementary modalities, 2D and 3D visual data and their local or global characteristics, to learn a function that can take more parameters of a grasp into account. Saxena et al. BIB008 extend their previous work on inferring 2D grasping points by taking the 3d point distribution within a sphere centered around a grasp candidate into account. This enhances the prediction of a stable grasp and also allows for the inference of grasp parameters like approach vector and finger spread. In earlier work BIB007 , only downward or outward grasp with a fixed pinch grasp configuration were possible. Rao et al. BIB012 distinguish between graspable and nongraspable object hypotheses in a scene. Using a combination of 2D and 3D features, an SVM is trained on labeled data of segmented objects. Among those features are for example the variance in depth and height as well as variance of the three channels in the Lab color space. These are some kind of meta features that are used instead of the values of e.g. the color channels directly. Rao et al. BIB012 achieve good classification rates on object hypotheses formed by segmentation on color and depth cues. Fig. 15 compares the segmentation results with and without taking depth into account. Le et al. BIB010 model grasp hypotheses as consisting of two contact points. They apply a learning approach to rank a sampled set of fingertip positions according to graspability. The feature vector consists of a combination of 2D and 3D cues such as gradient angle or depth variation along the line connecting the two grasping points. Example grasp candidates are shown in Fig. 16 . Bohg and Kragic BIB011 propose an approach that instead of using local features, encodes global 2D object shape. It is represented relative to a potential grasping point by shape contexts as introduced by Belongie et al. BIB013 . Fig. 17 shows a potential grasping point and the associated feature. Similar to Saxena et al. BIB008 , Bergström et al.  see the result of the 2D based grasp selection as a way to search in a 3D object representation for a full grasp configuration. The authors extend their previous approach BIB011 to work on a sparse edge-based object representation. They show that integrating 3D and 2D based methods for grasp hypotheses generation results in a sparser set of grasps with a good quality. Different from the above approaches, Ramisa et al.  consider the problem of manipulating deformable objects, specifically folding shirts. They aim at detecting the shirt collars that exhibit deformability but also have distinct features. The authors show that a combination of local 2D and 3D descriptors works well for this task. Results are presented in terms of how reliable collars can be detected when only a single shirt or several shirts are present in the scene.