Data-Driven Grasp Synthesis - A Survey <s> A. Open Problems <s> Vision and manipulation are inextricably intertwined in the primate brain. Tantalizing results from neuroscience are shedding light on the mixed motor and sensory representations used by the brain during reaching, grasping, and object recognition. We now know a great deal about what happens in the brain during these activities, but not necessarily why. Is the integration we see functionally important, or just a reection of evolution’s lack of enthusiasm for sharp modularity? We wish to instantiate these results in robotic form to probe their technical advantages and to nd any lacunae in existing models. We believe it would be missing the point to investigate this on a platform where dextrous manipulation and sophisticated machine vision are already implemented in their mature form, and instead follow a developmental approach from simpler primitives. We begin with a precursor to manipulation, simple poking and prodding, and show how it facilitates object segmentation, a long-standing problem in machine vision. The robot can familiarize itself with the objects in its environment by acting upon them. It can then recognize other actors (such as humans) in the environment through their eect on the objects it has learned about. We argue that following causal chains of events out from the robot’s body into the environment allows for a very natural developmental progression of visual competence, and relate this idea to results in neuroscience. <s> BIB001 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Open Problems <s> Robust vision-based grasping is still a hard problem for humanoid robot systems. When being restricted to using the camera system built-in into the robot's head for object localization, the scenarios get often very simplified in order to allow the robot to grasp autonomously. Within the computer vision community, many object recognition and localization systems exist, but in general, they are not tailored to the application on a humanoid robot. In particular, accurate 6D object localization in the camera coordinate system with respect to a 3D rigid model is crucial for a general framework for grasping. While many approaches try to avoid the use of stereo calibration, we will present a system that makes explicit use of the stereo camera system in order to achieve maximum depth accuracy. Our system can deal with textured objects as well as objects that can be segmented globally and are defined by their shape. Thus, it covers the cases of objects with complex texture and complex shape. Our work is directly linked to a grasping framework being implemented on the humanoid robot ARM AR and serves as its perception module for various grasping and manipulation experiments in a kitchen scenario. <s> BIB002 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Open Problems <s> We describe a process in which the segmentation of objects as well as the extraction of the object shape becomes realized through active exploration of a robot vision system. In the exploration process, two behavioral modules that link robot actions to the visual and haptic perception of objects interact. First, by making use of an object independent grasping mechanism, physical control over potential objects can be gained. Having evaluated the initial grasping mechanism as being successful, a second behavior extracts the object shape by making use of prediction based on the motion induced by the robot. This also leads to the concept of an "object" as a set of features that change predictably over different frames. The system is equipped with a certain degree of generic prior knowledge about the world in terms of a sophisticated visual feature extraction process in an early cognitive vision system, knowledge about its own embodiment as well as knowledge about geometric relationships such as rigid body motion. This prior knowledge allows the extraction of representations that are semantically richer compared to many other approaches. <s> BIB003 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Open Problems <s> This paper proposes a novel 3D scene interpretation approach for robots in mobile manipulation scenarios using a set of 3D point features (Fast Point Feature Histograms) and probabilistic graphical methods (Conditional Random Fields). Our system uses real time stereo with textured light to obtain dense depth maps in the robot's manipulators working space. For the purposes of manipulation, we want to interpret the planar supporting surfaces of the scene, recognize and segment the object classes into their primitive parts in 6 degrees of freedom (6DOF) so that the robot knows what it is attempting to use and where it may be handled. The scene interpretation algorithm uses a two-layer classification scheme: i) we estimate Fast Point Feature Histograms (FPFH) as local 3D point features to segment the objects of interest into geometric primitives; and ii) we learn and categorize object classes using a novel Global Fast Point Feature Histogram (GFPFH) scheme which uses the previously estimated primitives at each point. To show the validity of our approach, we analyze the proposed system for the problem of recognizing the object class of 20 objects in 500 table settings scenarios. Our algorithm identifies the planar surfaces, decomposes the scene and objects into geometric primitives with 98.27% accuracy and uses the geometric primitives to identify the object's class with an accuracy of 96.69%. <s> BIB004 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Open Problems <s> We present an object representation framework that encodes probabilistic spatial relations between 3D features and organizes these features in a hierarchy. Features at the bottom of the hierarchy are bound to local 3D descriptors. Higher level features recursively encode probabilistic spatial configurations of more elementary features. The hierarchy is implemented in a Markov network. Detection is carried out by a belief propagation algorithm, which infers the pose of high-level features from local evidence and reinforces local evidence from globally consistent knowledge, effectively producing a likelihood for the pose of the object in the detection scene. We also present a simple learning algorithm that autonomously builds hierarchies from local object descriptors. We explain how to use our framework to estimate the pose of a known object in an unknown scene. Experiments demonstrate the robustness of hierarchies to input noise, viewpoint changes, and occlusions. <s> BIB005 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Open Problems <s> To perform successful manipulation, robots depend on information about objects in their environment. In unstructured environments, such information cannot be given to the robot a priori. It is thus critical for the robot to be able to continuously acquire task-specific information about objects. Towards this goal, we present a robust perceptual skill for identifying, tracking, and segmenting objects in a cluttered environment. We increase the robot's perceptual capabilities by closely coupling them with the robot's manipulation skills. The robot's interaction with objects in the environment creates a perceptual signal, i.e. motion, that renders segmentation and tracking robust and reliable. In addition, the resulting perceptual signal reveals the type of segmentation most relevant to manipulation, namely a segmentation of rigidly connected physical bodies. We demonstrate our approach with experiments on a real world mobile manipulation platform with multiple objects in a cluttered scene. <s> BIB006 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Open Problems <s> This paper addresses the problem of robot grasping in conditions of uncertainty. We propose a grasp controller that deals robustly with this uncertainty using feedback from different contact-based sensors. This controller assumes a description of grasp consisting of a primitive that only determines the initial configuration of the hand and the control law to be used. We exhaustively validate the controller by carrying out a large number of tests with different degrees of inaccuracy in the pose of the target objects and by comparing it with results of a naive grasp controller. <s> BIB007 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Open Problems <s> Robotic grasping in unstructured environments requires the ability to select grasps for unknown objects and execute them while dealing with uncertainty due to sensor noise or calibration errors. In this work, we propose a simple but robust approach to grasp selection for unknown objects, and a reactive adjustment approach to deal with uncertainty in object location and shape. The grasp selection method uses 3D sensor data directly to determine a ranked set of grasps for objects in a scene, using heuristics based on both the overall shape of the object and its local features. The reactive grasping approach uses tactile feedback from fingertip sensors to execute a compliant robust grasp. We present experimental results to validate our approach by grasping a wide range of unknown objects. Our results show that reactive grasping can correct for a fair amount of uncertainty in the measured position or shape of the objects, and that our grasp selection approach is successful in grasping objects with a variety of shapes. <s> BIB008 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Open Problems <s> We present MOPED, a framework for Multiple Object Pose Estimation and Detection that seamlessly integrates single-image and multi-image object recognition and pose estimation in one optimized, robust, and scalable framework. We address two main challenges in computer vision for robotics: robust performance in complex scenes, and low latency for real-time operation. We achieve robust performance with Iterative Clustering Estimation (ICE), a novel algorithm that iteratively combines feature clustering with robust pose estimation. Feature clustering quickly partitions the scene and produces object hypotheses. The hypotheses are used to further refine the feature clusters, and the two steps iterate until convergence. ICE is easy to parallelize, and easily integrates single- and multi-camera object recognition and pose estimation. We also introduce a novel object hypothesis scoring function based on M-estimator theory, and a novel pose clustering algorithm that robustly handles recognition outliers. We achieve scalability and low latency with an improved feature matching algorithm for large databases, a GPU/CPU hybrid architecture that exploits parallelism at all levels, and an optimized resource scheduler. We provide extensive experimental results demonstrating state-of-the-art performance in terms of recognition, scalability, and latency in real-world robotic applications. <s> BIB009 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Open Problems <s> We propose a method for interactive modeling of objects and object relations based on real-time segmentation of video sequences. In interaction with a human, the robot can perform multi-object segmentation through principled modeling of physical constraints. The key contribution is an efficient multi-labeling framework, that allows object modeling and disambiguation in natural scenes. Object modeling and labeling is done in a real-time segmentation system, to which hypotheses and constraints denoting relations between objects can be added incrementally. Through instructions such as key presses or spoken words, a scene can be segmented in regions corresponding to multiple physical objects. The approach solves some of the difficult problems related to disambiguation of objects merged due to their direct physical contact. Results show that even a limited set of simple interactions with a human operator can substantially improve segmentation results. <s> BIB010 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Open Problems <s> This paper reports on a new approach for segmentation and learning of new, unknown objects with a humanoid robot. No prior knowledge about the objects or the environment is needed. The only necessary assumptions are firstly, that the object has a (partly) smooth surface that contains some distinctive visual features and secondly, that the object moves as a rigid body. The robot uses both its visual and manipulative capabilities to segment and learn unknown objects in unknown environments. The segmentation algorithm is based on pushing hypothetical objects by the robot, which provides a sufficient amount of information to distinguish the object from the background. In the case of a successful segmentation, additional features are associated with the object over several pushing-and-verification iterations. The accumulated features are used to learn the appearance of the object from multiple viewing directions. We show that the learned model, in combination with the proposed segmentation process, allows robust object recognition in cluttered scenes. <s> BIB011 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Open Problems <s> A major challenge in modeling with BNs is learning the structure from both discrete and multivariate continuous data. A common approach in such situations is to discretize continuous data before structure learning. However efficient methods to discretize high-dimensional variables are largely lacking. This paper presents a novel method specifically aiming at discretization of high-dimensional, high-correlated data. The method consists of two integrated steps: non-linear dimensionality reduction using sparse Gaussian process latent variable models, and discretization by application of a mixture model. The model is fully probabilistic and capable to facilitate structure learning from discretized data, while at the same time retain the continuous representation. We evaluate the effectiveness of the method in the domain of robot grasping. Compared with traditional discretization schemes, our model excels both in task classification and prediction of hand grasp configurations. Further, being a fully probabilistic model it handles uncertainty in the data and can easily be integrated into other frameworks in a principled manner. <s> BIB012 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Open Problems <s> Personal robots can only become widespread if they are capable of safely operating among humans. In uncertain and highly dynamic environments such as human households, robots need to be able to instantly adapt their behavior to unforseen events. In this paper, we propose a general framework to achieve very contact-reactive motions for robotic grasping and manipulation. Associating stereotypical movements to particular tasks enables our system to use previous sensor experiences as a predictive model for subsequent task executions. We use dynamical systems, named Dynamic Movement Primitives (DMPs), to learn goal-directed behaviors from demonstration. We exploit their dynamic properties by coupling them with the measured and predicted sensor traces. This feedback loop allows for online adaptation of the movement plan. Our system can create a rich set of possible motions that account for external perturbations and perception uncertainty to generate truly robust behaviors. As an example, we present an application to grasping with the WAM robot arm. <s> BIB013 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Open Problems <s> In this paper, we argue that there is a strong correlation between local 3D structure and object placement in everyday scenes. We call this the 3D context of the object. In previous work, this is typically hand-coded and limited to flat horizontal surfaces. In contrast, we propose to use a more general model for 3D context and learn the relationship between 3D context and different object classes. This way, we can capture more complex 3D contexts without implementing specialized routines. We present extensive experiments with both qualitative and quantitative evaluations of our method for different object classes. We show that our method can be used in conjunction with an object detection algorithm to reduce the rate of false positives. Our results support that the 3D structure surrounding objects in everyday scenes is a strong indicator of their placement and that it can give significant improvements in the performance of, for example, an object detection system. For evaluation, we have collected a large dataset of Microsoft Kinect frames from five different locations, which we also make publicly available. <s> BIB014 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Open Problems <s> In this paper, we present an efficient 3D object recognition and pose estimation approach for grasping procedures in cluttered and occluded environments. In contrast to common appearance-based approaches, we rely solely on 3D geometry information. Our method is based on a robust geometric descriptor, a hashing technique and an efficient, localized RANSAC-like sampling strategy. We assume that each object is represented by a model consisting of a set of points with corresponding surface normals. Our method simultaneously recognizes multiple model instances and estimates their pose in the scene. A variety of tests shows that the proposed method performs well on noisy, cluttered and unsegmented range scans in which only small parts of the objects are visible. The main procedure of the algorithm has a linear time complexity resulting in a high recognition speed which allows a direct integration of the method into a continuous manipulation task. The experimental validation with a seven-degree-of-freedom Cartesian impedance controlled robot shows how the method can be used for grasping objects from a complex random stack. This application demonstrates how the integration of computer vision and soft-robotics leads to a robotic system capable of acting in unstructured and occluded environments. <s> BIB015 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Open Problems <s> The community-based generation of content has been tremendously successful in the World Wide Web - people help each other by providing information that could be useful to others. We are trying to transfer this approach to robotics in order to help robots acquire the vast amounts of knowledge needed to competently perform everyday tasks. RoboEarth is intended to be a web community by robots for robots to autonomously share descriptions of tasks they have learned, object models they have created, and environments they have explored. In this paper, we report on the formal language we developed for encoding this information and present our approaches to solve the inference problems related to finding information, to determining if information is usable by a robot, and to grounding it on the robot platform. <s> BIB016 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Open Problems <s> We design an example based planning framework to generate semantic grasps, stable grasps that are functionally suitable for specific object manipulation tasks. We propose to use partial object geometry, tactile contacts, and hand kinematic data as proxies to encode semantic constraints, which are task-related constraints. We introduce a semantic affordance map, which relates local geometry to a set of predefined semantic grasps that are appropriate to different tasks. Using this map, the pose of a robotic hand can be estimated so that the hand is adjusted to achieve the ideal approach direction required by a particular task. A grasp planner is then used to generate a set of final grasps which have appropriate stability, tactile contacts, and hand kinematics along this approach direction. We show experiments planning semantic grasps on everyday objects and executing these grasps with a physical robot. <s> BIB017 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Open Problems <s> This paper presents a model based approach to autonomous dexterous manipulation, developed as part of the DARPA Autonomous Robotic Manipulation (ARM) program. The developed autonomy system uses robot, object, and environment models to identify and localize objects, and well as plan and execute required manipulation tasks. Deliberate interaction with objects and the environment increases system knowledge about the combined robot and environmental state, enabling high precision tasks such as key insertion to be performed in a consistent framework. This approach has been demonstrated across a wide range of manipulation tasks, and in independent DARPA testing archived the most successfully completed tasks with the fastest average task execution of any evaluated team. <s> BIB018 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Open Problems <s> We address the problem of grasping everyday objects that are small relative to an anthropomorphic hand, such as pens, screwdrivers, cellphones, and hammers from their natural poses on a support surface, e.g., a table top. In such conditions, state of the art grasp generation techniques fail to provide robust, achievable solutions due to either ignoring or trying to avoid contact with the support surface. In contrast, we show that contact with support surfaces is critical for grasping small objects. This also conforms with our anecdotal observations of human grasping behaviors. We develop a simple closed-loop hybrid controller that mimics this interactive, contact-rich strategy by a position-force, pre-grasp and landing strategy for finger placement. The approach uses a compliant control of the hand during the grasp and release of objects in order to preserve safety. We conducted extensive grasping experiments on a variety of small objects with similar shape and size. The results demonstrate that our approach is robust to localization uncertainties and applies to many everyday objects. <s> BIB019 </s> Data-Driven Grasp Synthesis - A Survey <s> A. Open Problems <s> We study visual servoing in a framework of detection and grasping of unknown objects. Classically, visual servoing has been used for applications where the object to be servoed on is known to the robot prior to the task execution. In addition, most of the methods concentrate on aligning the robot hand with the object without grasping it. In our work, visual servoing techniques are used as building blocks in a system capable of detecting and grasping unknown objects in natural scenes. We show how different visual servoing techniques facilitate a complete grasping cycle. <s> BIB020
We have identified four major areas that form open problems in the area of robotic grasping: 1) Object Segmentation: Many of the approaches that are mentioned in this survey usually assume that the object to be grasped is already segmented from the background. Since segmentation is a very hard problem in itself, many methods make the simplifying assumption that objects are standing on a planar surface. Detecting this surface in a 3D point cloud and performing Euclidean clustering results in a set of segmented point clouds that serve as object hypotheses BIB004 . Although the dominant surface assumption is viable in certain scenarios and to shortcut the problem of segmentation, we believe that we need a more general approach to solve this. First of all, some objects might usually occur in a specific spatial context. This can be on a planar surface, but it might also be on a shelf or in the fridge. Aydemir and Jensfelt BIB014 propose to learn this context for each known object to guide the search for them. One could also imagine that this context could help segmenting foreground from background. Furthermore, there are model-based object detection methods BIB002  BIB005 BIB009 BIB015 that can segment a scene as a by-product of detection and without making strong assumptions about the environment. In case of unknown objects, some methods have been proposed that employ the interaction capabilities of a robot, e.g. visual fixation or pushing movements with the robot hand, to segment the scene BIB001 BIB003 BIB006 BIB010 BIB011 . A general solution towards object segmentation might be a combination of these two methods. The robot first interacts with objects to acquire a model. Once it has an object model, it can be used for detecting and thereby segmenting it from the background. Once again, how the object is represented plays a crucial role in the segmentation problem. 2) Learning to Grasp: Let us consider the goal of having a robotic companion helping us in our household. In this scenario, we cannot expect that the programmer has foreseen all the different situations that this robot will be confronted with. Therefore, the ideal household robot should have the ability to continuously learn about new objects and how to manipulate them while it is operating in the environment. In the future, we will also not be able to rely on having 3D models readily available of all objects the robot could possibly encounter. This requires the ability to learn a model that could generalize from previous experience to new situations. Many open questions arise: How is the experience regarding one object and grasp represented in memory? How can success and failure be autonomously quantified? How can a model be learned from this experience that would generalize to new situations? Should it be a discriminative, a generative or exemplar-based model? What are the features that encode object affordances? Can these be autonomously learned? In which space are we comparing new objects to already encountered ones? Can we bootstrap learning by using simulation or by human demonstration? The methods that we have discussed in Section III about grasping familiar objects approach these questions. However, we are still far from a method that answers all of them in a satisfying way. 3) Autonomous Manipulation Planning: Recently, more complex scenarios than just grasping from a table top have been approached by a number of research labs. How a robot can autonomously sequence a set of actions to perform such a task is still an open problem. Towards this end, Tenorth et al. BIB016 propose a cloud robotics infrastructure under which robots can share their experience such as action recipes and manipulation strategies. An inference engine is provided for checking whether all requirements are fulfilled for performing a full manipulation strategy. It would be interesting to study how the uncertainty in perception and execution can be dealt with in conjunction with such a symbolic reasoning engine. When considering a complex action, grasp synthesis cannot be considered as an isolated problem. On the contrary, higherlevel tasks influence what the best grasp in a specific scenario might be, e.g. when grasping a specific tool. Task constraints have not yet been considered extensively in the community. Current approaches, e.g. BIB012 BIB017 , achieve impressive results. As an open question stands how to scale to life-long learning. 4) Robust Execution: It has been noticed by many researchers that inferring a grasp for a given object is necessary but not sufficient. Only if execution is robust to uncertainties in sensing and actuation, a grasp can succeed with high probability. There are a number of approaches that use constant tactile or visual feedback during grasp execution to adapt to unforeseen situations BIB007 BIB008 BIB013  BIB018 BIB019 BIB020 . Tactile feedback can be from haptic or force-torque sensors. Visual feedback can be the result from tracking the hand and object simultaneously. Also in this area, there are a number of open questions. How can tactile feedback be interpreted to choose an appropriate corrective action independent of the object, the task and environment? How can visual and tactile information be fused in the controller?