Data-Driven Grasp Synthesis - A Survey <s> B. Online Object Pose Estimation <s> The authors describe a general-purpose, representation-independent method for the accurate and computationally efficient registration of 3-D shapes including free-form curves and surfaces. The method handles the full six degrees of freedom and is based on the iterative closest point (ICP) algorithm, which requires only a procedure to find the closest point on a geometric entity to a given point. The ICP algorithm always converges monotonically to the nearest local minimum of a mean-square distance metric, and the rate of convergence is rapid during the first few iterations. Therefore, given an adequate set of initial rotations and translations for a particular class of objects with a certain level of 'shape complexity', one can globally minimize the mean-square distance metric over all six degrees of freedom by testing each initial registration. One important application of this method is to register sensed data from unfixtured rigid objects with an ideal geometric model, prior to shape inspection. Experimental results show the capabilities of the registration algorithm on point sets, curves, and surfaces. > <s> BIB001 </s> Data-Driven Grasp Synthesis - A Survey <s> B. Online Object Pose Estimation <s> In this paper we present a framework for grasp planning with a humanoid robot arm and a five-fingered hand. The aim is to provide the humanoid robot with the ability of grasping objects that appear in a kitchen environment. Our approach is based on the use of an object model database that contains the description of all the objects that can appear in the robot workspace. This database is completed with two modules that make use of this object representation: An exhaustive offline grasp analysis system and a real-time stereo vision system. The offline grasp analysis system determines the best grasp for the objects by employing a simulation system, together with CAD models of the objects and the five-fingered hand. The results of this analysis are added to the object database using a description suited to the requirements of the grasp execution modules. A stereo camera system is used for a real-time object localization using a combination of appearance-based and model-based methods. The different components are integrated in a controller architecture to achieve manipulation task goals for the humanoid robot. <s> BIB002 </s> Data-Driven Grasp Synthesis - A Survey <s> B. Online Object Pose Estimation <s> Robust vision-based grasping is still a hard problem for humanoid robot systems. When being restricted to using the camera system built-in into the robot's head for object localization, the scenarios get often very simplified in order to allow the robot to grasp autonomously. Within the computer vision community, many object recognition and localization systems exist, but in general, they are not tailored to the application on a humanoid robot. In particular, accurate 6D object localization in the camera coordinate system with respect to a 3D rigid model is crucial for a general framework for grasping. While many approaches try to avoid the use of stereo calibration, we will present a system that makes explicit use of the stereo camera system in order to achieve maximum depth accuracy. Our system can deal with textured objects as well as objects that can be segmented globally and are defined by their shape. Thus, it covers the cases of objects with complex texture and complex shape. Our work is directly linked to a grasping framework being implemented on the humanoid robot ARM AR and serves as its perception module for various grasping and manipulation experiments in a kitchen scenario. <s> BIB003 </s> Data-Driven Grasp Synthesis - A Survey <s> B. Online Object Pose Estimation <s> In this paper, we address the problem of automatic grasp generation for robotic hands where experience and shape primitives are used in synergy so to provide a basis not only for grasp generation but also for a grasp evaluation process when the exact pose of the object is not available. One of the main challenges in automatic grasping is the choice of the object approach vector, which is dependent both on the object shape and pose as well as the grasp type. Using the proposed method, the approach vector is chosen not only based on the sensory input but also on experience that some approach vectors will provide useful tactile information that finally results in stable grasps. A methodology for developing and evaluating grasp controllers is presented where the focus lies on obtaining stable grasps under imperfect vision. The method is used in a teleoperation or a programming by demonstration setting where a human demonstrates to a robot how to grasp an object. The system first recognizes the object and grasp type which can then be used by the robot to perform the same action using a mapped version of the human grasping posture. <s> BIB004 </s> Data-Driven Grasp Synthesis - A Survey <s> B. Online Object Pose Estimation <s> In order for humanoid robots to enter human-centred environments, it is indispensable to equip them with manipulative, perceptive and communicative skills necessary for real-time interaction with the environment and humans. The goal of our work is to provide reliable and highly integrated humanoid platforms which on the one hand allow the implementation and tests of various research activities and on the other hand the realization of service tasks in a household scenario. In this paper, we present a new humanoid robot currently being developed for applications in human-centred environments. In addition, we present an integrated grasping and manipulation system consisting of a motion planner for the generation of collision-free paths and a vision system for the recognition and localization of a subset of household objects as well as a grasp analysis component which provides the most feasible grasp configurations for each object. <s> BIB005 </s> Data-Driven Grasp Synthesis - A Survey <s> B. Online Object Pose Estimation <s> This paper addresses the issue of learning and representing object grasp affordances, i.e. object-gripper relative configurations that lead to successful grasps. The purpose of grasp affordances is to organize and store the whole knowledge that an agent has about the grasping of an object, in order to facilitate reasoning on grasping solutions and their achievability. The affordance representation consists in a continuous probability density function defined on the 6D gripper pose space - 3D position and orientation -, within an object-relative reference frame. Grasp affordances are initially learned from various sources, e.g. from imitation or from visual cues, leading to grasp hypothesis densities. Grasp densities are attached to a learned 3D visual object model, and pose estimation of the visual model allows a robotic agent to execute samples from a grasp hypothesis density under various object poses. Grasp outcomes are used to learn grasp empirical densities, i.e. grasps that have been confirmed through experience. We show the result of learning grasp hypothesis densities from both imitation and visual cues, and present grasp empirical densities learned from physical experience by a robot. <s> BIB006 </s> Data-Driven Grasp Synthesis - A Survey <s> B. Online Object Pose Estimation <s> We present an object representation framework that encodes probabilistic spatial relations between 3D features and organizes these features in a hierarchy. Features at the bottom of the hierarchy are bound to local 3D descriptors. Higher level features recursively encode probabilistic spatial configurations of more elementary features. The hierarchy is implemented in a Markov network. Detection is carried out by a belief propagation algorithm, which infers the pose of high-level features from local evidence and reinforces local evidence from globally consistent knowledge, effectively producing a likelihood for the pose of the object in the detection scene. We also present a simple learning algorithm that autonomously builds hierarchies from local object descriptors. We explain how to use our framework to estimate the pose of a known object in an unknown scene. Experiments demonstrate the robustness of hierarchies to input noise, viewpoint changes, and occlusions. <s> BIB007 </s> Data-Driven Grasp Synthesis - A Survey <s> B. Online Object Pose Estimation <s> Autonomous grasping of household objects is one of the major skills that an intelligent service robot necessarily has to provide in order to interact with the environment. In this paper, we propose a grasping strategy for known objects, comprising an off-line, box-based grasp generation technique on 3D shape representations. The complete system is able to robustly detect an object and estimate its pose, flexibly generate grasp hypotheses from the assigned model and perform such hypotheses using visual servoing. We will present experiments implemented on the humanoid platform ARMAR-III. <s> BIB008 </s> Data-Driven Grasp Synthesis - A Survey <s> B. Online Object Pose Estimation <s> We study the problem of human to robot grasp mapping as a basic building block of a learning by imitation system. The human hand posture, including both the grasp type and hand orientation, is first classified based on a single image and mapped to a specific robot hand. A metric for the evaluation based on the notion of virtual fingers is proposed. The first part of the experimental evaluation, performed in simulation, shows how the differences in the embodiment between human and robotic hand affect the grasp strategy. The second part, performed with a robotic system, demonstrates the feasibility of the proposed methodology in realistic applications. <s> BIB009 </s> Data-Driven Grasp Synthesis - A Survey <s> B. Online Object Pose Estimation <s> Robust perception is a vital capability for robotic manipulation in unstructured scenes. In this context, full pose estimation of relevant objects in a scene is a critical step towards the introduction of robots into household environments. In this paper, we present an approach for building metric 3D models of objects using local descriptors from several images. Each model is optimized to fit a set of calibrated training images, thus obtaining the best possible alignment between the 3D model and the real object. Given a new test image, we match the local descriptors to our stored models online, using a novel combination of the RANSAC and Mean Shift algorithms to register multiple instances of each object. A robust initialization step allows for arbitrary rotation, translation and scaling of objects in the test images. The resulting system provides markerless 6-DOF pose estimation for complex objects in cluttered scenes. We provide experimental results demonstrating orientation and translation accuracy, as well a physical implementation of the pose output being used by an autonomous robot to perform grasping in highly cluttered scenes. <s> BIB010 </s> Data-Driven Grasp Synthesis - A Survey <s> B. Online Object Pose Estimation <s> We present a method for learning object grasp affordance models in 3D from experience, and demonstrate its applicability through extensive testing and evaluation on a realistic and largely autonomous platform. Grasp affordance refers here to relative object-gripper configurations that yield stable grasps. These affordances are represented probabilistically with grasp densities, which correspond to continuous density functions defined on the space of 6D gripper poses. A grasp density characterizes an object's grasp affordance; densities are linked to visual stimuli through registration with a visual model of the object they characterize. We explore a batch-oriented, experience-based learning paradigm where grasps sampled randomly from a density are performed, and an importance-sampling algorithm learns a refined density from the outcomes of these experiences. The first such learning cycle is bootstrapped with a grasp density formed from visual cues. We show that the robot effectively applies its experience by downweighting poor grasp solutions, which results in increased success rates at subsequent learning cycles. We also present success rates in a practical scenario where a robot needs to repeatedly grasp an object lying in an arbitrary pose, where each pose imposes a specific reaching constraint, and thus forces the robot to make use of the entire grasp density to select the most promising achievable grasp. <s> BIB011 </s> Data-Driven Grasp Synthesis - A Survey <s> B. Online Object Pose Estimation <s> Grasping an object is a task that inherently needs to be treated in a hybrid fashion. The system must decide both where and how to grasp the object. While selecting where to grasp requires learning about the object as a whole, the execution only needs to reactively adapt to the context close to the grasp's location. We propose a hierarchical controller that reflects the structure of these two sub-problems, and attempts to learn solutions that work for both. A hybrid architecture is employed by the controller to make use of various machine learning methods that can cope with the large amount of uncertainty inherent to the task. The controller's upper level selects where to grasp the object using a reinforcement learner, while the lower level comprises an imitation learner and a vision-based reactive controller to determine appropriate grasping motions. The resulting system is able to quickly learn good grasps of a novel object in an unstructured environment, by executing smooth reaching motions and preshaping the hand depending on the object's geometry. The system was evaluated both in simulation and on a real robot. <s> BIB012 </s> Data-Driven Grasp Synthesis - A Survey <s> B. Online Object Pose Estimation <s> We present a complete software architecture for reliable grasping of household objects. Our work combines aspects such as scene interpretation from 3D range data, grasp planning, motion planning, and grasp failure identification and recovery using tactile sensors. We build upon, and add several new contributions to the significant prior work in these areas. A salient feature of our work is the tight coupling between perception (both visual and tactile) and manipulation, aiming to address the uncertainty due to sensor and execution errors. This integration effort has revealed new challenges, some of which can be addressed through system and software engineering, and some of which present opportunities for future research. Our approach is aimed at typical indoor environments, and is validated by long running experiments where the PR2 robotic platform was able to consistently grasp a large variety of known and unknown objects. The set of tools and algorithms for object grasping presented here have been integrated into the open-source Robot Operating System (ROS). <s> BIB013 </s> Data-Driven Grasp Synthesis - A Survey <s> B. Online Object Pose Estimation <s> In this paper, we present an efficient algorithm for 3D object recognition in presence of clutter and occlusions in noisy, sparse and unsegmented range data. The method uses a robust geometric descriptor, a hashing technique and an efficient RANSAC-like sampling strategy. We assume that each object is represented by a model consisting of a set of points with corresponding surface normals. Our method recognizes multiple model instances and estimates their position and orientation in the scene. The algorithm scales well with the number of models and its main procedure runs in linear time in the number of scene points. Moreover, the approach is conceptually simple and easy to implement. Tests on a variety of real data sets show that the proposed method performs well on noisy and cluttered scenes in which only small parts of the objects are visible. <s> BIB014 </s> Data-Driven Grasp Synthesis - A Survey <s> B. Online Object Pose Estimation <s> Applying model-free reinforcement learning to manipulation remains challenging for several reasons. First, manipulation involves physical contact, which causes discontinuous cost functions. Second, in manipulation, the end-point of the movement must be chosen carefully, as it represents a grasp which must be adapted to the pose and shape of the object. Finally, there is uncertainty in the object pose, and even the most carefully planned movement may fail if the object is not at the expected position. To address these challenges we 1) present a simplified, computationally more efficient version of our model-free reinforcement learning algorithm PI2; 2) extend PI2 so that it simultaneously learns shape parameters and goal parameters of motion primitives; 3) use shape and goal learning to acquire motion primitives that are robust to object pose uncertainty. We evaluate these contributions on a manipulation platform consisting of a 7-DOF arm with a 4-DOF hand. <s> BIB015 </s> Data-Driven Grasp Synthesis - A Survey <s> B. Online Object Pose Estimation <s> We present MOPED, a framework for Multiple Object Pose Estimation and Detection that seamlessly integrates single-image and multi-image object recognition and pose estimation in one optimized, robust, and scalable framework. We address two main challenges in computer vision for robotics: robust performance in complex scenes, and low latency for real-time operation. We achieve robust performance with Iterative Clustering Estimation (ICE), a novel algorithm that iteratively combines feature clustering with robust pose estimation. Feature clustering quickly partitions the scene and produces object hypotheses. The hypotheses are used to further refine the feature clusters, and the two steps iterate until convergence. ICE is easy to parallelize, and easily integrates single- and multi-camera object recognition and pose estimation. We also introduce a novel object hypothesis scoring function based on M-estimator theory, and a novel pose clustering algorithm that robustly handles recognition outliers. We achieve scalability and low latency with an improved feature matching algorithm for large databases, a GPU/CPU hybrid architecture that exploits parallelism at all levels, and an optimized resource scheduler. We provide extensive experimental results demonstrating state-of-the-art performance in terms of recognition, scalability, and latency in real-world robotic applications. <s> BIB016 </s> Data-Driven Grasp Synthesis - A Survey <s> B. Online Object Pose Estimation <s> In this paper, we present an efficient 3D object recognition and pose estimation approach for grasping procedures in cluttered and occluded environments. In contrast to common appearance-based approaches, we rely solely on 3D geometry information. Our method is based on a robust geometric descriptor, a hashing technique and an efficient, localized RANSAC-like sampling strategy. We assume that each object is represented by a model consisting of a set of points with corresponding surface normals. Our method simultaneously recognizes multiple model instances and estimates their pose in the scene. A variety of tests shows that the proposed method performs well on noisy, cluttered and unsegmented range scans in which only small parts of the objects are visible. The main procedure of the algorithm has a linear time complexity resulting in a high recognition speed which allows a direct integration of the method into a continuous manipulation task. The experimental validation with a seven-degree-of-freedom Cartesian impedance controlled robot shows how the method can be used for grasping objects from a complex random stack. This application demonstrates how the integration of computer vision and soft-robotics leads to a robotic system capable of acting in unstructured and occluded environments. <s> BIB017 </s> Data-Driven Grasp Synthesis - A Survey <s> B. Online Object Pose Estimation <s> The ability to grasp unknown objects is an important skill for personal robots, which has been addressed by many present and past research projects, but still remains an open problem. A crucial aspect of grasping is choosing an appropriate grasp configuration, i.e. the 6d pose of the hand relative to the object and its finger configuration. Finding feasible grasp configurations for novel objects, however, is challenging because of the huge variety in shape and size of these objects. Moreover, possible configurations also depend on the specific kinematics of the robotic arm and hand in use. In this paper, we introduce a new grasp selection algorithm able to find object grasp poses based on previously demonstrated grasps. Assuming that objects with similar shapes can be grasped in a similar way, we associate to each demonstrated grasp a grasp template. The template is a local shape descriptor for a possible grasp pose and is constructed using 3d information from depth sensors. For each new object to grasp, the algorithm then finds the best grasp candidate in the library of templates. The grasp selection is also able to improve over time using the information of previous grasp attempts to adapt the ranking of the templates. We tested the algorithm on two different platforms, the Willow Garage PR2 and the Barrett WAM arm which have very different hands. Our results show that the algorithm is able to find good grasp configurations for a large set of objects from a relatively small set of demonstrations, and does indeed improve its performance over time. <s> BIB018
In the previous section, we reviewed different approaches towards grasping known objects regarding their way to gen- First, a known object in the scene is segmented and recognized. Past experiences with that object are retrieved and a new grasp hypothesis is generated or selected among the already tested ones. After execution of the selected grasp, the performance is evaluated and the memory of past experiences with the object is updated.The following approaches use trial-and-error learning: BIB006 BIB011 BIB012 BIB015 . erate and rank candidate grasps. During online execution, an object has to be recognized and its pose estimated before the offline trained grasps can be executed. Furthermore, from the set of hypotheses not all grasps might be feasible in the current scene. They have to be filtered by reachability. The lower part of Fig. 3 visualizes the data flow during grasp execution and how the offline generated data is employed. Several of the aforementioned grasp generation methods BIB012 BIB006 BIB011 use the probabilistic approach towards object representation and pose estimation proposed by Detry et al. BIB007 . It is visualized in Fig. 8e . Grasps are either selected by sampling from densities BIB006 BIB011 or a grasp policy refined from a human demonstration is applied BIB012 . Morales et al. BIB002 use the method proposed by Azad et al. BIB003 to recognize an object and estimate its pose from a monocular image as shown in Fig. 8a . Given this information, an appropriate grasp configuration can be selected from a grasp experience database that has been acquired offline. The whole system is demonstrated on the robotic platform described in Asfour et al. BIB005 . Huebner et al. BIB008 demonstrate grasping of known objects on the same humanoid platform and using the same method for object recognition and pose estimation. The offline selection of grasp hypotheses is based on a decomposition into boxes. Ciocarlie et al. BIB013 propose a robust grasping pipeline in which known object models are fitted to point cloud clusters using standard ICP BIB001 . The search space of potential object poses is reduced by assuming a dominant plane and rotationally-symmetric objects that are always standing upright. An example scene is shown in Fig. 8b . Papazov et al. BIB017 demonstrate their previous approach on 3D object recognition and pose estimation BIB014 in a grasping scenario. Multiple objects in cluttered scenes can be robustly recognized and their pose estimated. No assumption is made about the geometry of the scene, shape of the objects or their pose. The aforementioned methods assume a-priori known rigid 3D object model. Glover et al.  consider known de- Figure 6 : Robot grasp learning from human demonstration. 6a) Kinesthetic Teaching BIB018 . 6b) Human-to-robot mapping of grasps using a data glove BIB004 . 6c) Human-to-robot mapping of grasps using visual grasp recognition BIB009 Figure 8 : Object representations for grasping and corresponding methods for pose estimation. 8a) Object pose estimation of textured and untextured objects in monocular images BIB003 . 8b) ICP-based object pose estimation from segmented point clouds BIB013 . 8c) Deformable object detection and pose estimation in monocular images  . 8d) Multi-view object representation composed of 2D and 3D features BIB010 . 8e) Probabilistic and hierarchical approach towards object pose estimation BIB007 . 8f) Grasp Candidates linked to groups of local contour descriptors. BIB006 . 8g) Empirical grasp density built by trial and error. BIB006 formable objects. Probabilistic models of their 2D shape are learned offline. The objects can then be detected in monocular images of cluttered scenes even when they are partially occluded. The visible object part serve as a basis for planning a grasp under consideration of the global object shape. An example for a successful detection is shown in Fig. 8c . Collet Romea et al. BIB016 use a combination of 2D and 3D features as an object model. Examples for objects from an earlier version of the system BIB010 are shown in Fig. 8d . The authors estimate the object's pose in a scene from a single image. The accuracy of their approach is demonstrated through a number of successful grasps.