Data-Driven Grasp Synthesis - A Survey <s> IV. GRASPING UNKNOWN OBJECTS <s> Abstract This paper addresses the problem of designing a practical system able to grasp real objects with a three-fingered robot hand. A general approach for synthesizing two- and three-finger grasps on planar unknown objects is presented. Visual perception is used to reduce the uncertainty and to obtain relevant information about the objects. We focus on non-modeled planar extruded objects, which can represent many real-world objects. In addition, particular mechanical constraints of the robot hand are considered. First, a vision processing module that extracts from object images the relevant information for the grasp synthesis is developed. This is completed with a set of algorithms for synthesizing two- and three-finger grasps taking into account force-closure and contact stability conditions, with a low computational effort. Finally, a procedure for constraining these results to the kinematics of the particular hand, is also developed. In addition, a set of heuristic metrics for assessing the quality of the computed grasps is described. All these components are integrated in a complete system. Experimental results using the Barrett hand are shown and discussed. <s> BIB001 </s> Data-Driven Grasp Synthesis - A Survey <s> IV. GRASPING UNKNOWN OBJECTS <s> We describe a process in which the segmentation of objects as well as the extraction of the object shape becomes realized through active exploration of a robot vision system. In the exploration process, two behavioral modules that link robot actions to the visual and haptic perception of objects interact. First, by making use of an object independent grasping mechanism, physical control over potential objects can be gained. Having evaluated the initial grasping mechanism as being successful, a second behavior extracts the object shape by making use of prediction based on the motion induced by the robot. This also leads to the concept of an "object" as a set of features that change predictably over different frames. The system is equipped with a certain degree of generic prior knowledge about the world in terms of a sophisticated visual feature extraction process in an early cognitive vision system, knowledge about its own embodiment as well as knowledge about geometric relationships such as rigid body motion. This prior knowledge allows the extraction of representations that are semantically richer compared to many other approaches. <s> BIB002 </s> Data-Driven Grasp Synthesis - A Survey <s> IV. GRASPING UNKNOWN OBJECTS <s> This paper describes the development of a novel vision-based grasping system for unknown objects based on range images. We realize a synthesis of the calculated grasp points with a 3D model of a hand prosthesis, which we are using as gripper. We locally find grasp point candidates based on the shape of the object and validate the globally by checking collisions between the gripper and surrounding objects and the table top. Our approach integrates a robust object segmentation and grasp point detection for every object on a table in front of a 7-DOF robot arm. The algorithm analyzes the top surface of every object and outputs the generated grasp points and the required gripper pose to grasp the desired object. Additionally we can calculate the optimal opening angle of the gripper. The first experimental results show that the presented automated grasping system is able to generate successful grasp points for a wide range of different objects. <s> BIB003 </s> Data-Driven Grasp Synthesis - A Survey <s> IV. GRASPING UNKNOWN OBJECTS <s> This paper describes the development of a novel vision-based modeling and grasping system for three-dimensional (3D) objects whose shape and location are unknown a priori. Our approach integrates online computer vision-based 3D object modeling with online 3D grasp planning and execution. A single wrist-mounted video camera is moved around the stationary object to obtain images from multiple viewpoints. Object silhouettes are extracted from these images and used to form a 3D solid model of the object. To refine the model, the object's top surface is modeled by scanning with a wrist-mounted line laser while recording images. The laser line in each image is used to form a 3D surface model that is combined with the silhouette result. The grasp planning algorithm is designed for the parallel-jaw grippers that are commonly used in industry. The algorithm analyses the solid model, generates a robust force closure grasp, and outputs the required gripper position and orientation for grasping the object. The robot then automatically picks up the object. Experiments are performed with two real-world 3D objects, a metal bracket and a hex nut. The shape, position and orientation of the objects are not known by the system a priori. The time required to compute an object model and plan a grasp was less than 4 s for each object. The experimental results demonstrate that the automated grasping system can obtain suitable models and generate successful grasps, even when the objects are not lying parallel to the supporting table. <s> BIB004 </s> Data-Driven Grasp Synthesis - A Survey <s> IV. GRASPING UNKNOWN OBJECTS <s> This paper presents a method to determine the rough shape of an object. This is a step in the development of a ldquoone click grasping toolrdquo, a grasping tool of everyday-life objects for an assistant robot dedicated to elderly or disabled. The goal is to determine the quadric that approximates at best the shape of an unknown object using multi-view measurements. Non-linear optimization techniques are considered to achieve this goal. Since multiple views are necessary, an active vision process is considered in order to minimize the uncertainty on the estimated parameters and determine the next best view. Finally, results that show the validity of the approach are presented. <s> BIB005 </s> Data-Driven Grasp Synthesis - A Survey <s> IV. GRASPING UNKNOWN OBJECTS <s> We present a novel vision-based grasp point detection algorithm that can reliably detect the corners of a piece of cloth, using only geometric cues that are robust to variation in texture. Furthermore, we demonstrate the effectiveness of our algorithm in the context of folding a towel using a general-purpose two-armed mobile robotic platform without the use of specialized end-effectors or tools. The robot begins by picking up a randomly dropped towel from a table, goes through a sequence of vision-based re-grasps and manipulations—partially in the air, partially on the table—and finally stacks the folded towel in a target location. The reliability and robustness of our algorithm enables for the first time a robot with general purpose manipulators to reliably and fully-autonomously fold previously unseen towels, demonstrating success on all 50 out of 50 single-towel trials as well as on a pile of 5 towels. <s> BIB006 </s> Data-Driven Grasp Synthesis - A Survey <s> IV. GRASPING UNKNOWN OBJECTS <s> Robotic grasping in unstructured environments requires the ability to select grasps for unknown objects and execute them while dealing with uncertainty due to sensor noise or calibration errors. In this work, we propose a simple but robust approach to grasp selection for unknown objects, and a reactive adjustment approach to deal with uncertainty in object location and shape. The grasp selection method uses 3D sensor data directly to determine a ranked set of grasps for objects in a scene, using heuristics based on both the overall shape of the object and its local features. The reactive grasping approach uses tactile feedback from fingertip sensors to execute a compliant robust grasp. We present experimental results to validate our approach by grasping a wide range of unknown objects. Our results show that reactive grasping can correct for a fair amount of uncertainty in the measured position or shape of the objects, and that our grasp selection approach is successful in grasping objects with a variety of shapes. <s> BIB007 </s> Data-Driven Grasp Synthesis - A Survey <s> IV. GRASPING UNKNOWN OBJECTS <s> Robotic grasping in an open environment requires both object-specific as well as general grasping skills. When the objects are previously known it is possible to employ techniques that exploit object models, like geometrical grasping simulators. On the other hand, a competent system will also be able to deal with unmodeled objects using general solutions. In this paper we present an integrated system for autonomous rigid-object pick-up tasks in domestic environments, focusing on the gripping of unmodeled objects and exploiting sensor feedback from the robot hand to monitor the grasp. We describe the perception system based on time-of-flight range data, the grasp pose optimization algorithm and the grasp execution. The performance and robustness of the system is validated by experiments including pick-up tasks on many different common kitchen items. <s> BIB008 </s> Data-Driven Grasp Synthesis - A Survey <s> IV. GRASPING UNKNOWN OBJECTS <s> In this paper we present a method for building models for grasping from a single 3D snapshot of a scene composed of objects of daily use in human living environments. We employ fast shape estimation, probabilistic model fitting and verification methods capable of dealing with different kinds of symmetries, and combine these with a triangular mesh of the parts that have no other representation to model previously unseen objects of arbitrary shape. Our approach is enhanced by the information given by the geometric clues about different parts of objects which serve as prior information for the selection of the appropriate reconstruction method. While we designed our system for grasping based on single view 3D data, its generality allows us to also use the combination of multiple views. We present two application scenarios that require complete geometric models: grasp planning and locating objects in camera images. <s> BIB009 </s> Data-Driven Grasp Synthesis - A Survey <s> IV. GRASPING UNKNOWN OBJECTS <s> We consider the task of grasping novel objects and cleaning fairly cluttered tables with many novel objects. Recent successful approaches employ machine learning algorithms to identify points on the scene that the robot should grasp. In this paper, we show that the task can be significantly simplified by using segmentation, especially with depth information. A supervised localization method is employed to select graspable segments. We also propose a shape completion and grasp planner method which takes partial 3D information and plans the most stable grasping strategy. Extensive experiments on our robot demonstrate the effectiveness of our approach. <s> BIB010 </s> Data-Driven Grasp Synthesis - A Survey <s> IV. GRASPING UNKNOWN OBJECTS <s> Grasping unknown objects based on real-world visual input is a challenging problem. In this paper, we present an Early Cognitive Vision system that builds a hierarchical representation based on edge and texture information, which is a sparse but powerful description of the scene. Based on this representation we generate edge-based and surface-based grasps. The results show that the method generates successful grasps, that the edge and surface information are complementary, and that the method can deal with more complex scenes. We furthermore present a benchmark for visual-based grasping. <s> BIB011 </s> Data-Driven Grasp Synthesis - A Survey <s> IV. GRASPING UNKNOWN OBJECTS <s> Grasp planning based on perceived sensor data of an object can be performed in different ways, depending on the chosen semantic interpretation of the sensed data. For example, if the object can be recognized and a complete 3D model is available, a different planning tool can be selected compared to the situation in which only the raw sensed data, such as a single point cloud, is available. Instead of choosing between these options, we present a framework that combines them, aiming to find consensus on how the object should be grasped by using the information from each object representation according to their confidence levels. We show that this method is robust to common errors in perception, such as incorrect object recognition, while also taking into account potential grasp execution errors due to imperfect robot calibration. We illustrate this method on the PR2 robot by grasping objects common in human environments. <s> BIB012 </s> Data-Driven Grasp Synthesis - A Survey <s> IV. GRASPING UNKNOWN OBJECTS <s> In this paper, we describe efficient methods for solving everyday mobile manipulation tasks that require object pick-up. In order to achieve real-time performance in complex environments, we focus our approach on fast yet robust solutions. For 3D perception of objects on planar surfaces, we develop scene segmentation methods that process Microsoft Kinect depth images in real-time at high frame rates. We efficiently plan feasible, collision-free grasps on the segmented objects directly from the perceived point clouds to achieve fast execution times. We evaluate our approaches quantitatively in lab experiments and also report on the successful integration of our methods in public demonstrations at RoboCup German Open 2011 and RoboCup 2011 in Istanbul, Turkey. Mobile manipulation tasks in domestic environments require a vast set of perception and action capabilities. The robot not only requires localization, mapping, path planning, and obsta- cle avoidance abilities to safely navigate through the environ- ment. It also needs to integrate object detection, recognition, and manipulation. A typical requirement for a service robot is not just to achieve the task, but to perform it in reasonable time. While much research has been invested into the general solution of complex perception and motion planning problems, only few work has been focused on methods that solve the tasks efficiently in order to allow for continuous task execution without interruptions. In this paper, we present fast methods to flexibly grasp objects from planar surfaces. To achieve fast performance, we combine real-time object perception with efficient grasp planning and motion control. For real-time perception, we combine rapid normal estimation using integral images with efficient segmentation techniques. We segment the scene into the support plane of interest and the objects thereon. Our perception algorithm processes depth images of a Microsoft Kinect in real-time at a frame rate of approx. 16 Hz. From the raw object point clouds our grasp planning method derives fea- sible, collision-free grasps within about 100 milliseconds. We consider grasps on objects from either the side or from above. The planned grasps are then executed using parametrized motion primitives. We integrate our approaches into a system that we publicly evaluate at RoboCup competitions. We further conduct experiments in our lab to demonstrate the robustness and efficiency of our approaches. This paper is organized as follows: after a brief system overview in Sec. III, we detail our approaches to real-time <s> BIB013 </s> Data-Driven Grasp Synthesis - A Survey <s> IV. GRASPING UNKNOWN OBJECTS <s> In this paper, we present a novel grasp selection algorithm to enable a robot with a two-fingered end-effector to autonomously grasp unknown objects. Our approach requires as input only the raw depth data obtained from a single frame of a 3D sensor. Additionally, our approach uses no explicit models of the objects and does not require a training phase. We use the grasping capability to demonstrate the application of a robot as an autonomous checkout clerk. To perform this task, the robot must identify how to grasp an object, locate the barcode on the object and read the numeric code. We evaluate our grasping algorithm in experiments where the robot was required to autonomously grasp unknown objects. The robot achieved a success of 91.6%in grasping novel objects. We performed two sets of experiments to evaluate the checkout robot application. In the first set, the objects were placed in many orientations in front of the robot one at a time. In the second set, the objects were placed several at a time with varying amounts of clutter. The robot was able to autonomously grasp and scan the objects in 49/50 of the single-object trials and 46/50 of the cluttered trials. <s> BIB014 </s> Data-Driven Grasp Synthesis - A Survey <s> IV. GRASPING UNKNOWN OBJECTS <s> We consider the problem of grasp and manipulation planning when the state of the world is only partially observable. Specifically, we address the task of picking up unknown objects from a table top. The proposed approach to object shape prediction aims at closing the knowledge gaps in the robot's understanding of the world. A completed state estimate of the environment can then be provided to a simulator in which stable grasps and collision-free movements are planned. <s> BIB015 </s> Data-Driven Grasp Synthesis - A Survey <s> IV. GRASPING UNKNOWN OBJECTS <s> This paper explores how Cloud Computing can facilitate grasping with shape uncertainty. We consider the most common robot gripper: a pair of thin parallel jaws, and a class of objects that can be modeled as extruded polygons. We model a conservative class of push-grasps that can enhance object alignment. The grasp planning algorithm takes as input an approximate object outline and Gaussian uncertainty around each vertex and center of mass. We define a grasp quality metric based on a lower bound on the probability of achieving force closure. We present a highly-parallelizable algorithm to compute this metric using Monte Carlo sampling. The algorithm uses Coulomb frictional grasp mechanics and a fast geometric test for conservative conditions for force closure. We run the algorithm on a set of sample shapes and compare the grasps with those from a planner that does not model shape uncertainty. We report computation times with single and multi-core computers and sensitivity analysis on algorithm parameters. We also describe physical grasp experiments using the Willow Garage PR2 robot. <s> BIB016 </s> Data-Driven Grasp Synthesis - A Survey <s> IV. GRASPING UNKNOWN OBJECTS <s> A method for fast visual grasping of unknown objects with a multifingered robotic hand is presented in this paper. The algorithm is composed of an object surface reconstruction algorithm and a local grasp planner, evolving in parallel. The reconstruction algorithm makes use of images taken by a camera carried by the robot arm. A virtual elastic reconstruction surface is placed around the object. The surface shrinks toward the object until some points intercept the object visual hull. Then, attractive forces with respect to the border of the visual hull are generated so as to compensate for the elastic forces: when an equilibrium between those forces is reached, the surface takes the form of the object shape. Running in parallel to the reconstruction algorithm, the grasp planner moves the fingertips on the current available reconstruction surface, toward points which are optimal (in a local sense) with respect to a number of indices weighting both the grasp quality and the kinematics configuration of the hand. This method, referred to as parallel visual grasp, may represent a valid candidate for applications where online grasp planning is required. A number of experiments are presented, showing the effectiveness of the proposed approach. <s> BIB017
If a robot has to grasp a previously unseen object, we refer to it as unknown. Approaches towards grasping known objects are obviously not applicable since they rely on the assumption that an object model is available. The approaches in this group also do not assume to have access to other kinds of grasp experiences. Instead, they propose and analyze heuristics that directly link structure in the sensory data to candidate grasps. There are various ways to deal with sparse, incomplete and noisy data from real sensors such as stereo cameras: we divided the approaches into methods that i) approximate the full shape of an object, ii) methods that generate grasps based on low-level features and a set of heuristics, and iii) methods that rely mostly on the global shape of the partially observed object hypothesis. The reviewed approaches are summarized in Table III . A flow chart that visualizes the data flow in the following approaches is shown in Fig. 24 . Figure 24 : Typical functional flow-chart of a grasping system for unknown objects. The scene is perceived and segmented to obtain object hypotheses and relevant perceptual features. Then the system follows either the right or left pathway. On the left, low level features are used to generate heuristically a set of grasp hypotheses. On the right, a mesh model approximating the global object shape is generated from the perceived features. Grasp candidates are then sampled and executed in a simulator. Classical analytic grasp metric are used to rank the grasp candidates. Finally non reachable grasp hypotheses are filtered out, and the best ranked grasp hypothesis is executed. The following approaches use the left pathway: BIB002 BIB011 BIB003 BIB006 BIB007 BIB012 BIB013 BIB014 BIB008 BIB001 . The following approaches estimate a full object model: BIB004 BIB015 BIB009 BIB017 BIB005 BIB016 A. Approximating Unknown Object Shape One approach towards generating grasp hypotheses for unknown objects is to approximate objects with shape primitives. Dunes et al. BIB005 approximate an object with a quadric whose minor axis is used to infer the wrist orientation. The object centroid serves as the approach target and the rough object size helps to determine the hand pre-shape. The quadric is estimated from multi-view measurements of the global object shape in monocular images. Marton et al. BIB009 show how grasp selection can be performed exploiting symmetry by fitting a curve to a cross section of the point cloud of an object. For grasp planning, the reconstructed object is imported to a simulator. Grasp candidates are generated through randomization of grasp parameters on which then the force-closure criteria is evaluated. Rao et al. BIB010 sample grasp points from the surface of a segmented object. The normal of the local surface at this point serves as a search direction for a second contact point. This is chosen to be at the intersection between the extended normal and the opposite side of the object. By assuming symmetry, this second contact point is assumed to have a contact normal in the direction opposite to the normal of the first contact point. Bohg et al. [115] propose a related approach that reconstructs full object shape assuming planar symmetry which subsumes all other kinds of symmetries. It takes the complete point cloud into account and not only a local patch. Two simple methods to generate grasp candidate on the resulting completed object models are proposed and evaluated. An example for an object whose full object shape is approximated with this approach is shown in Fig. 25 . Opposed to the above mentioned techniques, Bone et al. [112] make no prior assumption about the shape of the object. They apply shape carving for the purpose of grasping with a parallel-jaw gripper. After obtaining a model of the object, they search for a pair of reasonably flat and parallel surfaces that are best suited for this kind of manipulator. An object reconstructed with this method is shown in Fig. 26 . Lippiello et al. BIB017 present a related approach for grasping an unknown object with a multi-fingered hand. The authors first record a number of views from around the object. Based on the object bounding box in each view, a polyhedron is defined that overestimates the visual object hull and is then approximated by a quadric. A pre-grasp shape is defined in which the fingertip contacts on the quadric are aligned with its two minor axes. This grasp is then refined given the local surface shape close to the contact point. This process is alternating with the refinement of the object shape through an elastic surface model. The quality of the grasps is evaluated by classic metrics. As previously discussed, it is not clear how well these metrics predict the outcome of a grasp. It remains to be shown whether grasps synthesized in this way are more successful than e.g. the just closing the fingers in the pre-grasp configuration.