We present and evaluate an approach for human-in-the-loop specification of shape reconstruction with annotations for basic robot-object interactions. Our method is based on the idea of model annotation: the addition of simple cues to an underlying object model to specify shape and delineate a simple task. The goal is to explore reducing the complexity of CAD-like interfaces so that novice users can quickly recover an object's shape and describe a manipulation task that is then carried out by a robot. The object modeling and interaction annotation capabilities are tested with a user study and compared against results obtained using existing approaches. The approach has been analyzed using a variety of shape comparison, grasping, and manipulation metrics, and tested with the PR2 robot platform, where it was shown to be successful.
Grasping and manipulation are closely related. For some grasps finding the final contacts of the gripper is insufficient. In several tasks, the approach to accomplish the grasp, as well as the task-dependent manipulation that will be performed affect the generated grasp hypotheses. Some studies have looked at shape category and the physical context to determine manipulation constraints @cite_69 , while others have used human experience to seed approach vectors @cite_33 . In @cite_23 , a grasp planning algorithm is used in conjunction to a set of task-dependent semantic constraints to choose grasps, which in turn may inform the approach vector. These semantic constraints are extracted from example grasps and then used to construct a semantic affordance map which directly relates the object class (obtained from object depth information) to the approach vectors and different task-appropriate grasps. This provides evidence that relating object instance or category models to manipulation annotations is of great utility.
@cite_69 @cite_33 @cite_23
2003511559 2110208234 2010274050
We design an example based planning framework to generate semantic grasps, stable grasps that are functionally suitable for specific object manipulation tasks. We propose to use partial object geometry, tactile contacts, and hand kinematic data as proxies to encode semantic constraints, which are task-related constraints. We introduce a semantic affordance map, which relates local geometry to a set of predefined semantic grasps that are appropriate to different tasks. Using this map, the pose of a robotic hand can be estimated so that the hand is adjusted to achieve the ideal approach direction required by a particular task. A grasp planner is then used to generate a set of final grasps which have appropriate stability, tactile contacts, and hand kinematics along this approach direction. We show experiments planning semantic grasps on everyday objects and executing these grasps with a physical robot. We present an example-based planning framework to generate semantic grasps, stable grasps that are functionally suitable for specific object manipulation tasks. We propose to use partial object geometry, tactile contacts, and hand kinematic data as proxies to encode task-related constraints, which we call semantic constraints. We introduce a semantic affordance map, which relates local geometry to a set of predefined semantic grasps that are appropriate to different tasks. Using this map, the pose of a robot hand with respect to the object can be estimated so that the hand is adjusted to achieve the ideal approach direction required by a particular task. A grasp planner is then used to search along this approach direction and generate a set of final grasps which have appropriate stability, tactile contacts, and hand kinematics. We show experiments planning semantic grasps on everyday objects and applying these grasps with a physical robot. Abstract The problem of finding stable grasps has been widely studied in robotics. However, in many applications the resulting grasps should not only be stable but also applicable for a particular task. Task-specific grasps are closely linked to object categories so that objects in a same category can be often used to perform the same task. This paper presents a probabilistic approach for task-specific stable grasping of objects with shape variations inside the category. An optimal grasp is found as a grasp that is maximally likely to be task compatible and stable taking into account shape uncertainty in a probabilistic context. The method requires only partial models of new objects for grasp generation and only few models and example grasps are used during the training stage. The experiments show that the approach can use multiple models to generalize to new objects in that it outperforms grasping based on the closest model. The method is shown to generate stable grasps for new objects belonging to the same class as well as for similar in shape objects of different categories. This paper presents a novel robot grasping planning approach that extracts grasp strategies (grasp type, and thumb placement and direction) from human demonstration and integrates them into the grasp planning procedure to generate a feasible grasp concerning the target object geometry and manipulation task. Our study results show that the grasp strategies of grasp type and thumb placement not only represent important human grasp intentions, but also provide meaningful constraints on hand posture and wrist position, which highly reduce both the feasible workspace of a robotic hand and the search space of the grasp planning. This approach has been thoroughly evaluated both in simulation and with a real robotic system for multiple daily living representative objects. We have also demonstrated the robustness of our approach in the experiment with different levels of perception uncertainties. In this paper, we address the problem of automatic grasp generation for robotic hands where experience and shape primitives are used in synergy so to provide a basis not only for grasp generation but also for a grasp evaluation process when the exact pose of the object is not available. One of the main challenges in automatic grasping is the choice of the object approach vector, which is dependent both on the object shape and pose as well as the grasp type. Using the proposed method, the approach vector is chosen not only based on the sensory input but also on experience that some approach vectors will provide useful tactile information that finally results in stable grasps. A methodology for developing and evaluating grasp controllers is presented where the focus lies on obtaining stable grasps under imperfect vision. The method is used in a teleoperation or a programming by demonstration setting where a human demonstrates to a robot how to grasp an object. The system first recognizes the object and grasp type which can then be used by the robot to perform the same action using a mapped version of the human grasping posture. We present a method for transferring grasps between objects of the same functional category. This transfer is intended to preserve the functionality of a grasp constructed for one of the objects, thus enabling the analogous action to be performed on a novel object for which no grasp has been specified. Manipulation knowledge is hence generalized from a single example to a class of objects with a significant amount of shape variability. The transfer is achieved through warping the surface geometry of the source object onto the target object, and along with it the contact points of a grasp. The warped contacts are locally replanned, if necessary, to ensure grasp stability, and a suitable grasp pose is computed. We present extensive results of experiments with a database of four-finger grasps, designed to systematically cover variations on grasping the mugs of the Princeton Shape Benchmark. Many manipulation planning problems involve several related sub-problems, such as the selection of grasping points on an object, choice of hand posture, and determination of the arm's configuration and evolving trajectory. Traditionally, these planning sub-problems have been handled separately, potentially leading to sub-optimal, or even infeasible, combinations of the individually determined solutions. This paper formulates the combined problem of grasp contact selection, grasp force optimization, and manipulator arm hand trajectory planning as a problem in optimal control. That is, the locally optimal trajectory for the manipulator, hand mechanism, and contact locations are determined during the pre-grasping, grasping, and subsequent object transport phase. Additionally, a barrier function approach allows for non-feasible grasps to be optimized, enlarging the region of convergence for the algorithm. A simulation of a simple planar object manipulation task is used to illustrate and validate the approach. An important challenge in robotics is to achieve robust performance in object grasping and manipulation, dealing with noise and uncertainty. This paper presents an approach for addressing the performance of dexterous grasping under shape uncertainty. In our approach, the uncertainty in object shape is parametrized and incorporated as a constraint into grasp planning. The proposed approach is used to plan feasible hand configurations for realizing planned contacts using different robotic hands. A compliant finger closing scheme is devised by exploiting both the object shape uncertainty and tactile sensing at fingertips. Experimental evaluation demonstrates that our method improves the performance of dexterous grasping under shape uncertainty. We considered object shape uncertainty in grasp planning and control.We proposed a probabilistic model to solve hand inverse kinematics.Our grasp planning approach is hand interchangeable.We presented a compliant uncertainty-aware controller for finger closing during grasp execution. This paper is the second in a two-part series analyzing human grasping behavior during a wide range of unstructured tasks. It investigates the tasks performed during the daily work of two housekeepers and two machinists and correlates grasp type and object properties with the attributes of the tasks being performed. The task or activity is classified according to the force required, the degrees of freedom, and the functional task type. We found that 46 percent of tasks are constrained, where the manipulated object is not allowed to move in a full six degrees of freedom. Analyzing the interrelationships between the grasp, object, and task data show that the best predictors of the grasp type are object size, task constraints, and object mass. Using these attributes, the grasp type can be predicted with 47 percent accuracy. Those parameters likely make useful heuristics for grasp planning systems. The results further suggest the common sub-categorization of grasps into power, intermediate, and precision categories may not be appropriate, indicating that grasps are generally more multi-functional than previously thought. We find large and heavy objects are grasped with a power grasp, but small and lightweight objects are not necessarily grasped with precision grasps—even with grasped object size less than 2 cm and mass less than 20 g, precision grasps are only used 61 percent of the time. These results have important implications for robotic hand design and grasp planners, since it appears while power grasps are frequently used for heavy objects, they can still be quite practical for small, lightweight objects. In a typical Human-Robot Interaction (HRI) scenario, the robot needs to perform various tasks for the human, hence should take into account human oriented constraints. In this context it is not sufficient that the robot selects grasp and placement of the object from the stability point of view only. Motivated from human behavioral psychology, in this paper we emphasize on the mutually depended nature of grasp and placement selections, which is further constrained by the task, the environment and the human's perspective. We will explore essential human oriented constraints on grasp and placement selections and present a framework to incorporate them in synthesizing key configurations of planning basic interactive manipulation tasks. Studies of human manipulation strategies suggest that pre-grasp object manipulation, such as rotation or sliding of the object to be grasped, can improve task performance by increasing both the task success rate and the quality of load-supporting postures. In previous demonstrations, pre-grasp object rotation by a robot manipulator was limited to manually-programmed actions. We present a method for automating the planning of pre-grasp rotation for object transport tasks. Our technique optimizes the grasp acquisition point by selecting a target object pose that can be grasped by high-payload manipulator configurations. Careful selection of the transition states leads to successful transport plans for tasks that are otherwise infeasible. In addition, optimization of the grasp acquisition posture also indirectly improves the transport plan quality, as measured by the safety margin of the manipulator payload limits. An important ability of a robot that interacts with the environment and manipulates objects is to deal with the uncertainty in sensory data. Sensory information is necessary to, for example, perform online assessment of grasp stability. We present methods to assess grasp stability based on haptic data and machine-learning methods, including AdaBoost, support vector machines (SVMs), and hidden Markov models (HMMs). In particular, we study the effect of different sensory streams to grasp stability. This includes object information such as shape; grasp information such as approach vector; tactile measurements from fingertips; and joint configuration of the hand. Sensory knowledge affects the success of the grasping process both in the planning stage (before a grasp is executed) and during the execution of the grasp (closed-loop online control). In this paper, we study both of these aspects. We propose a probabilistic learning framework to assess grasp stability and demonstrate that knowledge about grasp stability can be inferred using information from tactile sensors. Experiments on both simulated and real data are shown. The results indicate that the idea to exploit the learning approach is applicable in realistic scenarios, which opens a number of interesting venues for the future research. This paper is the first of a two-part series analyzing human grasping behavior during a wide range of unstructured tasks. The results help clarify overall characteristics of human hand to inform many domains, such as the design of robotic manipulators, targeting rehabilitation toward important hand functionality, and designing haptic devices for use by the hand. It investigates the properties of objects grasped by two housekeepers and two machinists during the course of almost 10,000 grasp instances and correlates the grasp types used to the properties of the object. We establish an object classification that assigns each object properties from a set of seven classes, including mass, shape and size of the grasp location, grasped dimension, rigidity, and roundness. The results showed that 55 percent of grasped objects had at least one dimension larger than 15 cm, suggesting that more than half of objects cannot physically be grasped using their largest axis. Ninety-two percent of objects had a mass of 500 g or less, implying that a high payload capacity may be unnecessary to accomplish a large subset of human grasping behavior. In terms of grasps, 96 percent of grasp locations were 7 cm or less in width, which can help to define requirements for hand rehabilitation and defines a reasonable grasp aperture size for a robotic hand. Subjects grasped the smallest overall major dimension of the object in 94 percent of the instances. This suggests that grasping the smallest axis of an object could be a reliable default behavior to implement in grasp planners. In this work, we present a part-based grasp planning approach that is capable of generating grasps that are applicable to multiple familiar objects. We show how object models can be decomposed according to their shape and local volumetric information. The resulting object parts are labeled with semantic information and used for generating robotic grasping information. We investigate how the transfer of such grasping information to familiar objects can be achieved and how the transferability of grasps can be measured. We show that the grasp transferability measure provides valuable information about how successful planned grasps can be applied to novel object instances of the same object category. We evaluate the approach in simulation, by applying it to multiple object categories and determine how successful the planned grasps can be transferred to novel, but familiar objects. In addition, we present a use case on the humanoid robot ARMAR-III. We present a system for grasping unknown objects, even from piles or cluttered scenes, given a point cloud. Our method is based on the topography of a given scene and abstracts grasp-relevant structures to enable machine learning techniques for grasping tasks. We describe how Height Accumulated Features HAF and their extension, Symmetry Height Accumulated Features, extract grasp relevant local shapes. We investigate grasp quality using an F-score metric. We demonstrate the gain and the expressive power of HAF by comparing its trained classifier with one that resulted from training on simple height grids. An efficient way to calculate HAF is presented. We describe how the trained grasp classifier is used to explore the whole grasp space and introduce a heuristic to find the most robust grasp. We show how to use our approach to adapt the gripper opening width before grasping. In robotic experiments we demonstrate different aspects of our system on three robot platforms: a Schunk seven-degree-of-freedom arm, a PR2 and a Kuka LWR arm. We perform tasks to grasp single objects, autonomously unload a box and clear the table. Thereby we show that our approach is easily adaptable and robust with respect to different manipulators. As part of the experiments we compare our algorithm with a state-of-the-art method and show significant improvements. Concrete examples are used to illustrate the benefit of our approach compared with established grasp approaches. Finally, we show advantages of the symbiosis between our approach and object recognition. Our goal is to automate the understanding of natural hand-object manipulation by developing computer visionbased techniques. Our hypothesis is that it is necessary to model the grasp types of hands and the attributes of manipulated objects in order to accurately recognize manipulation actions. Specifically, we focus on recognizing hand grasp types, object attributes and actions from a single image within an unified model. First, we explore the contextual relationship between grasp types and object attributes, and show how that context can be used to boost the recognition of both grasp types and object attributes. Second, we propose to model actions with grasp types and object attributes based on the hypothesis that grasp types and object attributes contain complementary information for characterizing different actions. Our proposed action model outperforms traditional appearance-based models which are not designed to take into account semantic constraints such as grasp types or object attributes. Experiment results on public egocentric activities datasets strongly support our hypothesis. The ability to grasp large objects with both hands enables bimanual robot systems to fully employ their capabilities in human-centered environments. Hence, algorithms are needed to precompute bimanual grasping configurations that can be used online to efficiently create whole body grasps. In this work we present a bimanual grasp planner that can be used to build a set of grasps together with manipulability information for a given object. For efficient grasp planning precomputed reachability information and a beneficial object representation, based on medial axis descriptions, are used. Since bimanual grasps may suffer from low manipulability, caused by a closed kinematic chain, we show how the manipulability of a bimanual grasp can be used as a quality measure. Therefore, manipulability clusters are introduced as an efficient way to approximatively describe the manipulability of a given bimanual grasp. The proposed approach is evaluated with a reference implementation, based on Simox [1], for the humanoid robot ARMAR-III [2]. Since the presented algorithms are robot-independent, there are no limitations for using this planner on other robot systems. In-hand manipulation is certainly one of the most challenging problems in robotic manipulation. Solutions to this problem depend on the specific device used to grab the object, but nowadays, the trend is to exploit not only the gripper but also external constraints, such as other objects in the environment or external forces, like gravity. This allows a robot to manipulate an object even with very simple grippers, like a parallel gripper. Nevertheless, even for a simple grasping task, which aims at grabbing the object with a given fixed orientation or for executing a controlled slip, information on the contact between the fingers of the gripper and the object is relevant. In these cases, both linear and rotational slipping should be controlled during the grasping phase and during the motion phase. The present paper proposes a control strategy for the first objective, namely slipping avoidance. The strategy is based on contact information provided by a six-axis force tactile sensor, able to measure contact force and torque as well as able to provide information on the contact geometry, that means orientation of the object with respect to the gripper. Experiments on a parallel gripper sensorized with a new force tactile sensor and mounted on a Kuka iiwa show how the strategy successfully allows the robot to safely manipulate a rigid object in various friction conditions of its surface. A grasped object is defined to be in equilibrium if the sum of all forces and moments acting on a body equals zero. An equilibrium grasp may be stable or unstable. Force closed grasps are a well-known subset of equilibrium grasps, and they are known to be stable. However, not all stable grasps are force closed, including many common and easily obtainable grasps. In this paper, we classify the categories of equilibrium grasps and establish a general framework for the determination of the stability of a grasp. In order to analyze the stability of grasps with multiple contacts, we first model the compliance at each contact. We develop expressions for the changes in contact forces as a function of the rigid body relative motion between the fingers and the grasped object. The stability of a grasp is shown to depend on the local curvature of the contacting bodies, as well as the magnitude and arrangement of the contact forces. We then derive results providing simple criteria to determine the stability of a grasped object, including the special but important limiting case of rigid bodies where the contact compliance is zero. Automatic grasp planning for robotic hands is a difficult problem because of the huge number of possible hand configurations. However, humans simplify the problem by choosing an appropriate prehensile posture appropriate for the object and task to be performed. By modeling an object as a set of shape primitives, such as spheres, cylinders, cones and boxes, we can use a set of rules to generate a set of grasp starting positions and pregrasp shapes that can then be tested on the object model. Each grasp is tested and evaluated within our grasping simulator "GraspIt!", and the best grasps are presented to the user. The simulator can also plan grasps in a complex environment involving obstacles and the reachability constraints of a robot arm.