Abstract Interaction in virtual reality (VR) environments (e.g. grasping and manipulating virtual objects) is essential to ensure a pleasant and immersive experience. In this work, we propose a visually realistic, flexible and robust grasping system that enables real-time interactions in virtual environments. Resulting grasps are visually realistic because hand is automatically fitted to the object shape from a position and orientation determined by the user using the VR handheld controllers (e.g. Oculus Touch motion controllers). Our approach is flexible because it can be adapted to different hand meshes (e.g. human or robotic hands) and it is also easily customizable. Moreover, it enables interaction with different objects regardless their geometries. In order to validate our proposal, an exhaustive qualitative and quantitative performance analysis has been carried out. On one hand, qualitative evaluation was used in the assessment of abstract aspects, such as motor control, finger movement realism, and interaction realism. On the other hand, for the quantitative evaluation a novel metric has been proposed to visually analyze the performed grips. Performance analysis results indicate that previous experience with our grasping system is not a prerequisite for an enjoyable, natural and intuitive VR interaction experience.
Grasping action is the most basic component of any interaction and it is composed of three major components @cite_21 . The first one is related to the process of approaching the arm and hand to the target object, considering the overall body movement. The second component focuses on the hand and body pre-shaping before the grasping action. Finally, the last component fits the hand to the geometry of the object by closing each of the fingers until contact is established.
@cite_21
1249953932 2074693532 2561266087 2002236162 2047217088 2133255759 2211217408 2066830245 1926515346 2130399002 2001709653 2154006563 1980602022 2023106861 1987985958 2169136412 2963199973 2070678636 1965961478 1969299770
We address the problem of grasping everyday objects that are small relative to an anthropomorphic hand, such as pens, screwdrivers, cellphones, and hammers from their natural poses on a support surface, e.g., a table top. In such conditions, state of the art grasp generation techniques fail to provide robust, achievable solutions due to either ignoring or trying to avoid contact with the support surface. In contrast, we show that contact with support surfaces is critical for grasping small objects. This also conforms with our anecdotal observations of human grasping behaviors. We develop a simple closed-loop hybrid controller that mimics this interactive, contact-rich strategy by a position-force, pre-grasp and landing strategy for finger placement. The approach uses a compliant control of the hand during the grasp and release of objects in order to preserve safety. We conducted extensive grasping experiments on a variety of small objects with similar shape and size. The results demonstrate that our approach is robust to localization uncertainties and applies to many everyday objects. Two recent studies have investigated the relations of eye and hand movements in extended food preparation tasks, and here the results are compared. The tasks could be divided into a series of actions performed on objects. The eyes usually reached the next object in the sequence before any sign of manipulative action, indicating that eye movements are planned into the motor pattern and lead each action. The eyes usually fixated the same object throughout the action upon it, although they often moved on to the next object in the sequence before completion of the preceding action. The specific roles of individual fixations could be identified as locating (establishing the locations of objects for future use), directing (establishing target direction prior to contact), guiding (supervising the relative movements of two or three objects) and checking (establishing whether some particular condition is met, prior to the termination of an action). It is argued that, at the beginning of each action, the oculomotor system is supplied with the identity of the required object, information about its location, and instructions about the nature of the monitoring required during the action. The eye movements during this kind of task are nearly all to task-relevant objects, and thus their control is seen as primarily ‘top-down’, and influenced very little by the ‘intrinsic salience’ of objects. © 2001 Elsevier Science Ltd. All rights reserved. How should a robot direct active vision so as to ensure reliable grasping? We answer this question for the case of dexterous grasping of unfamiliar objects. By dexterous grasping we simply mean grasping by any hand with more than two fingers, such that the robot has some choice about where to place each finger. Such grasps typically fail in one of two ways, either unmodeled objects in the scene cause collisions or object reconstruction is insufficient to ensure that the grasp points provide a stable force closure. These problems can be solved more easily if active sensing is guided by the anticipated actions. Our approach has three stages. First, we take a single view and generate candidate grasps from the resulting partial object reconstruction. Second, we drive the active vision approach to maximise surface reconstruction quality around the planned contact points. During this phase, the anticipated grasp is continually refined. Third, we direct gaze to improve the safety of the planned reach to grasp trajectory. We show, on a dexterous manipulator with a camera on the wrist, that our approach (80.4 success rate) outperforms a randomised algorithm (64.3 success rate). This paper is the first of a two-part series analyzing human grasping behavior during a wide range of unstructured tasks. The results help clarify overall characteristics of human hand to inform many domains, such as the design of robotic manipulators, targeting rehabilitation toward important hand functionality, and designing haptic devices for use by the hand. It investigates the properties of objects grasped by two housekeepers and two machinists during the course of almost 10,000 grasp instances and correlates the grasp types used to the properties of the object. We establish an object classification that assigns each object properties from a set of seven classes, including mass, shape and size of the grasp location, grasped dimension, rigidity, and roundness. The results showed that 55 percent of grasped objects had at least one dimension larger than 15 cm, suggesting that more than half of objects cannot physically be grasped using their largest axis. Ninety-two percent of objects had a mass of 500 g or less, implying that a high payload capacity may be unnecessary to accomplish a large subset of human grasping behavior. In terms of grasps, 96 percent of grasp locations were 7 cm or less in width, which can help to define requirements for hand rehabilitation and defines a reasonable grasp aperture size for a robotic hand. Subjects grasped the smallest overall major dimension of the object in 94 percent of the instances. This suggests that grasping the smallest axis of an object could be a reliable default behavior to implement in grasp planners. This paper is the second in a two-part series analyzing human grasping behavior during a wide range of unstructured tasks. It investigates the tasks performed during the daily work of two housekeepers and two machinists and correlates grasp type and object properties with the attributes of the tasks being performed. The task or activity is classified according to the force required, the degrees of freedom, and the functional task type. We found that 46 percent of tasks are constrained, where the manipulated object is not allowed to move in a full six degrees of freedom. Analyzing the interrelationships between the grasp, object, and task data show that the best predictors of the grasp type are object size, task constraints, and object mass. Using these attributes, the grasp type can be predicted with 47 percent accuracy. Those parameters likely make useful heuristics for grasp planning systems. The results further suggest the common sub-categorization of grasps into power, intermediate, and precision categories may not be appropriate, indicating that grasps are generally more multi-functional than previously thought. We find large and heavy objects are grasped with a power grasp, but small and lightweight objects are not necessarily grasped with precision grasps—even with grasped object size less than 2 cm and mass less than 20 g, precision grasps are only used 61 percent of the time. These results have important implications for robotic hand design and grasp planners, since it appears while power grasps are frequently used for heavy objects, they can still be quite practical for small, lightweight objects. Five normal subjects were tested in a simulated grasping task. A cylindrical container filled with water was placed on the center of a horizontal monitor screen. Subjects used a precision grip formed by the thumb and index finger of their right hand. After a preliminary run during which the container was present, it was replaced by an image of the upper surface of the cylinder appearing on the horizontal computer screen on which the real cylinder was placed during the preliminary run. In each trial the image was marked with two contact points which defined an opposition axis in various orientations with respect to the frontal plane. The subjects' task consisted, once shown a stimulus, of judging as quickly as possible whether the previously experienced action of grasping the container full of water and pouring the water out would be easy, difficult or impossible with the fingers placed according to the opposition axis indicated on the circle. Response times were found to be longer for the grasps judged to be more difficult due to the orientation and position of the opposition axis. In a control experiment, three subjects actually performed the grasps with different orientations and positions of the opposition axis. The effects of these parameters on response time followed the same trends as during simulated movements. This result shows that simulated hand movements take into account the same biomechanical limitations as actually performed movements. 1. The prehensile movements of the hand as a whole are analysed from both an anatomical anda functional viewpoint. 2. It is shown that movements of the hand consist of two basic patterns of movements which are termed precision grip and power grip. 3. In precision grip the object is pinched between the flexor aspects of the fingers and that of the opposing thumb. 4. In power grip the object is held as in a clamp between the flexed fingers and the palm, counter pressure being applied by the thumb lying more or less in the plane of the palm. 5. These two patterns appear to cover the whole range of prehensile activity of the human hand. Typical patterns of hand-joint covariation arising in the context of grasping actions enable one to provide simplified descriptions of these actions in terms of small sets of hand-joint parameters. The computational model of mirror mechanisms introduced here hypothesizes that mirror neurons are crucially involved in coding and making this simplified motor information available for both action recognition and control processes. In particular, grasping action recognition processes are modeled in terms of a visuo-motor loop enabling one to make iterated use of mirror-coded motor information. In simulation experiments concerning the classification of reach-to-grasp actions, mirror-coded information was found to simplify the processing of visual inputs and to improve action recognition results with respect to recognition procedures that are solely based on visual processing. The visuo-motor loop involved in action recognition is a distinctive feature of this model which is coherent with the direct matching hypothesis. Moreover, the visuo-motor loop sets the model introduced here apart from those computational models that identify mirror neuron activity in action observation with the final outcome of computational processes unidirectionally flowing from sensory (and usually visual) to motor systems. We analyzed the coordination between gaze behavior, fingertip movements, and movements of the manipulated object when subjects reached for and grasped a bar and moved it to press a target-switch. Subjects almost exclusively fixated certain landmarks critical for the control of the task. Landmarks at which contact events took place were obligatory gaze targets. These included the grasp site on the bar, the target, and the support surface where the bar was returned after target contact. Any obstacle in the direct movement path and the tip of the bar were optional landmarks. Subjects never fixated the hand or the moving bar. Gaze and hand bar movements were linked concerning landmarks, with gaze leading. The instant that gaze exited a given landmark coincided with a kinematic event at that landmark in a manner suggesting that subjects monitored critical kinematic events for phasic verification of task progress and subgoal completion. For both the obstacle and target, subjects directed saccades and fixations to sites that were offset from the physical extension of the objects. Fixations related to an obstacle appeared to specify a location around which the extending tip of the bar should travel. We conclude that gaze supports hand movement planning by marking key positions to which the fingertips or grasped object are subsequently directed. The salience of gaze targets arises from the functional sensorimotor requirements of the task. We further suggest that gaze control contributes to the development and maintenance of sensorimotor correlation matrices that support predictive motor control in manipulation. In several hand-object(s) interaction scenarios, the change in the objects' state is a direct consequence of the hand's motion. This has a straightforward representation in Newtonian dynamics. We present the first approach that exploits this observation to perform model-based 3D tracking of a table-top scene comprising passive objects and an active hand. Our forward modelling of 3D hand-object(s) interaction regards both the appearance and the physical state of the scene and is parameterized over the hand motion (26 DoFs) between two successive instants in time. We demonstrate that our approach manages to track the 3D pose of all objects and the 3D pose and articulation of the hand by only searching for the parameters of the hand motion. In the proposed framework, covert scene state is inferred by connecting it to the overt state, through the incorporation of physics. Thus, our tracking approach treats a variety of challenging observability issues in a principled manner, without the need to resort to heuristics. Gestures are characterized by intermediary or final landmarks (real or virtual) in task space or joint space that can change during the course of the motion, and that are described by varying accuracy and correlation constraints. Generalizing these trajectories in robot learning by imitation is challenging, because of the small number of demonstrations provided by the user. We present an approach to statistically encode movements in a task-parameterized mixture model, and derive an expectation-maximization (EM) algorithm to train it. The model automatically extracts the relevance of candidate coordinate systems during the task, and exploits this information during reproduction to adapt the movement in real-time to changing position and orientation of landmarks or objects. The approach is tested with a robotic arm learning to roll out a pizza dough. It is compared to three categories of task-parameterized models: 1) Gaussian process regression (GPR) with a trajectory models database; 2) Multi-streams approach with models trained in several frames of reference; and 3) Parametric Gaussian mixture model (PGMM) modulating the Gaussian centers with the task parameters. We show that the extrapolation capability of the proposed approach outperforms existing methods, by extracting the local structures of the task instead of relying on interpolation principles. Subjects were asked to shape the right hand as if to grasp and use a large number of familiar objects. The chosen objects typically are held with a variety of grips, including “precision” and “power” grips. Static hand posture was measured by recording the angular position of 15 joint angles of the fingers and of the thumb. Although subjects adopted distinct hand shapes for the various objects, the joint angles of the digits did not vary independently. Principal components analysis showed that the first two components could account for .80 of the variance, implying a substantial reduction from the 15 degrees of freedom that were recorded. However, even though they were small, higher-order (more than three) principal components did not represent random variability but instead provided additional information about the object. These results suggest that the control of hand posture involves a few postural synergies, regulating the general shape of the hand, coupled with a finer control mechanism providing for small, subtle adjustments. Because the postural synergies did not coincide with grip taxonomies, the results suggest that hand posture may be regulated independently from the control of the contact forces that are used to grasp an object. We present a novel and simple experimental method called physical human interactive guidance to study human-planned grasping. Instead of studying how the human uses his her own biological hand or how a human teleoperates a robot hand in a grasping task, the method involves a human interacting physically with a robot arm and hand, carefully moving and guiding the robot into the grasping pose, while the robot's configuration is recorded. Analysis of the grasps from this simple method has produced two interesting results. First, the grasps produced by this method perform better than grasps generated through a state-of-the-art automated grasp planner. Second, this method when combined with a detailed statistical analysis using a variety of grasp measures (physics-based heuristics considered critical for a good grasp) offered insights into how the human grasping method is similar or different from automated grasping synthesis techniques. Specifically, data from the physical human interactive guidance method showed that the human-planned grasping method provides grasps that are similar to grasps from a state-of-the-art automated grasp planner, but differed in one key aspect. The robot wrists were aligned with the object's principal axes in the human-planned grasps (termed low skewness in this paper), while the automated grasps used arbitrary wrist orientation. Preliminary tests show that grasps with low skewness were significantly more robust than grasps with high skewness (77-93 ). We conclude with a detailed discussion of how the physical human interactive guidance method relates to existing methods to extract the human principles for physical interaction. Traditionally, grasp and arm motion planning are considered as separate tasks. This paper presents an integrated approach that only requires the initial configuration of the robotic arm and the pose of the target object to simultaneously plan a good hand pose and arm trajectory to grasp the object. The planner exploits the concept of independent contact regions to look for the best possible grasp. The goal poses for the end effector are obtained using two different methods: one that biases a sampling approach towards favorable regions using principal component analysis, and another one that considers the capabilities of the robotic arm to decide the most promising hand poses. The proposed method is evaluated using different scenarios for the humanoid robot Spacejustin. Many manipulation planning problems involve several related sub-problems, such as the selection of grasping points on an object, choice of hand posture, and determination of the arm's configuration and evolving trajectory. Traditionally, these planning sub-problems have been handled separately, potentially leading to sub-optimal, or even infeasible, combinations of the individually determined solutions. This paper formulates the combined problem of grasp contact selection, grasp force optimization, and manipulator arm hand trajectory planning as a problem in optimal control. That is, the locally optimal trajectory for the manipulator, hand mechanism, and contact locations are determined during the pre-grasping, grasping, and subsequent object transport phase. Additionally, a barrier function approach allows for non-feasible grasps to be optimized, enlarging the region of convergence for the algorithm. A simulation of a simple planar object manipulation task is used to illustrate and validate the approach. We present an approach that enables robots to learn motion primitives that are robust towards state estimation uncertainties. During reaching and preshaping, the robot learns to use fine manipulation strategies to maneuver the object into a pose at which closing the hand to perform the grasp is more likely to succeed. In contrast, common assumptions in grasp planning and motion planning for reaching are that these tasks can be performed independently, and that the robot has perfect knowledge of the pose of the objects in the environment. This paper explores the problem of autonomous, in-hand regrasping-the problem of moving from an initial grasp on an object to a desired grasp using the dexterity of a robot's fingers. We propose a planner for this problem which alternates between finger gaiting, and in-grasp manipulation. Finger gaiting enables the robot to move a single finger to a new contact location on the object, while the remaining fingers stably hold the object. In-grasp manipulation moves the object to a new pose relative to the robot's palm, while maintaining the contact locations between the hand and object. Given the object's geometry (as a mesh), the hand's kinematic structure, and the initial and desired grasps, we plan a sequence of finger gaits and object reposing actions to reach the desired grasp without dropping the object. We propose an optimization based approach and report in-hand regrasping plans for 5 objects over 5 in-hand regrasp goals each. The plans generated by our planner are collision free and guarantee kinematic feasibility. Grasping an object is a task that inherently needs to be treated in a hybrid fashion. The system must decide both where and how to grasp the object. While selecting where to grasp requires learning about the object as a whole, the execution only needs to reactively adapt to the context close to the grasp's location. We propose a hierarchical controller that reflects the structure of these two sub-problems, and attempts to learn solutions that work for both. A hybrid architecture is employed by the controller to make use of various machine learning methods that can cope with the large amount of uncertainty inherent to the task. The controller's upper level selects where to grasp the object using a reinforcement learner, while the lower level comprises an imitation learner and a vision-based reactive controller to determine appropriate grasping motions. The resulting system is able to quickly learn good grasps of a novel object in an unstructured environment, by executing smooth reaching motions and preshaping the hand depending on the object's geometry. The system was evaluated both in simulation and on a real robot. A grasped object is defined to be in equilibrium if the sum of all forces and moments acting on a body equals zero. An equilibrium grasp may be stable or unstable. Force closed grasps are a well-known subset of equilibrium grasps, and they are known to be stable. However, not all stable grasps are force closed, including many common and easily obtainable grasps. In this paper, we classify the categories of equilibrium grasps and establish a general framework for the determination of the stability of a grasp. In order to analyze the stability of grasps with multiple contacts, we first model the compliance at each contact. We develop expressions for the changes in contact forces as a function of the rigid body relative motion between the fingers and the grasped object. The stability of a grasp is shown to depend on the local curvature of the contacting bodies, as well as the magnitude and arrangement of the contact forces. We then derive results providing simple criteria to determine the stability of a grasped object, including the special but important limiting case of rigid bodies where the contact compliance is zero. Abstract Two experiments establish links between desired knowledge about objects and hand movements during haptic object exploration. Experiment 1 used a match-to-sample task, in which blindfolded subjects were directed to match objects on a particular dimension (e.g., texture). Hand movements during object exploration were reliably classified as “exploratory procedures,” each procedure defined by its invariant and typical properties. The movement profile, i.e., the distribution of exploratory procedures, was directly related to the desired object knowledge that was required for the match. Experiment 2 addressed the reasons for the specific links between exploratory procedures and knowledge goals. Hand movements were constrained, and performance on various matching tasks was assessed. The procedures were considered in terms of their necessity, sufficiency, and optimality of performance for each task. The results establish that in free exploration, a procedure is generally used to acquire information about an object property, not because it is merely sufficient, but because it is optimal or even necessary. Hand movements can serve as “windows,” through which it is possible to learn about the underlying representation of objects in memory and the processes by which such representations are derived and utilized.