Grasping is a complex process involving knowledge of the object, the surroundings, and of oneself. While humans are able to integrate and process all of the sensory information required for performing this task, equipping machines with this capability is an extremely challenging endeavor. In this paper, we investigate how deep learning techniques can allow us to translate high-level concepts such as motor imagery to the problem of robotic grasp synthesis. We explore a paradigm based on generative models for learning integrated object-action representations, and demonstrate its capacity for capturing and generating multimodal, multi-finger grasp configurations on a simulated grasping dataset.
Among many studies that have examined this phenomenon, we highlight one by Frak al @cite_27 who explored it in the context of which frame of reference is adopted during implicit (unconcious) MI performance. The authors presented evidence that even though MI is an internal process, participants mentally simulating a grasp on a water container did so under real-world biomechanical constraints. That is, grasps or actions that would have been uncomfortable to perform in the real world (e.g. due to awkward joint positioning) were correlated with responses of the mentally simulated action.
@cite_27
2133255759 2963033241 2758237641 2962899390
Five normal subjects were tested in a simulated grasping task. A cylindrical container filled with water was placed on the center of a horizontal monitor screen. Subjects used a precision grip formed by the thumb and index finger of their right hand. After a preliminary run during which the container was present, it was replaced by an image of the upper surface of the cylinder appearing on the horizontal computer screen on which the real cylinder was placed during the preliminary run. In each trial the image was marked with two contact points which defined an opposition axis in various orientations with respect to the frontal plane. The subjects' task consisted, once shown a stimulus, of judging as quickly as possible whether the previously experienced action of grasping the container full of water and pouring the water out would be easy, difficult or impossible with the fingers placed according to the opposition axis indicated on the circle. Response times were found to be longer for the grasps judged to be more difficult due to the orientation and position of the opposition axis. In a control experiment, three subjects actually performed the grasps with different orientations and positions of the opposition axis. The effects of these parameters on response time followed the same trends as during simulated movements. This result shows that simulated hand movements take into account the same biomechanical limitations as actually performed movements. This paper presents a real-time, object-independent grasp synthesis method which can be used for closed-loop grasping. Our proposed Generative Grasping Convolutional Neural Network (GG-CNN) predicts the quality and pose of grasps at every pixel. This one-to-one mapping from a depth image overcomes limitations of current deep-learning grasping techniques by avoiding discrete sampling of grasp candidates and long computation times. Additionally, our GG-CNN is orders of magnitude smaller while detecting stable grasps with equivalent performance to current state-of-the-art techniques. The light- weight and single-pass generative nature of our GG-CNN allows for closed-loop control at up to 50Hz, enabling accurate grasping in non-static environments where objects move and in the presence of robot control inaccuracies. In our real-world tests, we achieve an 83 grasp success rate on a set of previously unseen objects with adversarial geometry and 88 on a set of household objects that are moved during the grasp attempt. We also achieve 81 accuracy when grasping in dynamic clutter. Instrumenting and collecting annotated visual grasping datasets to train modern machine learning algorithms can be extremely time-consuming and expensive. An appealing alternative is to use off-the-shelf simulators to render synthetic data for which ground-truth annotations are generated automatically. Unfortunately, models trained purely on simulated data often fail to generalize to the real world. We study how randomized simulated environments and domain adaptation methods can be extended to train a grasping system to grasp novel objects from raw monocular RGB images. We extensively evaluate our approaches with a total of more than 25,000 physical test grasps, studying a range of simulation conditions and domain adaptation methods, including a novel extension of pixel-level domain adaptation that we term the GraspGAN. We show that, by using synthetic data and domain adaptation, we are able to reduce the number of real-world samples needed to achieve a given level of performance by up to 50 times, using only randomly generated simulated objects. We also show that by using only unlabeled real-world data and our GraspGAN methodology, we obtain real-world grasping performance without any real-world labels that is similar to that achieved with 939,777 labeled real-world samples. Instrumenting and collecting annotated visual grasping datasets to train modern machine learning algorithms can be extremely time-consuming and expensive. An appealing alternative is to use off-the-shelf simulators to render synthetic data for which ground-truth annotations are generated automatically. Unfortunately, models trained purely on simulated data often fail to generalize to the real world. We study how randomized simulated environments and domain adaptation methods can be extended to train a grasping system to grasp novel objects from raw monocular RGB images. We extensively evaluate our approaches with a total of more than 25,000 physical test grasps, studying a range of simulation conditions and domain adaptation methods, including a novel extension of pixel-level domain adaptation that we term the GraspGAN. We show that, by using synthetic data and domain adaptation, we are able to reduce the number of real-world samples needed to achieve a given level of performance by up to 50 times, using only randomly generated simulated objects. We also show that by using only unlabeled real-world data and our GraspGAN methodology, we obtain real-world grasping performance without any real-world labels that is similar to that achieved with 939,777 labeled real-world samples.