Grasping is a complex process involving knowledge of the object, the surroundings, and of oneself. While humans are able to integrate and process all of the sensory information required for performing this task, equipping machines with this capability is an extremely challenging endeavor. In this paper, we investigate how deep learning techniques can allow us to translate high-level concepts such as motor imagery to the problem of robotic grasp synthesis. We explore a paradigm based on generative models for learning integrated object-action representations, and demonstrate its capacity for capturing and generating multimodal, multi-finger grasp configurations on a simulated grasping dataset.
Among many studies that have examined this phenomenon, we highlight one by Frak al @cite_27 who explored it in the context of which frame of reference is adopted during implicit (unconcious) MI performance. The authors presented evidence that even though MI is an internal process, participants mentally simulating a grasp on a water container did so under real-world biomechanical constraints. That is, grasps or actions that would have been uncomfortable to perform in the real world (e.g. due to awkward joint positioning) were correlated with responses of the mentally simulated action.
@cite_27
2133255759
Five normal subjects were tested in a simulated grasping task. A cylindrical container filled with water was placed on the center of a horizontal monitor screen. Subjects used a precision grip formed by the thumb and index finger of their right hand. After a preliminary run during which the container was present, it was replaced by an image of the upper surface of the cylinder appearing on the horizontal computer screen on which the real cylinder was placed during the preliminary run. In each trial the image was marked with two contact points which defined an opposition axis in various orientations with respect to the frontal plane. The subjects' task consisted, once shown a stimulus, of judging as quickly as possible whether the previously experienced action of grasping the container full of water and pouring the water out would be easy, difficult or impossible with the fingers placed according to the opposition axis indicated on the circle. Response times were found to be longer for the grasps judged to be more difficult due to the orientation and position of the opposition axis. In a control experiment, three subjects actually performed the grasps with different orientations and positions of the opposition axis. The effects of these parameters on response time followed the same trends as during simulated movements. This result shows that simulated hand movements take into account the same biomechanical limitations as actually performed movements.