In this work, we provide a solution for posturing the anthropomorphic Robonaut-2 hand and arm for grasping based on visual information. A mapping between visual features extracted from a convolutional neural network (CNN) to grasp points is learned. We demonstrate that a pre-trained CNN for image classification can be applied to a grasping task based on a small set of grasping examples. Our approach takes advantage of the hierarchical nature of the CNN and identifies the 3D positions of features that capture the hierarchical support relations between filters in different CNN layers by tracing the activation of higher level features in the CNN backward. When this backward trace terminates in the RGB-D image, important manipulable structures comprising the objects are, thus, localized. These features located in different layers of the CNN are then associated to controllers belonging to different hierarchies of the robot morphology for grasping. A Grasping Dataset is collected using demonstrated hand object relationships for Robonaut-2 to evaluate the proposed approach in terms of the precision of the resulting preshape postures. We demonstrate that this approach outperforms base-line approaches in cluttered scenarios on the Grasping Dataset and a point cloud based approach on a grasping task using Robonaut-2.
The idea that our brain encodes visual stimuli in two separate regions was first proposed by Schneider @cite_7 . Ungerleider and Mishkin further discovered the ventral and dorsal streams and proposed the hypothesis often known as the distinction of what" and where" between the two visual pathways @cite_6 . However, in 1992 Goodale and Milner proposed an alternative perspective on the functionality of these two visual pathways based on many observations made with patient DF @cite_20 . Patient DF developed a profound visual form agnosia due to damage to her ventral stream. Despite DF's inability to recognize the shape, size and orientation of visual objects, she is capable of grasping the object with accurate hand and finger movements. Based on a series of experiments @cite_31 , Goodale and Milner proposed the , which suggests that the dorsal pathway provides action-relevant information about the structural characteristic of objects in addition to their position. Our work is inspired by these observations and associates grasp configurations with visual features instead of object identities.
@cite_31 @cite_20 @cite_6 @cite_7
2107366978 2082627290 2183231851 1497599070
Humans recognize visually-presented objects rapidly and accurately. To understand this ability, we seek to construct models of the ventral stream, the series of cortical areas thought to subserve object recognition. One tool to assess the quality of a model of the ventral stream is the Representational Dissimilarity Matrix (RDM), which uses a set of visual stimuli and measures the distances produced in either the brain (i.e. fMRI voxel responses, neural firing rates) or in models (features). Previous work has shown that all known models of the ventral stream fail to capture the RDM pattern observed in either IT cortex, the highest ventral area, or in the human ventral stream. In this work, we construct models of the ventral stream using a novel optimization procedure for category-level object recognition problems, and produce RDMs resembling both macaque IT and human ventral stream. The model, while novel in the optimization procedure, further develops a long-standing functional hypothesis that the ventral visual stream is a hierarchically arranged series of processing stages optimized for visual object recognition. Abstract Accumulating neuropsychological, electrophysiological and behavioural evidence suggests that the neural substrates of visual perception may be quite distinct from those underlying the visual control of actions. In other words, the set of object descriptions that permit identification and recognition may be computed independently of the set of descriptions that allow an observer to shape the hand appropriately to pick up an object. We propose that the ventral stream of projections from the striate cortex to the inferotemporal cortex plays the major role in the perceptual identification of objects, while the dorsal stream projecting from the striate cortex to the posterior pariet al region mediates the required sensorimotor transformations for visually guided actions directed at such objects. We propose a novel attentional model for simultaneous object tracking and recognition that is driven by gaze data. Motivated by theories of the human perceptual system, the model consists of two interacting pathways: ventral and dorsal. The ventral pathway models object appearance and classification using deep (factored)-restricted Boltzmann machines. At each point in time, the observations consist of retinal images, with decaying resolution toward the periphery of the gaze. The dorsal pathway models the location, orientation, scale and speed of the attended object. The posterior distribution of these states is estimated with particle filtering. Deeper in the dorsal pathway, we encounter an attentional mechanism that learns to control gazes so as to minimize tracking uncertainty. The approach is modular (with each module easily replaceable with more sophisticated algorithms), straightforward to implement, practically efficient, and works well in simple video sequences. A number of psychophysical studies concerning the detection, localization and recognition of objects in the visual field have suggested a two-stage theory of human visual perception. The first stage is the “preattentive” mode, in which simple features are processed rapidly and in parallel over the entire visual field. In the second, “attentive” mode, a specialized processing focus, usually called the focus of attention, is directed to particular locations in the visual field. The analysis of complex forms and the recognition of objects are associated with this second stage.1 The computational justification for such a hypothesis comes from the realization that while it is possible to imagine specific algorithms performing tasks such as shape analysis and recognition at specific locations, it is difficult to imagine these algorithms operating in parallel over the whole visual scene, since such an approach will quickly lead to a combinatorial explosion in terms of required computational resources.2 This is essentially the major critique of Minsky and Papert to a universal application of perceptrons in visual perception.3 Taken together, these empirical and theoretical studies suggest that beyond a certain preprocessing stage, the analysis of visual information proceeds in a sequence of operations, each one applied to a selected location (or locations).