Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> A number of psychophysical studies concerning the detection, localization and recognition of objects in the visual field have suggested a two-stage theory of human visual perception. The first stage is the “preattentive” mode, in which simple features are processed rapidly and in parallel over the entire visual field. In the second, “attentive” mode, a specialized processing focus, usually called the focus of attention, is directed to particular locations in the visual field. The analysis of complex forms and the recognition of objects are associated with this second stage.1 The computational justification for such a hypothesis comes from the realization that while it is possible to imagine specific algorithms performing tasks such as shape analysis and recognition at specific locations, it is difficult to imagine these algorithms operating in parallel over the whole visual scene, since such an approach will quickly lead to a combinatorial explosion in terms of required computational resources.2 This is essentially the major critique of Minsky and Papert to a universal application of perceptrons in visual perception.3 Taken together, these empirical and theoretical studies suggest that beyond a certain preprocessing stage, the analysis of visual information proceeds in a sequence of operations, each one applied to a selected location (or locations). <s> BIB001 </s> Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> This article provides a systems framework for the analysis of cortical and subcortical interactions in the control of saccadic eye movements, A major thesis of this model is that a topography of saccade direction and amplitude is preserved through multiple projections between brain regions until they are finally transformed into a temporal pattern of activity that drives the eyes to the target. The control of voluntary saccades to visual and remembered targets is modeled in terms of interactions between posterior parietal cortex, frontal eye fields, the basal ganglia (caudate and substantia nigra), superior colliculus, mediodorsal thalamus, and the saccade generator of the brainstem. Interactions include the modulation of eye movement motor error maps by topographic inhibitory projections, dynamic remapping of spatial target representations in saccade motor error maps, and sustained neural activity that embodies spatial memory. Models of these mechanisms implemented in our Neural Simulation Language simulate behavior and neural activity described in the literature, and suggest new experiments. <s> BIB002 </s> Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> Abstract The spatial interaction of visual attention and saccadic eye movements was investigated in a dual-task paradigm that required a target-directed saccade in combination with a letter discrimination task. Subjects had to saccade to locations within horizontal letter strings left and right of a central fixation cross. The performance in discriminating between the symbols “E” and “∃”, presented tachistoscopically before the saccade within the surrounding distractors was taken as a measure of visual attention. The data show that visual discrimination is best when discrimination stimulus and saccade target refer to the same object; discrimination at neighboring items is close to chance level. Also, it is not possible, in spite of prior knowledge of discrimination target position, to direct attention to the discrimination target while saccading to a spatially close saccade target. The data strongly argue for an obligatory and selective coupling of saccade programming and visual attention to one common target object. The results favor a model in which a single attentional mechanism selects objects for perceptual processing and recognition, and also provides the information necessary for motor action. <s> BIB003 </s> Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> An important aspect of computer-controlled camera motion systems is that of the generation of saccadic movements, which shift the camera gaze quickly from one fixation position to another. Recent psychophysical experiments suggest that there exists a causal connection between spatial shifts in visual attention and the production of saccadic eye movements in humans. Motivated by this experimental evidence, we propose a winner-take-all based model of exogenous spatial attention, involving both sustained and transient feature detection channels, and link it to the targetting and triggering of saccadic eye movements. We show that this model accounts for a range of oculomotor phenomena observed in human subjects. We describe the application of this model to a robotic camera gaze control system. <s> BIB004 </s> Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> For a humanoid robot to interact easily with a person, the robot should have human-like sensory capabilities and attentional mechanisms. Particularly useful is an active vision head controlled by a visual attention system that selects viewpoints in the environment as a function of the robot's task. This paper describes a model of human visual attention called FeatureGate, which is a locally connected, pyramidal, artificial neural network that operates on 2D feature maps of the environment. Given a set of feature maps, and the description of a specific target, FeatureGate finds the location whose features most closely match those of the target. The paper describes the network, its implementation, a series of tests that characterize its performance with respect to a person's performance on a similar task, and its use in the control of an active vision system. <s> BIB005 </s> Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> A visual attention system, inspired by the behavior and the neuronal architecture of the early primate visual system, is presented. Multiscale image features are combined into a single topographical saliency map. A dynamical neural network then selects attended locations in order of decreasing saliency. The system breaks down the complex problem of scene understanding by rapidly selecting, in a computationally efficient manner, conspicuous locations to be analyzed in detail. Index Terms—Visual attention, scene analysis, feature extraction, target detection, visual search. ———————— F ———————— <s> BIB006 </s> Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> According to the object-based theory, visual attention can be divided within a single object more efficiently than between separate objects. In this paper, the object-based theory of visual attention was validated through a comparison with the spatially based view, suggesting that visual attention is directed to unparsed regions of space. In testing this theory, various perceptual judgements of location were given by subjects in response to objects. Among the variables in presentation were: the direction of the objects, the distance between two locations, the number of objects involved and the display condition (joined, congruent, incongruent, etc.). In other words, the single- and two-object displays were compared by investigating the effects of these experimental factors. As a result, the object-based theory of visual attention was found to be applicable to hexagons of equal convexity, but not applicable to the other forms within the range of this experiment. <s> BIB007 </s> Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> The model presented here is an attempt to explain the results from a number of different studies in visual attention, including parallel feature searches and serial conjunction searches, variations in search slope with variations in feature contrast and individual subject differences, attentional gradients triggered by cuing, feature-driven spatial selection, split attention, inhibition of distractor locations, and flanking inhibition. The model is implemented in a neural network consisting of a hierarchy of spatial maps. Attentional gates control the flow of information from each level of the hierarchy to the next. The gates are jointly controlled by a Bottom-Up System favoring locations with unique features and a Top-Down System favoring locations with features designated as target features. Because the gating of each location depends on the features present there, the model is called FeatureGate. <s> BIB008 </s> Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> Most models of visual search, whether involving overt eye movements or covert shifts of attention, are based on the concept of a saliency map, that is, an explicit two-dimensional map that encodes the saliency or conspicuity of objects in the visual environment. Competition among neurons in this map gives rise to a single winning location that corresponds to the next attended target. Inhibiting this location automatically allows the system to attend to the next most salient location. We describe a detailed computer implementation of such a scheme, focusing on the problem of combining information across modalities, here orientation, intensity and color information, in a purely stimulus-driven manner. The model is applied to common psychophysical stimuli as well as to a very demanding visual search task. Its successful performance is used to address the extent to which the primate visual system carries out visual search via one or more such saliency maps and how this can be tested. <s> BIB009 </s> Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> We analyzed the coordination between gaze behavior, fingertip movements, and movements of the manipulated object when subjects reached for and grasped a bar and moved it to press a target-switch. Subjects almost exclusively fixated certain landmarks critical for the control of the task. Landmarks at which contact events took place were obligatory gaze targets. These included the grasp site on the bar, the target, and the support surface where the bar was returned after target contact. Any obstacle in the direct movement path and the tip of the bar were optional landmarks. Subjects never fixated the hand or the moving bar. Gaze and hand/bar movements were linked concerning landmarks, with gaze leading. The instant that gaze exited a given landmark coincided with a kinematic event at that landmark in a manner suggesting that subjects monitored critical kinematic events for phasic verification of task progress and subgoal completion. For both the obstacle and target, subjects directed saccades and fixations to sites that were offset from the physical extension of the objects. Fixations related to an obstacle appeared to specify a location around which the extending tip of the bar should travel. We conclude that gaze supports hand movement planning by marking key positions to which the fingertips or grasped object are subsequently directed. The salience of gaze targets arises from the functional sensorimotor requirements of the task. We further suggest that gaze control contributes to the development and maintenance of sensorimotor correlation matrices that support predictive motor control in manipulation. <s> BIB010 </s> Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> The goal of our research is to investigate the interplay between oculomotor control, visual processing, and limb control in humans and primates by exploring the computational issues of these processes with a biologically inspired artificial oculomotor system on an anthropomorphic robot. In this paper, we investigate the computational mechanisms for visual attention in such a system. Stimuli in the environment excite a dynamical neural network that implements a saliency map, i.e., a winner-take-all competition between stimuli while simultaneously smoothing out noise and suppressing irrelevant inputs. In real-time, this system computes new targets for the shift of gaze, executed by the head-eye system of the robot. The redundant degrees-of-freedom of the head-eye system are resolved through a learned inverse kinematics with optimization criterion. We also address important issues how to ensure that the coordinate system of the saliency map remains correct after movement of the robot. The presented attention system is built on principled modules and generally applicable for any sensory modality. <s> BIB011 </s> Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> One of the first steps of any visual system is that of locating suitable interest points, "salient regions", in the scene, to detect events, and eventually to direct gaze toward these locations. In the last few years, object-based visual attention models have received an increasing interest in the literature, the problem, in this case, being that of creating a model of "objecthood" that eventually guides a saliency mechanism. We propose here an object-based model of visual attention and show its instantiation on a humanoid robot. The robot employs action to learn and define its own concept of objecthood. <s> BIB012 </s> Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> Complex visual processes such as visual attention are often computationally too expensive to allow real-time implementation on a single computer. To solve this problem we study distributed computer architectures that enable us to divide complex tasks into several smaller problems. In this paper we demonstrate how to implement distributed visual attention system on a humanoid robot to achieve real-time operation at relatively high resolutions and frame rates. We start from a popular theory of bottom-up visual attention that assumes that information across various modalities is used for the early encoding of visual information. Our system uses five different modalities including color, intensity, edges, stereo, and motion. We show how to distribute the attention processing on a computer cluster and study the issues arising on such systems. The system was fully implemented on a workstation cluster comprised of eight PCs. It was used to drive the gaze of a humanoid head towards potential regions of interest <s> BIB013 </s> Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> Although biomimetic autonomous robotics relies on the massively parallel architecture of the brain, the key issue is to temporally organize behaviour. The distributed representation of the sensory information has to be coherently processed to generate relevant actions. In the visual domain, we propose here a model of visual exploration of a scene by the means of localized computations in neural populations whose architecture allows the emergence of a coherent behaviour of sequential scanning of salient stimuli. It has been implemented on a real robotic platform exploring a moving and noisy scene including several identical targets. <s> BIB014 </s> Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> Robot companions are intended for operation in private homes with naive users. For this purpose, they need to be endowed with natural interaction capabilities. Additionally, such robots will need to be taught unknown objects that are present in private homes. We present a multi-modal object attention system that is able to identify objects referenced by the user with gestures and verbal instructions. The proposed system can detect known and unknown objects and stores newly acquired object information in a scene model for later retrieval. This way, the growing knowledge base of the robot companion improves the interaction quality as the robot can more easily focus its attention on objects it has been taught previously. <s> BIB015 </s> Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> A major challenge for the realization of intelligent robots is to supply them with cognitive abilities in order to allow ordinary users to program them easily and intuitively. One approach to such programming is teaching work tasks by interactive demonstration. To make this effective and convenient for the user, the machine must be capable of establishing a common focus of attention and be able to use and integrate spoken instructions, visual perception, and non-verbal clues like gestural commands. We report progress in building a hybrid architecture that combines statistical methods, neural networks, and finite state machines into an integrated system for instructing grasping tasks by man-machine interaction. The system combines the GRAVIS-robot for visual attention and gestural instruction with an intelligent interface for speech recognition and linguistic interpretation, and a modality fusion module to allow multi-modal task-oriented man-machine communication with respect to dextrous robot manipulation of objects. <s> BIB016 </s> Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> The Sensory Ego-Sphere (SES) is a short-term memory for a robot in the form of an egocentric, tessellated, spherical, sensory-motor map of the robot's locale. This paper reports on: (1) the mapping to the SES of images with much higher resolution than the SES itself, and (2) the processing of visual attention on the SES. Described is a procedure to store spatially-overlapping imagery in the SES database and to generate and update a spherical composite of its visual contents. The composite image serves both as a visual map of the locale and as a representation of the local contents of the underlying full-resolution imagery. Visual attention enables fast alignment of overlapping images without warping or position optimization, since an attentional point (AP) on the composite typically corresponds to one on each of the collocated regions in the images. Such alignment speeds analysis of the multiple images of the area. Compositing and attention were performed two ways and compared: (1) APs were computed directly on the composite and not on the full resolution images until the time of retrieval. (2) The attentional operator was applied to all incoming imagery. Then the locations and values of the APs on the composite were computed as a function of those points within the corresponding regions of the collocated images. It was found that although the second method was slower, it produced consistent and, thereby, more useful APs. <s> BIB017 </s> Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> We develop an architecture for reactive visual analysis of dynamic scenes. We specify a minimal set of system features based upon biological observations. We implement feature on a processing network based around an active stereo vision mechanism. Active rectification and mosaicing allows static stereo algorithms to operate on the active platform. Foveal zero disparity operations permit attended object extraction and ensures coordinated stereo fixation upon visual surfaces. Activedynamic inhibition of return, and task dependent biasing result in a flexible, preemptive and retrospective system that responds to unique visual stimuli and is capable of top-down modulation of attention towards regions and cues relevant to tasks. <s> BIB018 </s> Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> This work presents a multimodal bottom-up attention system for the humanoid robot iCub where the robot's decisions to move eyes and neck are based on visual and acoustic saliency maps. We introduce a modular and distributed software architecture which is capable of fusing visual and acoustic saliency maps into one egocentric frame of reference. This system endows the iCub with an emergent exploratory behavior reacting to combined visual and auditory saliency. The developed software modules provide a flexible foundation for the open iCub platform and for further experiments and developments, including higher levels of attention and representation of the peripersonal space. <s> BIB019 </s> Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> Abstract A visual overt attention mechanism is presented in this paper. Our algorithm chooses the next fixation point for a robot mobile camera in order to track several objects around the robot simultaneously, even if the whole set of them can not be covered by the same camera image. Our approach is based on two related measurements, liveliness and saliency, that dynamically evolve depending on the image observations and the camera movements. The attention is shared among exploring the surroundings for salient features, reobserving the tracked objects and other task-dependent points to look at. A winner-takes-all competition provides a flexible time sharing behavior with natural appearances of new objects, disappearances and inhibition of return. It also accepts top down features to look for and different priorities for them. Several experiments have been carried out with a real Pioneer robot endowed with mobile camera and are also described. <s> BIB020 </s> Visual Attention for Robotic Cognition: A Survey <s> A. Overt Attention Models <s> During the lifetime of a mobile robot, the number and complexity of the stimuli it receives may be quite high. Therefore, the construction of a detection system considering the whole sensorial space is usually not a viable proposition when aiming for real time operation. It becomes necessary to build some kind of sensorial hierarchy map to put some order into how detectors are applied. This is what is usually called an attentional system, and it provides a framework for applying detectors in a more efficient manner. In this paper, an architecture for developing attentional functions for robots that must operate in real time in dynamic environments is presented. This architecture is based on the concept of attentor and it allows for the real time adaptation to the environment and tasks to be performed in a natural manner. One of the main requirements imposed on the design of the architecture was the capability of handling different sensorial modalities and attentional streams in a transparent manner while, at the same time, being able to progressively create more complex attentional structures. The architecture is particularized for its implementation in a real robot. <s> BIB021
The attention mechanism in the primates integrates the overt and covert modes of attention in a highly efficient manner. The stimulus of interest is selected covertly and then placed at the foveal region through overt movement of the eyes BIB003 . Evidences are also available in favor of the independent occurrence of covert and overt attention BIB010 ,  . In case of robotics applications, however, direction of attention mediated by eye/head movement is the most suitable choice. The major reason behind this is placing the object of interest at the center of visual field facilitates the learning process. Besides, head/eye movement of the robot provides a way for the user to understand the current gaze of the robot which is specially important in many applications (e.g., HRI). Inspired by these requirements, a number of efforts are observed in the robotic literature for modeling of overt visual attention. At the early stage of this research the principle of overt attention (to place an object of interest at the center of visual field) helped the concept of "active vision"  , "active perception"  , or "animate vision"  to be established in computer vision. For instance, the theme of "active vision" is to actively position a sensor (preferably a camera) for obtaining enriched information to solve the basic computer vision problems (e.g., shape from shading and depth computation, shape from contour, shape from texture, and structure from motion). A number of active vision models propose mechanism of positioning a camera based on the feedback from a visual attention model BIB004 - BIB005 . The major focus of most of these models is the control aspects of saccade generation and/or smooth pursuit tracking. A common practice among these works is to use some well-known covert models of attention (e.g., BIB001 and BIB006 ) to identify the most interesting/salient region in the image. These active vision models, therefore, are less concerned about the research issues stated in Section III-B. The overt attention models described in BIB012 and BIB013 - BIB019 are designed to implement in the robots/robotic heads as a component of their cognition. Among them, the models in BIB012 , BIB013 , BIB011 , and BIB018 adopt different variants of the covert model NVT BIB006 to identify the visually salient/task-relevant stimuli and introduced different measures to deal with the research issues involved with robotic overt attention. For instance, the model in BIB011 addresses the Issue 1.1 by adopting the idea of shifting the entire content of the saliency map in the direction of head movement as suggested in BIB002 . Another approach to address the Issue 1.1 is to consider that attention is directed to unparsed regions of space and thereby making the perception independent of space BIB007 . The object-based overt attention system proposed in BIB012 implements a simple form of integrated object-and space-based IOR to deal with Issue 1.2 and Issue 2. The overt model described in BIB018 suggests to remap the location of the recently attended object to the transformed image coordinate in order to implement a space-based IOR (Issue 1.2) . The problem involved with the partial appearance of objects (Issue 1.3) is not noticeable in the experiments demonstrated in BIB018 due to the use of a wide angle camera. The model in BIB013 demonstrates few simple cases of overt attention and does not provide any effective solution to any of the research issues. The neural network-based overt model reported in BIB014 is tightly coupled with biology (with respect to motor aspects of attention) and is focused on implementing visual exploration behavior guided by the novelty preference characteristics of primates attention. The identification of novelty in BIB014 , however, is achieved though the implementation of space-based IOR, i.e., the robot moves to novel locations (through successive application of space-based IOR) and thereby attends to novel objects. The model BIB014 also relies on NVT BIB009 for visual saliency calculation. The issue of dynamic IOR (Issue 1.2) is addressed by remembering the locations of the previously visited stimuli. To comply with this strategy the model BIB014 assumes that all of the stimuli stay within the visual field of the robot at all times. This is a strong assumption which is valid in the experiments demonstrated in BIB014 , but generally does not hold good in most real world scenario. The Feature Gate model -based BIB008 overt model in  claims to proposes a general purpose model of visual attention for the humanoid robots but mostly focuses on mimicking the feature-processing attributes of the primates attention system (e.g., log-polar retino-cortical mapping, banks of oriented filter). All of the overt models discussed thus far follow an imagecentric approach where the attention model operates absolutely in the image plane. Focus of attention is evaluated based on the content of a given image and necessary motion command is calculated based on the image dimension and the parameters of the camera optics. As opposed to this traditional image-centric approach, the recent models of overt attention adopt a robot-centric solution for attentional selection BIB021 , BIB020 , BIB019 , BIB017 . In case of robot-centric approach it is assumed that a robot is a human-like autonomous entity which decides "what to look at?" based on its perception of surrounding with respect to an egocentric frame of reference. For instance, the model in BIB019 considers an ego-sphere of infinite radius around a robotic head and the robot is able to project the perceptual information collected through different modality on the surface of the ego-sphere. The concept of the head-centric ego-sphere provides an elegant solution of the issues involved with overt shift of attention (Issues 1.1, 1.2, 1.3) . The multimodal attention model in BIB019 considers both acoustic and visual information and combines them into a single head-centric saliency map by taking the maximum value between the two modes. This straightforward methodology of fusing multimodal perception into a single saliency map has several shortcomings, e.g., saliency map from different modes have same influence on the aggregated saliency map. A detailed analysis of this problem is available in  . The model BIB019 operates in a purely bottom-up fashion and performs NVT BIB009 -like space-based analysis for saliency calculation. The concept of an ego-sphere is also present in the attention model reported in BIB017 . The model BIB017 , however, uses the principle of attention for updating a sensory ego-sphere with overlapping images perceived by the robot. The multimodal attention model in  also uses sensory ego-sphere to focus, learn, and then track the salient stimuli (bright colored moving objects or human faces) in the visual field. The model  integrates the visual search and visual exploration in the same framework and thereby eliminates the presence of a training phase during visual search (Issue 4, 5) . The overt model in BIB020 uses the term "scene space" instead of "ego-sphere" to represent a two dimensional surface which contains the information perceived by the robot with respect to the robot's head-centric coordinate system. The purpose of the model BIB020 , however, is to track a set of predefined object in the surrounding. To achieve this goal it uses only the color information of the target object and performs object-based analysis to implement the IOR (Issue 1.2, 2 ). Although the model might have the potential to be extended for complex attention scenario, the current implementation in BIB020 is dealing with only few simple cases. The models in BIB015 and BIB016 use scaffolding where the human operator heavily guides the robot to teach what to focus on through speech command and hand-gesture. This solves the problem of prior training (Issue 5) and optimal learning strategy (Issue 3) with the price of having a dedicated human operator throughout the attention process. Unfortunately, having such a dedicated human operator severs the generality problem (Issue 4). A reduced amount of human-dependency for learning of attention is observed in the multimodal overt attention model described in BIB021 . The model BIB021 proposes the idea of an attention map, similar to "probabilistic occupancy grid" widely used in robotic mapping  , to encode the saliency of the robot's surrounding. The attention map can be modulated by the task-demand conveyed to the robot through speech command. The model, however, requires significant amount of prior training and manual work to create an useful attention map for any specific robotic application. For quick reference, Table I shows a comparative analysis of the overt attention models discussed in this section.