Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> Perceiving, for all its nicety of functioning in the dark room under strict instructions for accuracy, comprises a highly complex series of little understood psychological processes. For under all conditions, perceiving represents a resultant of two complex sets of specifications. One set describes the conditions of stimulation. This is done either in terms of physical measures such as wave length, or in terms of psychological norms such as in the description of a picture as that picture is seen by "normal" observers under optimal conditions and with a set for accuracy. This first set of specifications we are used to calling "stimulus" factors. <s> BIB001 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> A number of psychophysical studies concerning the detection, localization and recognition of objects in the visual field have suggested a two-stage theory of human visual perception. The first stage is the “preattentive” mode, in which simple features are processed rapidly and in parallel over the entire visual field. In the second, “attentive” mode, a specialized processing focus, usually called the focus of attention, is directed to particular locations in the visual field. The analysis of complex forms and the recognition of objects are associated with this second stage.1 The computational justification for such a hypothesis comes from the realization that while it is possible to imagine specific algorithms performing tasks such as shape analysis and recognition at specific locations, it is difficult to imagine these algorithms operating in parallel over the whole visual scene, since such an approach will quickly lead to a combinatorial explosion in terms of required computational resources.2 This is essentially the major critique of Minsky and Papert to a universal application of perceptrons in visual perception.3 Taken together, these empirical and theoretical studies suggest that beyond a certain preprocessing stage, the analysis of visual information proceeds in a sequence of operations, each one applied to a selected location (or locations). <s> BIB002 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> A visual attention system, inspired by the behavior and the neuronal architecture of the early primate visual system, is presented. Multiscale image features are combined into a single topographical saliency map. A dynamical neural network then selects attended locations in order of decreasing saliency. The system breaks down the complex problem of scene understanding by rapidly selecting, in a computationally efficient manner, conspicuous locations to be analyzed in detail. Index Terms—Visual attention, scene analysis, feature extraction, target detection, visual search. ———————— F ———————— <s> BIB003 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> The model presented here is an attempt to explain the results from a number of different studies in visual attention, including parallel feature searches and serial conjunction searches, variations in search slope with variations in feature contrast and individual subject differences, attentional gradients triggered by cuing, feature-driven spatial selection, split attention, inhibition of distractor locations, and flanking inhibition. The model is implemented in a neural network consisting of a hierarchy of spatial maps. Attentional gates control the flow of information from each level of the hierarchy to the next. The gates are jointly controlled by a Bottom-Up System favoring locations with unique features and a Top-Down System favoring locations with features designated as target features. Because the gating of each location depends on the features present there, the model is called FeatureGate. <s> BIB004 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> The goal of our research is to investigate the interplay between oculomotor control, visual processing, and limb control in humans and primates by exploring the computational issues of these processes with a biologically inspired artificial oculomotor system on an anthropomorphic robot. In this paper, we investigate the computational mechanisms for visual attention in such a system. Stimuli in the environment excite a dynamical neural network that implements a saliency map, i.e., a winner-take-all competition between stimuli while simultaneously smoothing out noise and suppressing irrelevant inputs. In real-time, this system computes new targets for the shift of gaze, executed by the head-eye system of the robot. The redundant degrees-of-freedom of the head-eye system are resolved through a learned inverse kinematics with optimization criterion. We also address important issues how to ensure that the coordinate system of the saliency map remains correct after movement of the robot. The presented attention system is built on principled modules and generally applicable for any sensory modality. <s> BIB005 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> This paper describes the implementation of the attentional system of a humanoid robot based completely on space variant vision (in particular logpolar). The aim is that of providing the robot with a suitable measure of position, speed and saliency of possibly interesting objects for saccading and tracking. The major advantage of log-polar based imaging is related to the reduced number of pixels while maintaining a large field of view. This arrangement is very well suited for motor control, where the high-resolution center (fovea) allows precise positioning and, at the same time, the coarse resolution periphery permits detection of potential targets. Algorithms for color processing, optic flow, and disparity computation were developed within this architecture. The attentional modules are intended as the first layer of a more complicated system, which shall include learning of object recognition, trajectory tracking, and naive physics understanding during the natural interaction of the robot with the environment. <s> BIB006 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> Social robots are receiving much interest in the robotics community. The most important goal for such robots lies in their interaction capabilities. An attention system is crucial, both as a filter to center the robot’s perceptual resources and as a mean of letting the observer know that the robot has intentionality. In this paper a simple but flexible and functional attentional model is described. The model, which has been implemented in an interactive robot currently under development, fuses both visual and auditive information extracted from the robot’s environment, and can incorporate knowledge-based influences on attention. <s> BIB007 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> A human-like intelligent robot is expected to have the capability to develop its cognitive functions through experience without a priori knowledge or explicit teaching. In addition, the realization of this kind of robot leads us to understand the developmental mechanisms of human beings. This paper proposes a bootstrap learning model by which a robot acquires the ability of joint attention without a caregiver's evaluation or a controlled environment based on the robot's embedded mechanisms: visual attention and learning with self-evaluation. Through learning based on the proposed model, the robot finds a correlation in sensorimotor coordination when joint attention succeeds and consequently acquires the ability of joint attention by accumulating the appropriate correlation and losing the uncorrelated coordination as statistical outliers. The experimental results show the validity of the proposed model. <s> BIB008 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> Our group has constructed a series of brain-based devices (BBDs); i.e. physical devices with simulated nervous systems that guide behavior, to serve as a heuristic for understanding brain function. Unlike conventional robots designed by engineering principles, BBDs are based on biological principles and alter their behavior to the environment through self-learning. The resulting systems autonomously generalize signals from the environment into perceptual categories and through adaptive behavior become increasingly successful in coping with the environment. Although the principal focus of developing BBDs has been to test theories of the nervous system, this approach may also provide a basis for robotic design and practical applications. <s> BIB009 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> This study presents experiments on the imitative interactions between a small humanoid robot and a user. A dynamic neural network model of a mirror system was implemented in a humanoid robot, based on the recurrent neural network model with parametric bias (RNNPB). The experiments showed that after the robot learns multiple cyclic movement patterns as embedded in the RNNPB, it can regenerate each pattern synchronously with the movements of a human who is demonstrating the corresponding movement pattern in front of the robot. Further, the robot exhibits diverse interactive responses when the user demonstrates novel cyclic movement patterns. Those responses were analyzed and categorized. We propose that the dynamics of coherence and incoherence between the robot’s and the user’s movements could enhance close interactions between them, and that they could also explain the essential psychological mechanism of joint attention. <s> BIB010 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> This paper studies a synthetic model for joint attentions andturn takingbyconductingexperimentsofrobot-userimitative interactions. A Sony humanoid robot learns multiple cyclic movement patterns as embedded in a neural network model proposed by us and each of memorized movement patterns can be retrieved and generated by means of entrainment by external inputs in terms of users’ movement patterns perceived. Our experiments on a simple imitation game showed that multiple movement patterns are generated as synchronized between the robot and users while the shared patterns shift spontaneously from one to another. Based on the experimental results, we show possible dynamical systems accounts for the underlying mechanisms for joint attentions and turn taking. <s> BIB011 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> Although biomimetic autonomous robotics relies on the massively parallel architecture of the brain, the key issue is to temporally organize behaviour. The distributed representation of the sensory information has to be coherently processed to generate relevant actions. In the visual domain, we propose here a model of visual exploration of a scene by the means of localized computations in neural populations whose architecture allows the emergence of a coherent behaviour of sequential scanning of salient stimuli. It has been implemented on a real robotic platform exploring a moving and noisy scene including several identical targets. <s> BIB012 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> One of the first steps of any visual system is that of locating suitable interest points, "salient regions", in the scene, to detect events, and eventually to direct gaze toward these locations. In the last few years, object-based visual attention models have received an increasing interest in the literature, the problem, in this case, being that of creating a model of "objecthood" that eventually guides a saliency mechanism. We propose here an object-based model of visual attention and show its instantiation on a humanoid robot. The robot employs action to learn and define its own concept of objecthood. <s> BIB013 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> Robot companions are intended for operation in private homes with naive users. For this purpose, they need to be endowed with natural interaction capabilities. Additionally, such robots will need to be taught unknown objects that are present in private homes. We present a multi-modal object attention system that is able to identify objects referenced by the user with gestures and verbal instructions. The proposed system can detect known and unknown objects and stores newly acquired object information in a scene model for later retrieval. This way, the growing knowledge base of the robot companion improves the interaction quality as the robot can more easily focus its attention on objects it has been taught previously. <s> BIB014 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> A major challenge for the realization of intelligent robots is to supply them with cognitive abilities in order to allow ordinary users to program them easily and intuitively. One approach to such programming is teaching work tasks by interactive demonstration. To make this effective and convenient for the user, the machine must be capable of establishing a common focus of attention and be able to use and integrate spoken instructions, visual perception, and non-verbal clues like gestural commands. We report progress in building a hybrid architecture that combines statistical methods, neural networks, and finite state machines into an integrated system for instructing grasping tasks by man-machine interaction. The system combines the GRAVIS-robot for visual attention and gestural instruction with an intelligent interface for speech recognition and linguistic interpretation, and a modality fusion module to allow multi-modal task-oriented man-machine communication with respect to dextrous robot manipulation of objects. <s> BIB015 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> This paper describes our efforts to develop a robot with a sense of self using a multiagent-based cognitive architecture and control with three distinctive memory systems, namely (1) spatio-temporal short-term memory, (2) procedural/declarative/episodic long-term memory and (3) a task-oriented adaptive working memory. Such a robot may be called a cognitive robot. Cognitive robots share a number of key features with conscious machines. We are exploring the interface between cognitive robots and machine consciousness through an internal model called the Self Agent. <s> BIB016 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> The Sensory Ego-Sphere (SES) is a short-term memory for a robot in the form of an egocentric, tessellated, spherical, sensory-motor map of the robot's locale. This paper reports on: (1) the mapping to the SES of images with much higher resolution than the SES itself, and (2) the processing of visual attention on the SES. Described is a procedure to store spatially-overlapping imagery in the SES database and to generate and update a spherical composite of its visual contents. The composite image serves both as a visual map of the locale and as a representation of the local contents of the underlying full-resolution imagery. Visual attention enables fast alignment of overlapping images without warping or position optimization, since an attentional point (AP) on the composite typically corresponds to one on each of the collocated regions in the images. Such alignment speeds analysis of the multiple images of the area. Compositing and attention were performed two ways and compared: (1) APs were computed directly on the composite and not on the full resolution images until the time of retrieval. (2) The attentional operator was applied to all incoming imagery. Then the locations and values of the APs on the composite were computed as a function of those points within the corresponding regions of the collocated images. It was found that although the second method was slower, it produced consistent and, thereby, more useful APs. <s> BIB017 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> We develop an architecture for reactive visual analysis of dynamic scenes. We specify a minimal set of system features based upon biological observations. We implement feature on a processing network based around an active stereo vision mechanism. Active rectification and mosaicing allows static stereo algorithms to operate on the active platform. Foveal zero disparity operations permit attended object extraction and ensures coordinated stereo fixation upon visual surfaces. Activedynamic inhibition of return, and task dependent biasing result in a flexible, preemptive and retrospective system that responds to unique visual stimuli and is capable of top-down modulation of attention towards regions and cues relevant to tasks. <s> BIB018 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> Joint attention is one of the most important cognitive functions for the emergence of communication not only between humans, but also between humans and robots. In previous work, we have demonstrated how a robot can acquire primary joint attention behavior (gaze following) without external evaluation. However, this method needs the human to tell the robot when to shift its gaze. This paper presents a method that does not need such a constraint by introducing an attention selector based on a measure consisting of saliencies of object features and motion cues. In order to realize natural interaction, a self-organizing map for real-time face pattern separation and contingency learning for gaze following without external evaluation are utilized. The attention selector controls the robot gaze to switch often from the human face to an object and vice versa, and pairs of a face pattern and a gaze motor command are input to the contingency learning. The motion cues are expected to reduce the number of incorrect t... <s> BIB019 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> Abstract A visual overt attention mechanism is presented in this paper. Our algorithm chooses the next fixation point for a robot mobile camera in order to track several objects around the robot simultaneously, even if the whole set of them can not be covered by the same camera image. Our approach is based on two related measurements, liveliness and saliency, that dynamically evolve depending on the image observations and the camera movements. The attention is shared among exploring the surroundings for salient features, reobserving the tracked objects and other task-dependent points to look at. A winner-takes-all competition provides a flexible time sharing behavior with natural appearances of new objects, disappearances and inhibition of return. It also accepts top down features to look for and different priorities for them. Several experiments have been carried out with a real Pioneer robot endowed with mobile camera and are also described. <s> BIB020 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> This work presents a multimodal bottom-up attention system for the humanoid robot iCub where the robot's decisions to move eyes and neck are based on visual and acoustic saliency maps. We introduce a modular and distributed software architecture which is capable of fusing visual and acoustic saliency maps into one egocentric frame of reference. This system endows the iCub with an emergent exploratory behavior reacting to combined visual and auditory saliency. The developed software modules provide a flexible foundation for the open iCub platform and for further experiments and developments, including higher levels of attention and representation of the peripersonal space. <s> BIB021 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> During the lifetime of a mobile robot, the number and complexity of the stimuli it receives may be quite high. Therefore, the construction of a detection system considering the whole sensorial space is usually not a viable proposition when aiming for real time operation. It becomes necessary to build some kind of sensorial hierarchy map to put some order into how detectors are applied. This is what is usually called an attentional system, and it provides a framework for applying detectors in a more efficient manner. In this paper, an architecture for developing attentional functions for robots that must operate in real time in dynamic environments is presented. This architecture is based on the concept of attentor and it allows for the real time adaptation to the environment and tasks to be performed in a natural manner. One of the main requirements imposed on the design of the architecture was the capability of handling different sensorial modalities and attentional streams in a transparent manner while, at the same time, being able to progressively create more complex attentional structures. The architecture is particularized for its implementation in a real robot. <s> BIB022 </s> Visual Attention for Robotic Cognition: A Survey <s> V. GENERAL DISCUSSION <s> The interaction between a robot and a human becomes meaningful when the robotic agent possesses some level of human-like cognition. This paper proposes an attention-based approach for multi-modal HRI. The core of the proposed approach is a bio-inspired artificial model of visual attention which enables a robot to focus on the visually salient and/or behaviorally relevant stimuli in the surrounding. The attention model provides the human partner with the opportunity to manipulate the attention behavior of the robot through natural speech command. Similarly the robot is able to manipulate the attention of the human partner using its actuators. Thus the bio-inspired visual attention mechanism along with the sensors and actuators enables the robot to establish joint attention with the human partner. Formation of this joint attention is the basis for further human-robot interaction. Experimental results validate different aspects of the proposed attention-based HRI framework. <s> BIB023
Robotic models of visual attention are generally inspired by the computer vision models. The rich literature of computer vision on the computational modeling of visual attention mostly focuses on developing covert model of visual attention. Although these models are remarkably successful in identifying the most salient stimuli or performing a visual search in a static image, their use for robotic visual attention is restricted by a number of real-world design issues. A number of these issues are involved with the fact that robotic visual attention are generally overt in nature as opposed to the covert notion of attention commonly followed by the computer vision models. The others are involved with the architecture of the computer vision models of visual attention. Because of these research issues the robotic models of visual attention are steadily drifting apart from the computer vision models with respect to architecture and implementation methodologies. The survey presented in this article shows that a very popular choice to tackle the problem arising from camera movements in overt attention (Issue 1) is the spatio-temporal transformation of the visual features. In case of such spatio-temporal transformation, the saliency map is projected to the new camera coordinates BIB005 . To solve the IOR issue either the locations of the previously attended objects are mapped to the new frame BIB018 , BIB012 , or an object-based IOR is implemented BIB013 , BIB006 . A major reason of popularity of the spatio-temporal transformation (to tackle the issues related to camera movements) is it keeps the process of constructing the saliency map same as the way it is generally constructed in the famous covert models, e.g., Koch's model BIB002 , NVT BIB003 , FeatureGate BIB004 , and VOCUS  . A parallel approach, however, started to gain even more popularity than the spatio-temporal transformation-based approach. This new approach tackles the whole problem of visual attention from a robot-centric perspective and advocates the idea of an ego-centric saliency map BIB022 , BIB020 ,  , BIB021 , BIB017 . Some efforts are also observed to develop such map using panoramic camera BIB007 . The reason of increasing popularity of the robot-centric approach is its natural ability to tackle the issues related to head-eye movements of the robot. The most popular choice to address Issue 3-Issue 5 discussed in Section III-B is the learning of visual attention. The learning is generally mediated by human interaction and through using multiple modalities. But the role of human in the learning process differs in different models, e.g., full human guidance as in the imitation learning BIB008 - BIB019 , BIB010 , BIB011 and scaffolding  , BIB014 , BIB015 ,  , BIB001 , and occasional guidance as in BIB023 . The survey presented in this article clearly shows that the research on robotic models of visual attention is far from perfection. There are only a handful of models which attempts to address the research issues involved with robotic attention in a generic manner. Aside from these research issues, a general constraint of the existing computational models of visual attention is they limit the top-down influence in attentional selection only to the case of visual search. In reality, the top-down influence plays a major role in the primates visual attention mechanism. In case of the primates the top-down influence in attentional selection, however, is the result of a complex interaction among knowledge, emotion, personality, and reasoning. Designing a visual attention model for robotic cognition with such kind of top-down influence will require a dynamic interaction with other components of artificial cognition, e.g., reasoning, planning, emotion, and knowledge-representation. The perfection in the modeling of robotic visual attention, therefore, is closely related to the perfection in the modeling of other cognitive functions. Few efforts are observed in the current literature on the modeling of value system, human-like reasoning and knowledge representation for robotic systems  - BIB016 . These efforts, however, are mostly discrete and do not investigate the effect of other cognitive abilities on the visual attention behavior. A major reason for this lack of investigation is that the underlying neural mechanism of many of the cognitive functions and their mutual effect on the cognitive development are still unknown to the researchers. We can, therefore, hope that further development on the modeling of robotic visual attention will go hand-to-hand with the improvement of our understanding about human cognition. A major consequence of having a full-fledged primates-like visual attention model is that it will advance the current efforts on employing robots for better understanding of the human attention behavior BIB009 . This will be a magnificent way to use technological advancement for the understanding of human abilities.