Visual Attention for Robotic Cognition: A Survey <s> A. General Characteristics of the Computer Vision Model of Visual Attention <s> A number of psychophysical studies concerning the detection, localization and recognition of objects in the visual field have suggested a two-stage theory of human visual perception. The first stage is the “preattentive” mode, in which simple features are processed rapidly and in parallel over the entire visual field. In the second, “attentive” mode, a specialized processing focus, usually called the focus of attention, is directed to particular locations in the visual field. The analysis of complex forms and the recognition of objects are associated with this second stage.1 The computational justification for such a hypothesis comes from the realization that while it is possible to imagine specific algorithms performing tasks such as shape analysis and recognition at specific locations, it is difficult to imagine these algorithms operating in parallel over the whole visual scene, since such an approach will quickly lead to a combinatorial explosion in terms of required computational resources.2 This is essentially the major critique of Minsky and Papert to a universal application of perceptrons in visual perception.3 Taken together, these empirical and theoretical studies suggest that beyond a certain preprocessing stage, the analysis of visual information proceeds in a sequence of operations, each one applied to a selected location (or locations). <s> BIB001 </s> Visual Attention for Robotic Cognition: A Survey <s> A. General Characteristics of the Computer Vision Model of Visual Attention <s> A model for aspects of visual attention based on the concept of selective tuning is presented. It provides for a solution to the problems of selection in an image, information routing through the visual processing hierarchy and task-specific attentional bias. The central thesis is that attention acts to optimize the search procedure inherent in a solution to vision. It does so by selectively tuning the visual processing network which is accomplished by a top-down hierarchy of winner-take-all processes embedded within the visual processing pyramid. Comparisons to other major computational models of attention and to the relevant neurobiology are included in detail throughout the paper. The model has been implemented; several examples of its performance are shown. This model is a hypothesis for primate visual attention, but it also outperforms existing computational solutions for attention in machine vision and is highly appropriate to solving the problem in a robot vision system. <s> BIB002 </s> Visual Attention for Robotic Cognition: A Survey <s> A. General Characteristics of the Computer Vision Model of Visual Attention <s> A visual attention system, inspired by the behavior and the neuronal architecture of the early primate visual system, is presented. Multiscale image features are combined into a single topographical saliency map. A dynamical neural network then selects attended locations in order of decreasing saliency. The system breaks down the complex problem of scene understanding by rapidly selecting, in a computationally efficient manner, conspicuous locations to be analyzed in detail. Index Terms—Visual attention, scene analysis, feature extraction, target detection, visual search. ———————— F ———————— <s> BIB003 </s> Visual Attention for Robotic Cognition: A Survey <s> A. General Characteristics of the Computer Vision Model of Visual Attention <s> The model presented here is an attempt to explain the results from a number of different studies in visual attention, including parallel feature searches and serial conjunction searches, variations in search slope with variations in feature contrast and individual subject differences, attentional gradients triggered by cuing, feature-driven spatial selection, split attention, inhibition of distractor locations, and flanking inhibition. The model is implemented in a neural network consisting of a hierarchy of spatial maps. Attentional gates control the flow of information from each level of the hierarchy to the next. The gates are jointly controlled by a Bottom-Up System favoring locations with unique features and a Top-Down System favoring locations with features designated as target features. Because the gating of each location depends on the features present there, the model is called FeatureGate. <s> BIB004 </s> Visual Attention for Robotic Cognition: A Survey <s> A. General Characteristics of the Computer Vision Model of Visual Attention <s> Most models of visual search, whether involving overt eye movements or covert shifts of attention, are based on the concept of a saliency map, that is, an explicit two-dimensional map that encodes the saliency or conspicuity of objects in the visual environment. Competition among neurons in this map gives rise to a single winning location that corresponds to the next attended target. Inhibiting this location automatically allows the system to attend to the next most salient location. We describe a detailed computer implementation of such a scheme, focusing on the problem of combining information across modalities, here orientation, intensity and color information, in a purely stimulus-driven manner. The model is applied to common psychophysical stimuli as well as to a very demanding visual search task. Its successful performance is used to address the extent to which the primate visual system carries out visual search via one or more such saliency maps and how this can be tested. <s> BIB005 </s> Visual Attention for Robotic Cognition: A Survey <s> A. General Characteristics of the Computer Vision Model of Visual Attention <s> Five important trends have emerged from recent work on computational models of focal visual attention that emphasize the bottom-up, image-based control of attentional deployment. First, the perceptual saliency of stimuli critically depends on the surrounding context. Second, a unique 'saliency map' that topographically encodes for stimulus conspicuity over the visual scene has proved to be an efficient and plausible bottom-up control strategy. Third, inhibition of return, the process by which the currently attended location is prevented from being attended again, is a crucial element of attentional deployment. Fourth, attention and eye movements tightly interplay, posing computational challenges with respect to the coordinate system used to control attention. And last, scene understanding and object recognition strongly constrain the selection of attended locations. Insights from these five key areas provide a framework for a computational and neurobiological understanding of visual attention. <s> BIB006 </s> Visual Attention for Robotic Cognition: A Survey <s> A. General Characteristics of the Computer Vision Model of Visual Attention <s> In this paper, a novel model of object-based visual attention extending Duncan's Integrated Competition Hypothesis [Phil. Trans. R. Soc. London B 353 (1998) 1307-1317] is presented. In contrast to the attention mechanisms used in most previous machine vision systems which drive attention based on the spatial location hypothesis, the mechanisms which direct visual attention in our system are object-driven as well as feature-driven. The competition to gain visual attention occurs not only within an object but also between objects. For this purpose, two new mechanisms in the proposed model are described and analyzed in detail. The first mechanism computes the visual salience of objects and groupings; the second one implements the hierarchical selectivity of attentional shifts. The results of the new approach on synthetic and natural images are reported. <s> BIB007 </s> Visual Attention for Robotic Cognition: A Survey <s> A. General Characteristics of the Computer Vision Model of Visual Attention <s> Visual motion analysis has focused on decomposing image sequences into their component features. There has been little success at re-combining those features into moving objects. Here, a novel model of attentive visual motion processing is presented that addresses both decomposition of the signal into constituent features as well as the re-combination, or binding, of those features into wholes. A new feed-forward motion-processing pyramid is presented motivated by the neurobiology of primate motion processes. On this structure the Selective Tuning (ST) model for visual attention is demonstrated. There are three main contributions: (1) a new feed-forward motion processing hierarchy, the first to include a multi-level decomposition with local spatial derivatives of velocity; (2) examples of how ST operates on this hierarchy to attend to motion and to localize and label motion patterns; and (3) a new solution to the feature binding problem sufficient for grouping motion features into coherent object motion. Binding is accomplished using a top-down selection mechanism that does not depend on a single location-based saliency representation. resentation. <s> BIB008
The computer vision model of visual attention share some common operating principles. Fig. 4 shows the architecture generally followed by the existing computer vision models of visual attention. The key differences among different models occur in BIB005 , (b) the saliency map calculated using the method described in  , and (c) five focus points in the image marked in the order of decreasing saliency. two sectors: 1) the methodology of implementing the overall architecture, e.g., connectionist approach BIB002 and BIB004 , filterbased approach BIB006 -  , BIB007 ,  , and  ; and 2) the mechanism of constructing the saliency map. Despite these two sectors of mismatch, the computer vision models of visual attention share a set of common characteristics. They are summarized below. 1) Saliency Operator: A unique, centralized "saliency map" plays a key role to guide attention toward different regions of an input image in almost all of the existing computer vision models of visual attention BIB003 ,  , BIB004 -  ,  . A saliency map, in simple words, is a two-dimensional image (of same size as the input image) in which the intensity value of a pixel represents the relative visual saliency of its corresponding pixel in the original input image. The higher the value is, the more salient the pixel is. A saliency map-based attention model generally reports the most salient pixel in the saliency map as the current focus of attention. In order to prevent the attention from revisiting the same location, the saliency of the current focus of attention is suppressed after being attended and thereby achieving the property of IOR  . Fig. 5 shows a typical saliency map corresponding to a natural image (obtained from the freely available standard image database provided in BIB005 ) along with the focuses of attention in the image in the order of decreasing saliency. 2) Covert Shift of Focus: Majority of the computer vision models of visual attention are designed based on the assumption that neither the eye nor the head moves to execute visual attention. The attention mechanism in the primates which obeys this assumption is called covert attention  . The absence of eye/head movement during the execution of visual attention has a number of consequences. • The retinal input remains unchanged throughout the attentional task. • The frame of reference remains unchanged in the subsequent directions of attention. This simplifies the implementation of the IOR. • The belief about the scene saliency remains unchanged causing no further requirement to recalculate it after each attentional shift. Most of the computer vision models of visual attention (e.g., BIB001 -  and  ) enjoy the simplicity of computation arising from the above three consequences of the covert nature of attentional shift. The covert shift of attention, however, makes it difficult to compare the performance of a model with the ground truth, e.g., with the attention behavior of the human. 3) Bottom-Up and Top-Down Analysis: The early computer vision models of attention (e.g., BIB001 , BIB003 , BIB002 ,  , and BIB005 ) mostly dealt with bottom-up (or stimulus driven) influence in attention selection. To be consistent with the biological findings, the recent models started to invoke the effect of top-down influence  , BIB004 ,  , BIB008 ,  . These latter models  , BIB004 ,  , BIB008 ,  , however, limit the influence of top-down information only to the case of visual search. Thus, the bottom-up cues guide the visual exploration (focusing on the most salient stimuli), while the top-down cues guide the visual search (an active scan of the visual field in search of a prespecified object or stimuli). In almost all of the existing models these two modes of attention (visual search and visual exploration) run in mutual exclusion of each other as shown in Fig. 4 . The desired mode of attention (search or exploration) is manually activated by the programmer depending on the task at hand. In some of the models, even the process of generating the saliency map for visual exploration considerably differs from that for visual search.