Visual Attention for Robotic Cognition: A Survey <s> II. EVOLUTION OF RESEARCH ON COMPUTATIONAL MODELING <s> A new hypothesis about the role of focused attention is proposed. The feature-integration theory of attention suggests that attention must be directed serially to each stimulus in a display whenever conjunctions of more than one separable feature are needed to characterize or distinguish the possible objects presented. A number of predictions were tested in a variety of paradigms including visual search, texture segregation, identification and localization, and using both separable dimensions (shape and color) and local elements or parts of figures (lines, curves, etc. in letters) as the features to be integrated into complex wholes. The results were in general consistent with the hypothesis. They offer a new set of criteria for distinguishing separable from integral features and a new rationale for predicting which tasks will show attention limits and which will not. <s> BIB001 </s> Visual Attention for Robotic Cognition: A Survey <s> II. EVOLUTION OF RESEARCH ON COMPUTATIONAL MODELING <s> The processing time for quantifying numerosity of two-dimensional dot patterns was investigated as a function of both number of dots and relative proximity between dots. A cluster algorithm (CODE) was first developed as a formal model of how human subjects organize neighboring dots into groups. CODE-based predictions of grouping effects on number processing latencies were then tested with patterns consisting of n dots (range n = 13–23). The results largely confirmed CODE-based predictions and thereby indicated that large collections of dots are preferably counted by groups. Small (n ≤ 5) groups are subitized and their partial results are summed to a running total. Based on criteria other than dot proximity, large (n > 5), proximity-based groups are subdivided into smaller groups of two or three dots, which are again subitized. <s> BIB002 </s> Visual Attention for Robotic Cognition: A Survey <s> II. EVOLUTION OF RESEARCH ON COMPUTATIONAL MODELING <s> A number of psychophysical studies concerning the detection, localization and recognition of objects in the visual field have suggested a two-stage theory of human visual perception. The first stage is the “preattentive” mode, in which simple features are processed rapidly and in parallel over the entire visual field. In the second, “attentive” mode, a specialized processing focus, usually called the focus of attention, is directed to particular locations in the visual field. The analysis of complex forms and the recognition of objects are associated with this second stage.1 The computational justification for such a hypothesis comes from the realization that while it is possible to imagine specific algorithms performing tasks such as shape analysis and recognition at specific locations, it is difficult to imagine these algorithms operating in parallel over the whole visual scene, since such an approach will quickly lead to a combinatorial explosion in terms of required computational resources.2 This is essentially the major critique of Minsky and Papert to a universal application of perceptrons in visual perception.3 Taken together, these empirical and theoretical studies suggest that beyond a certain preprocessing stage, the analysis of visual information proceeds in a sequence of operations, each one applied to a selected location (or locations). <s> BIB003 </s> Visual Attention for Robotic Cognition: A Survey <s> II. EVOLUTION OF RESEARCH ON COMPUTATIONAL MODELING <s> Subjects searched sets of items for targets defined by conjunctions of color and form, color and orientation, or color and size. Set size was varied and reaction times (RT) were measured. For many unpracticed subjects, the slopes of the resulting RT X Set Size functions are too shallow to be consistent with Treisman's feature integration model, which proposes serial, self-terminating search for conjunctions. Searches for triple conjunctions (Color X Size X Form) are easier than searches for standard conjunctions and can be independent of set size. A guided search model similar to Hoffman's (1979) two-stage model can account for these data. In the model, parallel processes use information about simple features to guide attention in the search for conjunctions. Triple conjunctions are found more efficiently than standard conjunctions because three parallel processes can guide attention more effectively than two. <s> BIB004 </s> Visual Attention for Robotic Cognition: A Survey <s> II. EVOLUTION OF RESEARCH ON COMPUTATIONAL MODELING <s> An important component of routine visual behavior is the ability to find one item in a visual world filled with other, distracting items. This ability to performvisual search has been the subject of a large body of research in the past 15 years. This paper reviews the visual search literature and presents a model of human search behavior. Built upon the work of Neisser, Treisman, Julesz, and others, the model distinguishes between a preattentive, massively parallel stage that processes information about basic visual features (color, motion, various depth cues, etc.) across large portions of the visual field and a subsequent limited-capacity stage that performs other, more complex operations (e.g., face recognition, reading, object identification) over a limited portion of the visual field. The spatial deployment of the limited-capacity process is under attentional control. The heart of the guided search model is the idea that attentional deployment of limited resources isguided by the output of the earlier parallel processes. Guided Search 2.0 (GS2) is a revision of the model in which virtually all aspects of the model have been made more explicit and/or revised in light of new data. The paper is organized into four parts: Part 1 presents the model and the details of its computer simulation. Part 2 reviews the visual search literature on preattentive processing of basic features and shows how the GS2 simulation reproduces those results. Part 3 reviews the literature on the attentional deployment of limited-capacity processes in conjunction and serial searches and shows how the simulation handles those conditions. Finally, Part 4 deals with shortcomings of the model and unresolved issues. <s> BIB005 </s> Visual Attention for Robotic Cognition: A Survey <s> II. EVOLUTION OF RESEARCH ON COMPUTATIONAL MODELING <s> The two basic phenomena that define the problem of visual attention can be illustrated in a simple example. Consider the arrays shown in each panel of Figure 1. In a typical experiment, before the arrays were presented, subjects would be asked to report letters appearing in one color (targets, here black letters), and to disregard letters in the other color (nontargets, here white letters). The array would then be briefly flashed, and the subjects, without any opportunity for eye movements, would give their report. The display mimics our. usual cluttered visual environment: It contains one or more objects that are relevant to current behavior, along with others that are irrelevant. The first basic phenomenon is limited capacity for processing information. At any given time, only a small amount of the information available on the retina can be processed and used in the control of behavior. Subjectively, giving attention to any one target leaves less available for others. In Figure 1, the probability of reporting the target letter N is much lower with two accompa­ nying targets (Figure la) than with none (Figure Ib). The second basic phenomenon is selectivity-the ability to filter out un­ wanted information. Subjectively, one is aware of attended stimuli and largely unaware of unattended ones. Correspondingly, accuracy in identifying an attended stimulus may be independent of the number of nontargets in a display (Figure la vs Ie) (see Bundesen 1990, Duncan 1980). <s> BIB006 </s> Visual Attention for Robotic Cognition: A Survey <s> II. EVOLUTION OF RESEARCH ON COMPUTATIONAL MODELING <s> A model for aspects of visual attention based on the concept of selective tuning is presented. It provides for a solution to the problems of selection in an image, information routing through the visual processing hierarchy and task-specific attentional bias. The central thesis is that attention acts to optimize the search procedure inherent in a solution to vision. It does so by selectively tuning the visual processing network which is accomplished by a top-down hierarchy of winner-take-all processes embedded within the visual processing pyramid. Comparisons to other major computational models of attention and to the relevant neurobiology are included in detail throughout the paper. The model has been implemented; several examples of its performance are shown. This model is a hypothesis for primate visual attention, but it also outperforms existing computational solutions for attention in machine vision and is highly appropriate to solving the problem in a robot vision system. <s> BIB007 </s> Visual Attention for Robotic Cognition: A Survey <s> II. EVOLUTION OF RESEARCH ON COMPUTATIONAL MODELING <s> This article presents a theory that integrates space-based and object-based approaches to visual attention. The theory puts together M.P. van Oeffelen and P.G. Vos's (1982, 1983) COntour DEtector (CODE) theory of perceptual grouping by proximity with C. Bundesen's (1990) theory of visual attention (TVA). CODE provides input to TVA, accounting for spatially based between-object selection, and TVA converts the input to output, accounting for feature- and category-based within-object selection. CODE clusters nearby items into perceptual groups that are both perceptual objects and regions of space, thereby integrating object-based and space-based approaches to attention. The combined theory provides a quantitative account of the effects of grouping by proximity and distance between items on reaction time and accuracy data in 7 empirical situations that shaped the current literature on visual spatial attention. <s> BIB008 </s> Visual Attention for Robotic Cognition: A Survey <s> II. EVOLUTION OF RESEARCH ON COMPUTATIONAL MODELING <s> We present an active object recognition strategy which combines the use of an attention mechanism for focusing the search for a 3D object in a 2D image, with a viewpoint control strategy for disambiguating recovered object features. The attention mechanism consists of a probabilistic search through a hierarchy of predicted feature observations, taking objects into a set of regions classified according to the shapes of their bounding contours. We motivate the use of image regions as a focus-feature and compare their uncertainty in inferring objects with the uncertainty of more commonly used features such as lines or corners. If the features recovered during the attention phase do not provide a unique mapping to the 3D object being searched, the probabilistic feature hierarchy can be used to guide the camera to a new viewpoint from where the object can be disambiguated. The power of the underlying representation is its ability to unify these object recognition behaviors within a single framework. We present the approach in detail and evaluate its performance in the context of a project providing robotic aids for the disabled. <s> BIB009 </s> Visual Attention for Robotic Cognition: A Survey <s> II. EVOLUTION OF RESEARCH ON COMPUTATIONAL MODELING <s> Chelazzi, Leonardo, John Duncan, Earl K. Miller, and Robert Desimone. Responses of neurons in inferior temporal cortex during memory-guided visual search. J. Neurophysiol. 80: 2918–2940, 1998. A typical scene will contain many different objects, few of which are relevant to behavior at any given moment. Thus attentional mechanisms are needed to select relevant objects for visual processing and control over behavior. We examined this role of attention in the inferior temporal cortex of macaque monkeys, using a visual search paradigm. While the monkey maintained fixation, a cue stimulus was presented at the center of gaze, followed by a blank delay period. After the delay, an array of two to five choice stimuli was presented extrafoveally, and the monkey was rewarded for detecting a target stimulus matching the cue. The behavioral response was a saccadic eye movement to the target in one version of the task and a lever release in another. The array was composed of one “good” stimulus (effective in driving the cell when presented alone) and one or more “poor” stimuli (ineffective in driving the cell when presented alone). Most cells showed higher delay activity after a good stimulus used as the cue than after a poor stimulus. The baseline activity of cells was also higher preceding a good cue, if the animal expected it to occur. This activity may depend on a top-down bias in favor of cells coding the relevant stimulus. When the choice array was presented, most cells showed suppressive interactions between the stimuli as well as strong attention effects. When the choice array was presented in the contralateral visual field, most cells initially responded the same, regardless of which stimulus was the target. However, within 150–200 ms of array onset, responses were determined by the target stimulus. If the target was the good stimulus, the response to the array became equal to the response to the good stimulus presented alone. If the target was a poor stimulus, the response approached the response to that stimulus presented alone. Thus the influence of the nontarget stimulus was eliminated. These effects occurred well in advance of the behavioral response. When the array was positioned with stimuli on opposite sides of the vertical meridian, the contralateral stimulus appeared to dominate the response, and this dominant effect could not be overcome by attention. Overall, the results support a “biased competition” model of attention, according to which 1 ) objects in the visual field compete for representation in the cortex, and 2 ) this competition is biased in favor of the behaviorally relevant object by virtue of “top-down” feedback from structures involved in working memory. <s> BIB010 </s> Visual Attention for Robotic Cognition: A Survey <s> II. EVOLUTION OF RESEARCH ON COMPUTATIONAL MODELING <s> A visual attention system, inspired by the behavior and the neuronal architecture of the early primate visual system, is presented. Multiscale image features are combined into a single topographical saliency map. A dynamical neural network then selects attended locations in order of decreasing saliency. The system breaks down the complex problem of scene understanding by rapidly selecting, in a computationally efficient manner, conspicuous locations to be analyzed in detail. Index Terms—Visual attention, scene analysis, feature extraction, target detection, visual search. ———————— F ———————— <s> BIB011 </s> Visual Attention for Robotic Cognition: A Survey <s> II. EVOLUTION OF RESEARCH ON COMPUTATIONAL MODELING <s> Most models of visual search, whether involving overt eye movements or covert shifts of attention, are based on the concept of a saliency map, that is, an explicit two-dimensional map that encodes the saliency or conspicuity of objects in the visual environment. Competition among neurons in this map gives rise to a single winning location that corresponds to the next attended target. Inhibiting this location automatically allows the system to attend to the next most salient location. We describe a detailed computer implementation of such a scheme, focusing on the problem of combining information across modalities, here orientation, intensity and color information, in a purely stimulus-driven manner. The model is applied to common psychophysical stimuli as well as to a very demanding visual search task. Its successful performance is used to address the extent to which the primate visual system carries out visual search via one or more such saliency maps and how this can be tested. <s> BIB012 </s> Visual Attention for Robotic Cognition: A Survey <s> II. EVOLUTION OF RESEARCH ON COMPUTATIONAL MODELING <s> Five important trends have emerged from recent work on computational models of focal visual attention that emphasize the bottom-up, image-based control of attentional deployment. First, the perceptual saliency of stimuli critically depends on the surrounding context. Second, a unique 'saliency map' that topographically encodes for stimulus conspicuity over the visual scene has proved to be an efficient and plausible bottom-up control strategy. Third, inhibition of return, the process by which the currently attended location is prevented from being attended again, is a crucial element of attentional deployment. Fourth, attention and eye movements tightly interplay, posing computational challenges with respect to the coordinate system used to control attention. And last, scene understanding and object recognition strongly constrain the selection of attended locations. Insights from these five key areas provide a framework for a computational and neurobiological understanding of visual attention. <s> BIB013 </s> Visual Attention for Robotic Cognition: A Survey <s> II. EVOLUTION OF RESEARCH ON COMPUTATIONAL MODELING <s> Visual feature integration theory was one of the most influential theories of visual information processing in the last quarter of the 20th century. This article provides an exposition of the theory and a review of the associated data. In the past much emphasis has been placed on how the theory explains performance in various visual search tasks. The relevant literature is discussed and alternative accounts are described. Amendments to the theory are also set out. Many other issues concerning internal processes and representations implicated by the theory are reviewed. The article closes with a synopsis of what has been learned from consideration of the theory, and it is concluded that some of the issues may remain intractable unless appropriate neuroscientific investigations are carried out. <s> BIB014 </s> Visual Attention for Robotic Cognition: A Survey <s> II. EVOLUTION OF RESEARCH ON COMPUTATIONAL MODELING <s> We review some of the major computational models of visual selective attention in terms of how they apply to psychological data and to theoretical concepts derived from experiments concerned with stimulus filtering, visual search and spatial cueing. We also review attempts to model neuropsychological disorders of visual object recognition and attention, including visual agnosia and neglect. We highlight how computational models can unite several dichotomies in psychological models: space-based versus object-based selection; early versus late selection; representational versus attentional accounts for neurological deficits. A comparison of the models exposes weaknesses and advantages in different accounts, but it also highlights that most models suggest that competitive interactions in visual selection are the basis for attentional effects in human data. There is also agreement across models on explaining neurological disorders as due to imbalances in competition following damage to certain areas of the brain. We propose that computational models clarify theoretical accounts of visual selective attention and integrate concepts across areas. The models provide a useful contribution to psychological research. <s> BIB015 </s> Visual Attention for Robotic Cognition: A Survey <s> II. EVOLUTION OF RESEARCH ON COMPUTATIONAL MODELING <s> Abstract Colour is more important than form in driving the scan path during active visual feature conjunction search. A model shows how feature-based attention in V4 provides information to LIP allowing it to represent behaviourally relevant locations and attract attention to target coloured locations. Attentional effects in V4 develop such that an initial spatial attention focus, scaled according to stimulus density, becomes object-based later in the response. Thus, this is the first model to replicate, with temporal precision, different attentional effects observed in single-cell studies of V4 and IT, in addition to the systems level scan path behaviour. <s> BIB016 </s> Visual Attention for Robotic Cognition: A Survey <s> II. EVOLUTION OF RESEARCH ON COMPUTATIONAL MODELING <s> Visual motion analysis has focused on decomposing image sequences into their component features. There has been little success at re-combining those features into moving objects. Here, a novel model of attentive visual motion processing is presented that addresses both decomposition of the signal into constituent features as well as the re-combination, or binding, of those features into wholes. A new feed-forward motion-processing pyramid is presented motivated by the neurobiology of primate motion processes. On this structure the Selective Tuning (ST) model for visual attention is demonstrated. There are three main contributions: (1) a new feed-forward motion processing hierarchy, the first to include a multi-level decomposition with local spatial derivatives of velocity; (2) examples of how ST operates on this hierarchy to attend to motion and to localize and label motion patterns; and (3) a new solution to the feature binding problem sufficient for grouping motion features into coherent object motion. Binding is accomplished using a top-down selection mechanism that does not depend on a single location-based saliency representation. resentation. <s> BIB017 </s> Visual Attention for Robotic Cognition: A Survey <s> II. EVOLUTION OF RESEARCH ON COMPUTATIONAL MODELING <s> The responses of neurons in cortical areas V2 and V4 can be significantly modulated by attention to particular locations within an input image. We show that such effects emerge naturally when perception is viewed as a probabilistic inference process governed by Bayesian principles and implemented in hierarchical cortical networks. The proposed model can explain a rich variety of attention-related responses in cortical area V4 including multiplicative modulation of tuning curves, restoration of neural responses in the presence of distracting stimuli, and influence of attention on neighboring unattended locations. Our results suggest a new interpretation of attention as a cortical mechanism for reducing perceptual uncertainty by combining top-down task-relevant information with bottom-up sensory inputs in a probabilistic manner. <s> BIB018
OF VISUAL ATTENTION The visual attention of the primates is not yet a fully understood mechanism and therefore, associated theories and hypothesis are being updated continuously. There exist a number of hypothesis in the literature of cognitive nueroscience and developmental psychology regarding the developmental process of visual attention in the primates BIB006 -  . The researchers, however, are not unified yet about their opinion on the exact mechanism that governs the development of attention in the primates. In spite of this lack of complete understanding, the research on 1943 -0604/$26.00 © 2010 IEEE visual attention in the past few decades has reached the status where we can comfortably derive a functional framework of the attention related activities. This development inspired the researchers in biology, psychology, computational neuroscience, computer vision, and robotics to develop synthetic models of visual attention which have potential applications in their respective fields. This section discusses about some of the most influential visual attention models in psychology, computational neuroscience, and computer vision. The pioneering work on visual attention, the feature integration theory BIB001 , was proposed in psychology. The generic purpose of the attention models proposed in psychology is to explain human perception and cognition  , BIB001 - BIB008 based on behavioral data (of the primates). The feature integration theory advocates the idea that perception of features comes prior to the perception of objects. The features in the visual field register themselves to our visual system and then are processed with the help of visual attention to form the concept of an object. Different visual features, such as color, orientation, spatial frequency, brightness, and direction of movement register their saliency in separate feature-maps. The feature maps also retain the physical location of different features in order to ensure the perfect synthesis of features for each object. In case of computational implementation the feature-maps are summed up to create a master-map of saliency of different features. The master-map is used to direct attention toward different locations in an image in the order of decreasing saliency. The feature integration theory went through several facets of development to accommodate the new findings on attention obtained from psychophysical experiments. A comprehensive survey on this popular theory is available in BIB014 . One major drawback of the early feature integration theory is that it considers only the effect of visual strength of different features (commonly known as bottom-up bias) in attentional selection. The guided search model BIB004 of visual attention overcomes this limitation by invoking the effect of top-down selection. The guided search is mostly focused on modeling the attention behavior related to visual search. Similar to the feature integration theory, gradual upgrading is observed in the guided search model BIB005 -  to accommodate the new findings of the primates visual search behavior. Another influential model of visual attention in psychology is the CODE theory of attention BIB008 . CODE theory is basically an integration of the theory of visual attention  with the theory of perceptual grouping by proximity BIB002 . A major difference of the CODE theory from the feature integration theory and the guided search model is it considers both space and object as the elemental unit for attentional selection, whereas the latter two only deal with space-based attention. There are many other models of visual attention available in psychology. Please see BIB015 for a comprehensive survey on the psychophysical models of visual attention. The synthetic models of visual attention developed in neurobiology and computational neuroscience are based on the findings of the primates visual attention obtained through lesion study and brain imaging techniques [e.g., functional magnetic resonance imaging (fMRI), positron emission tomography (PET)] [4]- BIB018 . The goal of these models is to faithfully reproduce the results obtained from the study of different attentional networks in the primates brain. Given the fact that study on the brain of live subjects is a very delicate matter and is subjected to different ethical bindings, accurate models in computational neuroscience play a critical role in understanding the operation of different brain networks. A survey on the neurobiological models of visual attention is available in  . One of the most popular theories of attention in neuroscience is the biased competition hypothesis (BC) (also known as integrated competition hypothesis BIB006 , BIB010 -  ). The BC hypothesis advocates the idea of a mutually suppressive interaction among visual neurons when excited by different visual stimuli. The attention mechanism of the primates biases this competition in favor of a certain specific stimulus through feedback bias mechanism (also known as top-down bias). The postulates of BC gained widespread popularity due to strong experimental evidences. A number of computational models has been proposed in computational neuroscience based on the postulates of BC  -  ,  , BIB016 . The BC-based models of visual attention invoke many new findings of attention, e.g., combination of objectand space-based analysis for attentional selection, integration of top-down and bottom-up bias, and integration of covert and overt shift of attention. The rise of the computer vision models of attention BIB013 -  ,  occurred almost in parallel with the psychophysical models. The model proposed in BIB003 is the first computer vision model of visual attention and follows the basic postulates of the feature integration theory. This model BIB003 coins the term "saliency map" to define a two-dimensional representation of scene saliency. The model BIB003 also introduces the concept of a winner-take-all (WTA) network to identify the focus of attention in a saliency map, as well as to implement the property of inhibition of return (IOR)  . Fig. 1 shows a schematic diagram of the model. The term "saliency map," as well as its underlying concept have been used extensively in the visual attention literature. Different variants of the WTA network proposed in BIB003 have been used in several models of visual attention. Probably the most influential computer vision model of visual attention is the neuromorphic vision toolkit (NVT) BIB011 which has been extensively used in many other models of visual attention in the computer vision and robotics. A number of concepts in the NVT are heavily inspired by the attention model in BIB003 , e.g., feature map, saliency map, and WTA network-based implementation of IOR. The NVT, however, Fig. 2 . Architecture of NVT BIB011 . Multiscale analysis of an input image is performed to evaluate the conspicuties of three image features: color, intensity, and orientation. Individual feature maps are combined to create a saliency map onto which a WTA network operates to identify the next focus of attention. proposes a computationally elegant process for calculation of the saliency map. Here, a multiscale analysis of the input images are performed for calculation of individual feature maps which are combined together to create a centralized saliency map. Fig. 2 shows the basic architecture of NVT. The early version of NVT BIB011 performed only bottom-up analysis of attention but a later modification reported in  invokes the effect of top-down selection during visual search. Some of the shortcomings of NVT have been alleviated in the NVT-based model visual object detection with a computational attention system (VOCUS)  . Similar to NVT, VOCUS also relies on the attention model in BIB003 regarding a number of core concepts of visual attention, e.g., saliency map, WTA-based selection of focus, and implementation of IOR. The calculation of saliency map in VOCUS is similar to that of NVT. VOCUS, however, performs a number of improvements over NVT (with respect to implementation and theoretical analysis) which results in better accuracy in identifying the focus of attention in a given image. Fig. 3 shows the basic structure of VOCUS  . There is a group of computer vision models which uses connectionist approach for visual attention modeling. Among these connectionist models the most famous is the selective tuning model BIB007 . The selective tuning model analyzes four kinds of visual features to identify the focus of attention in an input image: luminance, orientation, color, and motion. The model performs a pyramid style processing of information where the stimuli of interest are located at the top and control an inhibitory beam. This inhibitory beam can inhibit or pass a zone for further processing. The top-down influence is modeled through manipulating the inhibitory beam. A unique characteristics of this model is, in spite of being a technical model of attention, the selective tuning model BIB007 and all of its variants BIB009 , BIB017 are tightly coupled with biological principles.  . The major differences with NVT BIB012 lie in a number of sectors in implementation, e.g., the center-surround mechanism, accross-scale addition, total number of image pyramids used for each feature, the learning of the top-down weight matrix, and addition of the top-down and bottom-up saliency map. There are several other computer vision models of visual attention, but the models discussed in this section are generally considered as the most influential models. This is mostly because the majority of the other existing models of visual attention are, to some extent, derived from them. The following section will shed light on the general characteristics of the computer vision models of visual attention which made them very popular over the last decade and, at the same time, triggered the evolution of a different group of visual attention models for the robot.