Humans and many other species sense visual information with varying spatial resolution across the visual field (foveated vision) and deploy eye movements to actively sample regions of interests in scenes. The advantage of such varying resolution architecture is a reduced computational, hence metabolic cost. But what are the performance costs of such processing strategy relative to a scheme that processes the visual field at high spatial resolution? Here we first focus on visual search and combine object detectors from computer vision with a recent model of peripheral pooling regions found at the V1 layer of the human visual system. We develop a foveated object detector that processes the entire scene with varying resolution, uses retino-specific object detection classifiers to guide eye movements, aligns its fovea with regions of interest in the input image and integrates observations across multiple fixations. We compared the foveated object detector against a non-foveated version of the same object detector which processes the entire image at homogeneous high spatial resolution. We evaluated the accuracy of the foveated and non-foveated object detectors identifying 20 different objects classes in scenes from a standard computer vision data set (the PASCAL VOC 2007 dataset). We show that the foveated object detector can approximate the performance of the object detector with homogeneous high spatial resolution processing while bringing significant computational cost savings. Additionally, we assessed the impact of foveation on the computation of bottom-up saliency. An implementation of a simple foveated bottom-up saliency model with eye movements showed agreement in the selection of top salient regions of scenes with those selected by a non-foveated high resolution saliency model. Together, our results might help explain the evolution of foveated visual systems with eye movements as a solution that preserves perceptual performance in visual search while resulting in computational and metabolic savings to the brain.
Many species from primates, birds and shrimps [1,2] have an area of their visual sensory system with heightened spatial fidelity and utilize eye and head movements to orient this area towards objects of interest in scenes. The pervasiveness of sensory systems with varying spatial resolution for species that heavily rely on vision to sense the world motivates the question about its advantages. The wide-accepted answer is that visual processing with varying spatial resolution reduces the brain’s computational cost. For example, for humans, the density of cones in the fovea is approximately 20 times larger than at 10 degrees into the periphery and 90 times at the far visual periphery [3]. The fovea occupies 0. 01% of the retina but utilizes approximately 8-10% of the neuronal machinery in primary visual cortex [4]. A high spatial resolution processing system across the entire visual field matching the fovea’s ratio of primary cortex (V1) neurons per mm of retina would lead to roughly a one thousand increase in the size of the primary visual cortex. A full high spatial resolution visual system would likely drastically increase the brain’s computational expenditures and thus the metabolic cost. The ability of organisms with a heightened area of spatial fidelity (i. e. , a fovea) to successfully support perceptual decision making relies critically on the guidance of saccadic eye movements to sample the visual world. Humans perform approximately three eye movements per second. The brain uses peripheral processing to extract critical information and guides the eyes across the scene. Eye movements can be directed to salient regions in a scene as potential locations of interest and for further processing [5–7]. During search, eye movements are also guided by information about the target including basic features including color, size, orientation and shape [8–11], probabilities of object occurrence, global scene statistics [12,13], object co-occurrence [14–16], and knowledge of the detectability of objects across the visual field [17] to successfully detect objects in cluttered scenes. The brain is also able to acquire information in the visual periphery to guide eye movements concurrent with analyses of information at the foveal region [18]. This foveated visual system with guided eye movements reduces the brain’s computational cost. What is not known are the decision accuracy costs of a foveated architecture relative to a non-foveated high spatial resolution system in ecologically relevant tasks. The current work aims at answering this question. There have been many proposals for computational models of search and artificial systems with foveated visual systems [19–23]. A number of models use an ideal Bayesian observer that searches for a known target in noise-limited images [17,24,25]. Such frameworks typically do not model the degradation in retinal eccentricity with varying resolution feature extraction and the approach is limited to synthetic images for which the statistical distribution of the noise is known. There are other object detectors that can be applied to real world images but use the same high resolution representation of the target (template) across the whole visual field [26,27]. One group has implemented in hardware a visual sensing approach with two components: a pre-attentive component providing a fixed field of view (FOV) at low resolution, and a localized shiftable FOV at high resolution, designed to recognize and interpret events detected by the pre-attentive system [19,20]. However, no previous study has implemented a physiologically based foveated visual system and compared its performance for ecologically relevant search tasks against a homogeneous high resolution (non-foveated) system to specifically assess the decision accuracy costs of incorporating a varying resolution system. The goal of the present work is to investigate the impact on object search performance of using a foveated visual field with physiologically plausible cortical peripheral pooling and saccade exploration, and compare it to a visual system with access to high spatial resolution at all points in the scene [28–30]. We evaluate the accuracy of the models in finding different objects in real scenes. To allow for a fair evaluation of the performance costs of foveation, both models (foveated and non-foveated) need to be implemented within a common computational framework. Our reasoning is that if a foveated object detection model with eye movements can achieve similar object detection accuracy as a non-foveated approach, it might suggest a possible reason for the evolution of foveated systems in organisms: achieving successful object detection while minimizing computational and metabolic costs. To achieve our goal we incorporate a visual field with varying spatial resolution [31–34] to contemporary object detection methods extended to use a latent linear discriminant formulation (§“The foveated object detector (FOD) ”). There are many possible methods to implement a foveated visual field in an object detection system. In primates, foveation arises from various properties of the visual system including varying photoreceptor density across the retina [35], larger convergence onto ganglion cells with increasing retinal eccentricity [36,37], the higher number of neurons per mm2 of retina at the visual cortex for foveal areas and spatial pooling possibly contributing to crowding effects (for a review of contributions, see Rosenholtz [38]). In this work, we opt to use a recent model [39] which specifies how responses of elementary sensors are pooled at the layers (V1 and V2) of the human visual cortex. The model specifies the shapes and sizes of V1, V2 regions which pool responses from the visual field. This is clearly a simplification of the multi-stage processing in the human visual system (lower photoreceptor density and higher input convergence at the ganglion cells, lower number of V1 neurons with increasing eccentricity) accounting for the foveated properties of vision. However, such a simplified model seems to capture many aspects of peripheral processing including some crowding effects [38,39]. We use a simplified version (only the V1 layer, see §“Foveated visual field” for details) of this model as the foveated visual field of our object detector (Fig 1). We call our detector the foveated object detector (FOD). The FOD computational savings arise from the fewer computations (dot products) related to the coarser spatial sampling of features (due to spatial pooling) in the visual periphery. Importantly, we used the same computer vision object detection framework to develop a foveated system, and a non-foveated system with homogeneous high spatial resolution across the visual field. The high spatial resolution system is the default object detection model which processes all information in the image with the same feature extraction stages and is known as a sliding window method (SW) in computer vision. The term sliding window refers to a re-current application of the feature extraction and classifier across the entire image by shifting spatial windows defining a region of interest. We compared performances of the models on a standard computer vision image dataset (PASCAL VOC [40]) allowing for direct evaluation of the impact of a biologically based foveated visual system on the default non-foveated object detector across 20 different classes of objects and about 5000 test images.
The non-foveated object detector, or the sliding window (SW) object detector (Fig 2), starts by extracting from the input image a set of image features known as the histogram of oriented gradients (HoG) [28,41] (Fig 3) which is a feature descriptor utilized in object detection models in computer vision. The HoGs refer to a distribution of orientations within small neighborhoods akin to the responses of various cell receptive fields with different orientations. The first stage is to convolve the image with a 1-D point discrete derivative kernel (i. e. , [-1,0, 1]) in both of the horizontal and vertical directions. The second stage entails computing the cell histograms. Each pixel within the cell codes a linear response to the various oriented kernels (filters). The local histograms are normalized relative to the mean of a block (Fig 3). This process results in a M-by-N-by-D feature map where M is the number of rows, N is the number columns and D is the number of features which is determined by the number of different edge orientations considered. Next, the feature map is convolved with the object template which was learned from the training images. The object template is a model of object appearance in the form of a P-by-K-by-D matrix of feature weights (typically P << M and K << N). The template is evaluated at all M * N locations on the feature map. Each evaluation is a dot product between the template weights and the HoG features of the spatial P-by-K region (which corresponds to a bounding box on the image) that is covered by the template. The dot product result is recorded as the detection score for the corresponding bounding box. The feature extraction and template evaluation process described above considers only a single template and a single scale of the input image, to simplify the explanation. However, in practice, there is uncertainty about the view-point and scale of the object appearing in the scene. Thus, the object detector had more than one template per object class, and each of these templates serves as an appearance model for a distinct view-point of the object (e. g. a bicycle viewed from the front and from the side). In all our experiments, we used two view-point templates per object class. In addition, since the scale of the object sought is not known apriori, a multiscale processing (we used 40 different scales as done in the Deformable Parts Model (DPM) model [28]) of the input image is required. Finally, all bounding boxes with detection scores higher than a given threshold are identified as object detections. However, if there are two bounding boxes that significantly overlap with each other, the one with the lower score is discarded (non-maxima suppression). We compared two models on the PASCAL VOC 2007 detection (comp3) challenge dataset and protocol [40]. The dataset contains 20 object classes, 5011 training images and 4952 test images. A training image might contain more than one instance of a specific object class. All results are obtained by training the classifiers on a different set of images than those utilized for testing. Our previous sections suggest that a computationally less costly foveated system can achieve similar performance accuracy finding an object in real scenes as a system with homogeneous high spatial resolution. Research has shown that visual areas in the brain also rapidly compute bottom-up information in terms of salient regions defined by contrast, edges and color [5–7]. These salient regions serve to identify potential locations in scenes for further computation. The impact of a foveated visual system in such saliency computations is not known. Here, we evaluate whether identifying the most salient region in an image, an important component of bottom-up attention useful to identify potential regions of a scene for further scrutiny, is affected by the process of foveation. Or in the contrary, can a foveated system with eye movements identify the same salient regions with less computation than a non-foveated system? We implemented a simple model of saliency that followed conceptually the model proposed by Li [6,49,50]. Such saliency model involves two computational aspects of saliency, namely iso-orientation suppression and contour enhancement, and also their dynamics. Here we only implemented the iso-orientation suppression aspect by using a simple center-surround operation. The current simplified implementation of the saliency model first extracts features by convolving Gabor receptive fields (4 scales and 8 orientations) with the input image. Each cell pools (sums) Gabor responses per orientation, within its receptive field. Then, a center surround computation is implemented by subtracting each cell’s response by the pooled response of the neighboring cells at that same orientation (iso-orientation suppression). The spatial distribution of responses after the suppressions were considered the saliency map and the highest value the top saliency score. We implemented two versions of this saliency model, a non-foveated version consisting of only foveal cells and a foveated version that uses the simplified Freeman-Simoncelli model as its visual field (The same visual field that is used by the FOD.) (§Foveated visual field). The non-foveated saliency model processes all locations of the input image with the same (high) resolution and each Gabor receptive field was suppressed by pooling the Gabors with the same orientation at eight neighboring locations. The foveated model processes the input image with varying resolution. The foveal center surround suppression consisted of subtracting from a cell’s receptive field response the pooled activity across eight (nearest) surrounding cells of the same orientation. However, the peripheral center surround was implemented by subtracting the pooled responses across four nearest neighboring cells. The foveated saliency model makes eye movements based on the saliency values computed at the peripheral cells. It executes a saccade to the location with the maximum saliency value. After saccade execution, the saliency values around foveation (a 2-degree radius area around each fixation point) is inhibited (inhibition of return). This prevents the model from getting caught at a maximally salient location and not executing additional eye movements. After n fixations, the model integrates saliency values for each location across fixations and selects the top saliency location. We compared decisions of the non-foveated and foveated saliency models in terms of their agreement in selecting the top salient location within the image. We compared the computational costs of the two models in terms of their associated total number of center-surround operations. Fig 9 shows the distance in degrees between the top salient location from the non-foveated saliency model (blue line) and that of the foveated saliency model. The comparison is plotted as a function of increasing number of fixations for the foveated saliency model. In red we show the fraction of the number of center surround operations of the foveated saliency model relative to the non-foveated model. This is shown as a function of number of fixations. The results show that the foveated model can generate a similar prediction for the most salient region as the non-foveated model but with a significantly less number of center-surround operations. Mathematical details of the foveated saliency model can be found in the supplementary section Part E in S1 Text.
The objective of our current work was to evaluate within a common framework the accuracy costs and computational savings of a foveated visual system relative to a full high resolution system. We evaluated an object detection paradigm over 20 object classes from a standard object detection dataset. Our results show that with five exploratory fixations, the foveated method achieves nearly the same performance as the non-foveated (high resolution SW) method (Fig 5). The foveated achieved such accuracy with 49. 6% of the sliding window method’s computational cost. Using a richer model (such as the Deformable Parts Model, DPM [28]) that selects potential locations for further object part processing, the foveated version of the model was able to match and even outperform the non-foveated SW-DPM while achieving computational savings (at less then 4. 16% of the SW-DPM’s computational cost). In addition, Fig 4 highlights the importance of guided eye movements in re-orienting the fovea to regions of interest in the scene. Eliminating the guidance diminishes the model’s ability to correctly detect the object with additional fixations. Together the results suggest that a foveated visual system with guided eye movements provide computational savings while preserving an organism’s ability to successfully detect objects in scenes. Our conclusions are limited by the utilized data set which although large does not represent all set of tasks that an organism or a human might face. For example, the PASCAL dataset does not contain a large number of images with small objects in the scenes. Such scenes might represent a more challenging test set for the foveated object detector and show potential accuracy losses beyond those quantified in the current investigation. On the other hand, our model only guides its eye movements based on peripheral information about the target while humans are known to utilize information about global statistics [12,51] and object co-occurrence [14–16,52] and multiple object configuration [53] to guide eye movements and aids perceptual decisions [54,55]. Inclusion of such sources of scene information might improve the FOD’s accuracy. In addition to evaluating performance of a foveated visual system for object search, we assessed the impact of a foveated visual system on the computation of saliency which is a fundamental bottom-up component that guides attention [5–7]. We compared a simplified implementation of a non-foveated saliency model and its foveated counterpart. The results showed that the foveated visual system with about 8-10 eye movements could approximate the same selection of the top salient location by a non-foveated high resolution saliency model. Future work should evaluate the generality of the results to more complex and different models of saliency [5–7,56,57]. In addition, our implementation of a foveated visual system concentrated on the spatial pooling losses occurring at cortical areas and possibly responsible for a great portion of the bottleneck of visual processing at the periphery [38]. A more detailed model could implement the peripheral loss starting with the reduction in photoreceptor density [35] and spatial sampling of retinal ganglion cells [58]. We believe that such detailed implementation would not qualitatively change the fundamental result of our paper, but future efforts need to test such prediction. Our work assessed the impact of foveation on perceptual performance but does not address the mechanism by which the foveated visual system is evolved or shaped during development. An interesting theory contends that cone density is shaped by the probability distribution of objects across the retina and is thus influenced by the frequency and accuracy of eye movements [59]. There have been previous efforts, (e. g. [60]), on biologically inspired object recognition. However, most of such models do not have a foveated visual field and thus do not execute eye movements. More recent work has implemented biologically inspired search methods. In [19], a fixed, pre-attentive, low-resolution wide-field camera is combined with a shiftable, attentive, high-resolution narrow-field camera, where the pre-attentive camera generates saccadic targets for the attentive, high-resolution camera. The difference between this and our method is that while their pre-attentive system has the same coarse resolution everywhere in the visual field, our method, which is a model of the V1 layer of the visual cortex, has a varying resolution that depends on the radial distance to the center of the fovea. There have been previous efforts to create foveated search models with eye movements [17,24,25,61]. Such models have been applied mostly to detect simple signals in computer generated noise [17,24,62] and used as benchmarks to compare against human eye movements and performance. Other biologically inspired methods include the target acquisition model (TAM) [26,63], the Infomax model [27] and artificial neural network based models [64,65]. TAM is a foveated model and it uses the Scale Invariant Feature Transform (SIFT) features [66] for representation and utilizes a training set of images to learn the appearance of the target object. However, their evaluation did not include variability in object appearance due to scale and viewpoint, i. e. the object instances always appeared at the same size and viewpoints both in training and testing sets. Simply using SIFT features does not guarantee successful detection of objects appearing at different sizes. Furthermore, their evaluation involves placing the objects on a uniform background rather than real scenes such as in the current work. The Infomax model, on the other hand, can use any previously trained object detector and works on natural images, although they report results for face detection only and not for generic object detection. Critically, the Infomax’s foveated architecture is not based on physiology and uses non-biological rectangular pooling regions. We emphasize that any evaluation of the performance cost of a foveated visual system will depend critically on the parameters of the pooling regions with retinal eccentricity. Thus, an accurate assessment of the performance of a human/primate foveated visual system requires implementing a model which pooling regions are bio-inspired and based on physiological measurements. Larochelle and Hinton [64], and Bazzani et al. [65] developed artificial neural network based models that have some sort of foveation. However, their application areas were different. Larochelle and Hinton [64] applied their model to image categorization and Bazzani et al. [65] applied their model to object tracking in videos. Most importantly, none of these models have evaluated a biologically plausible foveated architecture relative to a high resolution scheme within a common theoretical framework to assess the potential performance loss of a system with a human foveated visual system and guided eye movements. There has been substantial progress (e. g. [28–30,47,67–73]) in object detection research in recent years. However, humans, for now, are still unsurpassed in their ability to search for objects in visual scenes. The human brain relies on a variety of strategies. Object detection approaches have increasingly included some of the human strategies [19,28,60,65,74]. One remaining crucial difference between the human visual system and a modern object detector is that while humans process the visual field with decreasing resolution away [31–34] from the fixation point and make saccades to collect information, typical object detectors [28] scan all locations at the same resolution and repeats this at multiple scales. The sliding window (SW) method is the dominant model of search in object detection. Efficient alternatives to sliding windows can be categorized in two groups: 1. methods aimed at reducing the number of locations (m), 2. methods aimed at reducing the number of object categories (n). Since typically m >> n, there are a larger number efforts in trying to reduce m, however, reducing the contribution of the number of object classes has recently been receiving increasing interest as search for hundreds of thousands of object classes has started to be tackled [69]. According to this categorization, our proposed FOD method falls into the first group as it is designed to locate object instances by making a set of sequential fixations where in each fixation only a sparse set of locations are evaluated. Thus our proposed FOD scheme might provide an alternative bio-inspired method to other proposed methods to reduce the number of evaluated locations. There are number of previously proposed methods to reduce the number of locations to be evaluated. One line of research is the branch-and-bound methods [75,76] where an upper bound on the quality function of the detection model is used in a global branch and bound optimization scheme. Although the authors provide efficiently computable upper bounds for popular quality functions (e. g. linear template, bag-of-words, spatial pyramid), it might not be trivial to derive suitable upper bounds for a custom quality function. Our method, on the other hand, uses binary classification detection model and is agnostic to the quality function used. Another line of research is the cascaded detection framework [77–79] where a series of cheap to expensive tests are done to locate the object. Cascaded detection is similar to our method in the sense that simple, coarse and cheap evaluations are used together with complex, fine and expensive evaluations. However, we differ with it in that it is essentially a sliding window method with a coarse-to-fine heuristic used to reduce the number of total evaluations. Another coarse-to-fine search scheme is presented in [80] where a set of low to high resolution templates are used. The method starts by evaluating the lowest resolution template—which is essentially a sliding window operation—and selecting the high responding locations for further processing with higher resolution templates. Our FOD method, too, uses a set of varying resolution templates; however, these templates are evaluated at every fixation instead of serializing their evaluations with respect to resolution. In [47], a segmentation based method is proposed to yield a small set of locations that are likely to correspond to objects, which are subsequently used to guide the search in a selective manner. The locations are identified in an object class-independent way using an unsupervised multiscale segmentation approach. Thus, the method evaluates the same set of locations regardless of which object class is being searched for. In contrast, in our method, selection of locations to be foveated is guided by learned object class templates. The method in [74], similar to ours, works like a fixational system: at a given time step, the location to be evaluated next is decided based on previous observations. However, there are important differences. In [74], only a single location is evaluated at a time step whereas we evaluate all template locations within the visual field at each fixation. Their method returns only one box as the result whereas our method is able to output many predictions. Mathe et al. [81] proposed a search model that has foveation. However, this model does not have peripheral processing. The next fixation location is decided based on the history of foveal observations. A foveal observation corresponds to evaluating several regions in high (original) resolution (produced by a third-party segmentation algorithm) around the current fixation location. Finally, in recent years, we have witnessed a surge of research in convolutional neural network (CNN) based object detection [70–73]. The new models have almost doubled the detection performance (34. 1 mAP our result versus 59. 9 mAP for Faster RCNN [70] on the same dataset). It is clear that the type of features we extract from images (i. e. HOG) limit FOD’s performance as indicated by the higher performance of neural network based models (e. g. Faster RCNN). The FOD’s performance would improve if CNN-features were used instead of HoGs (but note that it would not be neurobiologically consistent to pool CNN features which have been shown to compute features beyond V1 [82], using a V1 model [39]. Nevertheless, it is important that future work evaluate the cost of a foveated system within the context of CNN framework to assess whether the findings in the current paper generalize to that approach. Also, the computational cost of the new CNN models is much higher compared to the DPM-like (e. g. HOG+SVM) motivating even further the development of alternatives to sliding window method. Thus, a number of region proposal methods including Selective Search [47], edge boxes [83], region proposal networks [70] have been proposed (see Hosang et al. ’s work [84] for a review). While it is true that region proposal methods greatly reduce the number of evaluation candidates, whether they are better than sliding window classifiers (in terms of accuracy and computational savings) is not a settled debate. State-of-the-art object detection (RCNN [85], Fast RCNN [86], Faster RCNN [70]) has abandoned region proposal methods. Faster RCNN, the best available object detector known to us, is not using a region proposal method to generate evaluation candidates. Instead, it uses a sliding window classifier which they call the “Region Proposal Network (RPN). ” RPN slides a 3x3 window on the output of the topmost convolutional layer, evaluates 9 different hypothesis (3 scales, 3 aspect ratios) at each location, and outputs the best scoring 300 hypothesis as the object candidates. Theoretically, RPN itself could potentially be made faster by using a foveated method such as ours. To summarize, the findings show that a foveated architecture with guided eye movements can preserve both bottom-up saliency and top-down search for objects of a homogeneous high resolution system while incurring important computational cost savings. The findings might suggest a possible explanation for the evolution of a foveated visual system with eye movements as a possible solution that gives the organism similar ability to that of a non-foveated high resolution system but with decreased metabolic costs for the brain as well as reduced neural resource allocation.
The Freeman-Simoncelli (FS) model [39] is a neuronal population model of V1 and V2 layers of the visual cortex. The model specifies how responses are pooled (averaged together) hierarchically beginning from the lateral geniculate nucleus to V1 and then the V2 layer. V1 cells encode information about local orientation and spatial frequency whereas the cells in V2 pools V1 responses non-linearly to achieve selectivity for compound features such as corners and junctions. The model is based on findings and physiological measurements of the primate visual cortex and specifies the shapes and sizes of the receptive fields of the cells in V1 and V2. According to the model, the sizes of receptive fields increase linearly as a function of the distance from the fovea and this rate of increase in V2 is larger than that of V1, which means V2 pools larger areas of the visual field in the periphery. The reader is referred to [39] for further details. We simplify the FS model in two ways. First, the model uses a Gabor filter bank to compute image features and we replace these with the HOG features [28,41]. Second, we only use the V1 layer and leave the non-linear pooling at V2 as future work. We use this simplified FS model as the foveated visual field of our object detector which is shown in Fig 1. The fovea subtends a radius of 2 degrees. We also only simulate a visual field with a radius of 10 degrees which is sufficient to cover the test images presented at a typical viewing distance of 40 cm. The square boxes with white borders (Fig 1 represent the pooling regions within the fovea. The surrounding colored regions are the peripheral pooling regions. While the foveal regions have equal sizes, the peripheral regions grow in size as a function—which is specified by the FS model—of their distance to the center of the fovea. The color represents the weights that are used in pooling, i. e. weighted summation of, the underlying responses. A pooling region partly overlaps with its neighboring pooling regions (see the supplementary material of [39] for details). Specifically, (i) spatial weights of the pooling regions, (ii) locations of pooling regions, and (iii) the number of angle and eccentricity bins, and (iv) the scaling factor of the pooling regions with eccentricity in our FOD model are all directly based on the FS model’s V1 layer. Assuming a viewing distance of 40cm, the whole visual field covers about a 500x500 pixel area (a pixel subtends 0. 08°). The foveal radius is 52 pixels subtending a visual angle of 4 degrees. The model M consists of the application of n retino-specific, linear templates (i. e. classifiers) corresponding to different object viewpoints and resolutions. Thus, the model has n components, each of which consists of a linear template and its specific location vector: Model M ′ s parameters: { (w i, ℓ i): i = 1,2, …, n } (1) where wi is a linear template and ℓi is the location of the template with respect to the center of the visual field. Among these parameters, wi are learnable (given a dataset) but ℓi are fixed (more on this in §“Initialization”) given the visual field parameters. The output of M given a fixation point f is an array of detection scores produced by the n retino-specific classifiers, corresponding to different locations within the image. The location variable ℓi defines a unique bounding box within the visual field for the ith template. Specifically, ℓi = (ωi, hi, xi, yi) is a vector whose variables respectively denote width, height and x, y coordinates of the ith template within the visual field. The template, wi, is a matrix of weights on the features extracted from the pooling regions underlying the bounding box ℓi. The dimensionality of wi, i. e. the total number of weights, depends both on the width and height of its bounding box and its location in the visual field. A component within the fovea covers a larger number of pooling regions compared to a peripheral component with the same width and height, hence the dimensionality of a foveal template is larger. Three example components are illustrated in Fig 10 where the foveal component (red) covers 7x5 = 35 pooling regions while the (blue and green) peripheral components cover 15 and 2 regions, respectively. Since a fixed number of features is extracted from each pooling region (regardless of its size), foveal components have higher-resolution templates associated with them. We use the feature extraction implementation of DPM (rel5) [28,87]. We use the maximum-a-posteriori (MAP) model [43] with inhibition of return (see next subsection) as the basic eye movement strategy of the FOD. The MAP model selects the location with the highest posterior probability of containing the target object as the next fixation location, that is fi+1 = center of ℓ* where ℓ * = arg max ℓ P (y ℓ = 1 | f 1, f 2, …, f i, I). (7) This search is done over uninhibited locations only. Finding the maximum of the posterior above is equivalent to finding the maximum of the posterior ratios, arg max ℓ P (y ℓ = 1 | f 1, …, f i, I) = arg max ℓ P (y ℓ = 1 | f 1, …, f i, I) P (y ℓ = 0 | f 1, …, f i, I) (8) since for two arbitrary locations ℓ1, ℓ2; let p1 = P (yℓ1 = 1|⋅) and p2 = P (yℓ2 = 1|⋅), then we have p 1 1 - p 1 > p 2 1 - p 2 ⇒ p 1 > p 2. (9) After each fixation, a circular area with approximately 2 degree radius around the fixation location is inhibited. The model is not allowed to fixate to a previously inhibited location.
A large number of species from primates to shrimps do not see the visual world with uniform spatial detail. An area with heightened sensitivity to spatial detail, known as the fovea in mammals is oriented through eye and head movements to scrutinize regions of interest in the visual environment. But why did many species evolve such foveated architecture for vision? Seeing with high spatial detail everywhere requires greater neuronal machinery and energy consumption, thus the advantage of a foveated visual system is to reduce metabolic costs. But does having a foveated visual system incur a price to the performance of the organism in visual tasks? Here, we show using a computer vision object detection model, that the foveated version of the model can attain similar search performance to its non-foveated version that processes the entire visual field with high spatial detail. The results might help explain the evolution of foveated visual systems with eye movements as a solution that preserves perceptual performance while resulting in computational and metabolic savings to the brain.
medicine and health sciences
linear discriminant analysis
ocular anatomy
brain
social sciences
neuroscience
mathematics
statistics (mathematics)
computer vision
vision
animal management
animal performance
research and analysis methods
sensory physiology
computer and information sciences
mathematical and statistical techniques
visual cortex
agriculture
visual system
psychology
eye movements
anatomy
physiology
biology and life sciences
sensory systems
sensory perception
ocular system
physical sciences
statistical methods
fovea centralis
Object detection through search with a foveated visual system