The Importance of Visual Attention in Improving the 3D-TV Viewing Experience: Overview and New Perspectives <s> C. 3D Visual Attention Models <s> A model for aspects of visual attention based on the concept of selective tuning is presented. It provides for a solution to the problems of selection in an image, information routing through the visual processing hierarchy and task-specific attentional bias. The central thesis is that attention acts to optimize the search procedure inherent in a solution to vision. It does so by selectively tuning the visual processing network which is accomplished by a top-down hierarchy of winner-take-all processes embedded within the visual processing pyramid. Comparisons to other major computational models of attention and to the relevant neurobiology are included in detail throughout the paper. The model has been implemented; several examples of its performance are shown. This model is a hypothesis for primate visual attention, but it also outperforms existing computational solutions for attention in machine vision and is highly appropriate to solving the problem in a robot vision system. <s> BIB001 </s> The Importance of Visual Attention in Improving the 3D-TV Viewing Experience: Overview and New Perspectives <s> C. 3D Visual Attention Models <s> Visual attention is the ability to rapidly detect the interesting parts of a given scene. Inspired by biological vision, the principle of visual attention is used with a similar goal in computer vision. Several previous works deal with the computation of visual attention from images provided by standard video cameras, but little attention has been devoted so far to scene depth as source for visual attention. The investigation presented in this paper aims at an extension of the visual attention model to the scene depth component. The first part of the paper is devoted to the integration of depth in the computational model built around conspicuity and saliency maps. The second part is devoted to experimental work in which results of visual attention, obtained form the extended model and for various 3D scenes, are presented. The results speak for the usefulness of the enhanced computational model. <s> BIB002 </s> The Importance of Visual Attention in Improving the 3D-TV Viewing Experience: Overview and New Perspectives <s> C. 3D Visual Attention Models <s> The necessity and utility of visual attention are discussed in the context of stereo vision in machines and primates. Specific problems that arise in this domain including binocular rivalry, and the deployment of attention in three-dimensional space are considered. Necessary conditions are outlined for achieving appropriate attentional behaviour in both the aforementioned domains. In this light, we outline classes of existing computational models of attention and discuss their applicability for realizing binocular attention. Finally, a stereo attention framework is presented by considering the tenets of an existing attentional architecture that extends naturally to the binocular domain, in conjunction with the connectivity of units involved in achieving stereo vision. <s> BIB003 </s> The Importance of Visual Attention in Improving the 3D-TV Viewing Experience: Overview and New Perspectives <s> C. 3D Visual Attention Models <s> Compared with traditional mono-view video, three-dimensional video (3DV) provides user interactive functionalities and stereoscopic perception, which makes people more interested in pop-out regions or the regions with small depth value. Thus, traditional visual attention model for mono-view video can hardly be directly applied to stereoscopic visual attention (SVA) analysis for 3DV. In this paper, we propose a bottom-up SVA model to simulate human visual system with stereoscopic vision more accurately. The proposed model is based on multiple perceptual stimuli including depth information, luminance, color, orientation and motion contrast. Then, a depth based dynamic fusion is proposed to integrate these features. The experimental results on multi-view video test sequences show that the proposed model maintains high robustness and is able to efficiently simulate SVA of human eyes. <s> BIB004
One early proposal of a computational model of depth-based visual attention for target detection came from Maki et al.  ,  . The simple architecture is based on a first stage of parallel detection of preattentive cues (image flow, stereo disparity, and motion detection), followed by a stage of cue integration using selection criteria based on nearness and motion. Two masks are first computed based on the pursuit and saccade modes found in human scan paths. Depth is then used to apply a priority criterion: in each frame either the target pursuit or the target mask is selected to be the final mask based on the depth. The hypothesis made by Maki et al. is that the target that is closer to the observer should be assigned highest priority. As discussed by the authors, this hypothesis may hold in a scenario where the observer has to avoid obstacles. However, we argue that this hypothesis may not necessarily hold in a scenario of viewing complex entertainment video content where the closest object may not be the only or main area of interest. In essence, the model proposed by Maki et al. serves only the purpose of detecting the closest moving object to the observer. Indeed, they demonstrated the application of their model using a scenario in which a moving or stationary stereo-camera selectively masks out different moving objects in real-scenes and holds gaze on them over some frames. They showed that their model kept focusing on the moving object that is the closest to the camera. Ouerhani and Hügli also proposed a model of visual attention using scene depth information to extend a 2D saliency-based computational model BIB002 . Firstly, a number of low-level features are extracted from the image to build feature maps. Secondly, each feature map is transformed into a corresponding conspicuity map based on a multi-resolution center-surround mechanism. Finally, a linear combination of the conspicuity maps is used to produce an overall saliency map for the image. In order to integrate the effect of depth, additional depth-related features are extracted from the scene, resulting in additional conspicuity maps to be integrated in the final linear combination. Several depth features were initially considered: depth representing the distance from camera to observer, mean curvature providing information on the geometry of objects in the scene, and depth gradient providing information on depth changes in the scene. However, Ouerhani and Hügli only integrated depth as an additional feature in their proposed model. Using a few images, they showed the usefulness of integrating depth information in a computational model. However, there is no mention of formal subjective experiments using an eye-tracker to evaluate the performance of the model and the added value of depth in the model. Although past works have studied the mechanisms of stereovision and have proposed perception models, one of the few studies on the modeling of stereo visual attention found in the literature were from Bruce and Tsotsos who discussed the issue of binocular rivalry occurring in stereo-vision and the deployment of attention in three-dimensional space BIB003 . They also discussed the difficult and biologically implausible translation of some types of 2D computational visual attention models to the case of stereo-vision. In particular, they singled out hierarchical models that extract basic and independent features to produce a saliency map used to predict shifts of attention. They criticized the independence of the extracted features as this is not completely biologically plausible and is potentially more difficult to justify for the modeling of binocular visual attention. It is argued that stereoscopic visual attention models must take into account conflicts between the two eyes resulting from occlusions or large disparities. In other words, the behavior of each eye and it corresponding eye gaze cannot be considered independently. There is a compelling argument that an appropriate model in a biological sense should accommodate shifts in the position of an attended event from one eye to another. Therefore, the representation of a 2D saliency map obtained independently for each stereoscopic view will discard the relationship and correspondence between the two eyes. Based on these considerations, Bruce and Tsotsos proposed a stereo attention framework from an existing 2D visual attention model using a visual pyramid processing architecture BIB001 . Their extension concerns the addition of interpretive neuronal units in the pyramid dedicated in achieving stereo-vision. The architecture of the model includes neurons tuned to a variety of disparities, while preserving the connectivity among interpretive units. The study by Bruce and Tsotsos used a few simple synthetic images with structural objects composed of mainly straight lines but did not mention any consideration on moving sequences with more complex scenarios. Unfortunately, results are not discussed in terms of comparison with ground-truth data collected through an eyetracking experiment. Despite the opinions of Bruce and Tsotsos on the applicability of hierarchical models to 3D video, Zhang et al. proposed a bottom-up visual attention model for stereoscopic content BIB004 . Their approach consists in extending a hierarchical model for the stereoscopic vision by using the depth map of the stereoscopic content as an additional cue. A spatial and a motion saliency map are constructed from features such as color, orientation and motion contrasts. A depth-based fusion with the spatial and motion saliency map is then used. The fusion is designed such that the relative importance of spatial or motion attention is set according to the strength of motion contrast. We can raise two main criticisms concerning the work by Zhang et al. Firstly, one single spatial and one single motion saliency map are constructed from features extracted from either of the two views. Although the model is presented to handle stereoscopic video content, there is no mention of fusion of information between the two views or interview competition. The model seems more appropriate to a Multiview Video plus Depth (MVD) sequence, where each view has a corresponding depth map. Therefore the model can be considered as computing a prediction of visual attention independently for each view, rather than taking into account the stereoscopic perception of the video. Secondly, there is no mention of comparison of the model's results to ground-truth data obtained with human participants. It is therefore not possible to judge whether the addition of the depth cue had a significant impact on the prediction behavior of the original 2D bottom-up model.