In this work, we apply state-of-the-art Convolutional Neural Network(CNN) architectures for saliency prediction. Our results show that better saliency features can be delivered by a deeper CNN model. However, it is very space-consuming to apply a complex model due to the large size of input images. The space complexity becomes even more problematic when we extract features from multiple convolutional layers or different models. In this paper, we propose a modular saliency system which aims at splitting the whole network into small modules. The main difference in our approach s that the encoder and decoder can be separately trained for the scalability. Furthermore, the encoder can contain more than one CNN model to extract features and the models can have different architectures or pre-trained on different datasets. This parallel design allows us to better utilize the computational space in order to apply more powerful encoder. More importantly, our network can be easily expanded almost without extra spaces, other pre-trained CNN models can be combined for a wider range of visual knowledge. We denote our expandable multi-layer network as EML-NET in this paper. Our method is evaluated on three public saliency benchmarks, SALICON, MIT300 and CAT2000. The proposed EML-NET achieves state-of-the-art results on the metric of Normalized Scanpath Saliency using a modified loss function.
Koch and Ullman studied saliency maps using a neural network and a set of basic visual features @cite_9 . Their work later inspired the first computational bottom-up saliency method @cite_36 , in which extracted 42 feature maps including colour, intensity and orientation features. @cite_37 applied independent component analysis on image patches to compute saliency maps. Valenti @ proposed to utilize the information of colour and edges for saliency prediction. @cite_5 investigated the contribution of each visual feature using region co-variance matrices. @cite_24 applied thresholds in the CIE Lab colour space to compute boolean maps and subsequently computing saliency maps.
@cite_37 @cite_36 @cite_9 @cite_24 @cite_5
2151900481 2135957164 1993939318 2344129934
Inspired by the primate visual system, computational saliency models decompose visual input into a set of feature maps across spatial scales in a number of pre-specified channels. The outputs of these feature maps are summed to yield the final saliency map. Here we use a least square technique to learn the weights associated with these maps from subjects freely fixating natural scenes drawn from four recent eye-tracking data sets. Depending on the data set, the weights can be quite different, with the face and orientation channels usually more important than color and intensity channels. Inter-subject differences are negligible. We also model a bias toward fixating at the center of images and consider both time-varying and constant factors that contribute to this bias. To compensate for the inadequacy of the standard method to judge performance (area under the ROC curve), we use two other metrics to comprehensively assess performance. Although our model retains the basic structure of the standard saliency model, it outperforms several state-of-the-art saliency algorithms. Furthermore, the simple structure makes the results applicable to numerous studies in psychophysics and physiology and leads to an extremely easy implementation for real-world applications. A new bottom-up visual saliency model, Graph-Based Visual Saliency (GBVS), is proposed. It consists of two steps: first forming activation maps on certain feature channels, and then normalizing them in a way which highlights conspicuity and admits combination with other maps. The model is simple, and biologically plausible insofar as it is naturally parallelized. This model powerfully predicts human fixations on 749 variations of 108 natural images, achieving 98 of the ROC area of a human-based control, whereas the classical algorithms of Itti & Koch ([2], [3], [4]) achieve only 84 . Previous saliency detection algorithms used to focus on low level features directly or utilize a bunch of sample images and manually labeled ground truth to train a high level learning model. In this paper, we propose a novel coding-based saliency measure by exploring both global and local cues for saliency computation. Firstly, we construct a bottom-up saliency map by considering global contrast information via low level features. Secondly, by using a locality-constrained linear coding algorithm, a top-down saliency map is formulated based on the reconstruction error. To better exploit the local and global information, we integrate the bottom-up and top-down maps as the final saliency map. Extensive experimental results on three large benchmark datasets demonstrate that the proposed approach outperforms 22 state-of-the-art methods in terms of three popular evaluation measures, i.e., the Precision and Recall curve, Area Under ROC Curve and F-measure value. Furthermore, the proposed coding-based method can be easily applied in other methods for significant improvement. HighlightsWe present a coding-based algorithm for salient object detection.Integration of local and global cues makes the saliency maps more accurate, intact.Bottom-up maps provide foreground and background codebooks for following steps.Fusion of FC and BC based results makes the saliency results more uniform, robust.Our coding-based method can be easily applied in other methods for improvement. The purpose of this paper is the detection of salient areas in natural video by using the new deep learning techniques. Salient patches in video frames are predicted first. Then the predicted visual fixation maps are built upon them. We design the deep architecture on the basis of CaffeNet implemented with Caffe toolkit. We show that changing the way of data selection for optimisation of network parameters, we can save computation cost up to @math times. We extend deep learning approaches for saliency prediction in still images with RGB values to specificity of video using the sensitivity of the human visual system to residual motion. Furthermore, we complete primary colour pixel values by contrast features proposed in classical visual attention prediction models. The experiments are conducted on two publicly available datasets. The first is IRCCYN video database containing @math videos with an overall amount of @math frames and eye fixations of @math subjects. The second one is HOLLYWOOD2 provided @math movie clips with the eye fixations of @math subjects. On IRCYYN dataset, the accuracy obtained is of @math . On HOLLYWOOD2 dataset, results in prediction of saliency of patches show the improvement up to @math with regard to RGB use only. The resulting accuracy of @math is obtained. The AUC metric in comparison of predicted saliency maps with visual fixation maps shows the increase up to @math on a sample of video clips from this dataset.