Learning fine-grained details is a key issue in image aesthetic assessment. Most of the previous methods extract the fine-grained details via random cropping strategy, which may undermine the integrity of semantic information. Extensive studies show that humans perceive fine-grained details with a mixture of foveal vision and peripheral vision. Fovea has the highest possible visual acuity and is responsible for seeing the details. The peripheral vision is used for perceiving the broad spatial scene and selecting the attended regions for the fovea. Inspired by these observations, we propose a Gated Peripheral-Foveal Convolutional Neural Network (GPF-CNN). It is a dedicated double-subnet neural network, i.e. a peripheral subnet and a foveal subnet. The former aims to mimic the functions of peripheral vision to encode the holistic information and provide the attended regions. The latter aims to extract fine-grained features on these key regions. Considering that the peripheral vision and foveal vision play different roles in processing different visual stimuli, we further employ a gated information fusion (GIF) network to weight their contributions. The weights are determined through the fully connected layers followed by a sigmoid function. We conduct comprehensive experiments on the standard AVA and this http URL datasets for unified aesthetic prediction tasks: (i) aesthetic quality classification; (ii) aesthetic score regression; and (iii) aesthetic score distribution prediction. The experimental results demonstrate the effectiveness of the proposed method.
Early attempts in image aesthetic assessment cast this problem as a classification problem, such as @cite_42 @cite_24 @cite_37 @cite_41 @cite_20 . They classified the images into high or low aesthetic quality based on the threshold of the weighted mean scores of human rating. Other research such as @cite_2 @cite_8 used the regression model to predict the aesthetic score. However, the image aesthetic quality assessment is highly subjective. The rated scores of different people may differ greatly due to the cultural background. Thus a scalar value is insufficient to provide the degree of consensus or diversity of opinion among annotators @cite_19 . Considering this, recent research focuses on directly predicting the label distribution of the scores. @cite_19 , Jin proposed a new CJS loss to predict the aesthetic label distribution. Murray @cite_5 used the Huber loss to predict the aesthetic score distribution. But they predicted each discrete probability independently. Talebi @cite_15 treated the score distribution as ordered classes and used squared EMD loss to predict the score distributions. In this paper, similar with @cite_15 , we optimize our networks by minimizing EMD loss.
@cite_37 @cite_8 @cite_41 @cite_42 @cite_24 @cite_19 @cite_2 @cite_5 @cite_15 @cite_20
2749958701 2806817979 2023169975 2748901724
Aesthetic quality prediction is a challenging task in the computer vision community because of the complex interplay with semantic contents and photographic technologies. Recent studies on the powerful deep learning based aesthetic quality assessment usually use a binary high-low label or a numerical score to represent the aesthetic quality. However the scalar representation cannot describe well the underlying varieties of the human perception of aesthetics. In this work, we propose to predict the aesthetic score distribution (i.e., a score distribution vector of the ordinal basic human ratings) using Deep Convolutional Neural Network (DCNN). Conventional DCNNs which aim to minimize the difference between the predicted scalar numbers or vectors and the ground truth cannot be directly used for the ordinal basic rating distribution. Thus, a novel CNN based on the Cumulative distribution with Jensen-Shannon divergence (CJS-CNN) is presented to predict the aesthetic score distribution of human ratings, with a new reliability-sensitive learning method based on the kurtosis of the score distribution, which eliminates the requirement of the original full data of human ratings (without normalization). Experimental results on large scale aesthetic dataset demonstrate the effectiveness of our introduced CJS-CNN in this task. The ability to rank the images based on their appearance finds many real-world applications, such as image retrieval or image album creation. Despite the recent dominance of deep learning methods in computer vision which often result in superior performance, they are not always the methods of choice because they lack interpretability. In this paper, we investigate the possibility of improving the image aesthetic inference of the convolutional neural networks with hand-designed features that rely on domain expertise in various fields. We perform a comparison of hand-crafted feature sets in their ability to predict fine-grained aesthetics scores on two image aesthetics data sets. We observe that even feature sets published earlier are able to compete with more recently published algorithms and, by combining the algorithms, a significant improvement in predicting image aesthetics is possible. By using a tree-based learner, we perform the feature elimination to understand the best performing features overall and across different image categories. Only roughly 15 and 8 of the features are needed to achieve full performance in predicting a fine-grained aesthetic score and binary classification, respectively. By combining the hand-crafted features with metafeatures that predict the quality of an image based on convolutional neural network features, the model performs better than a baseline VGG16 model. One can, however, achieve more significant improvement in both aesthetics score prediction and binary classification by fusing the hand-crafted features and the penultimate layer activations. Our experiments indicate an improvement up to 2.2 achieving current state-of-the-art binary classification accuracy on the aesthetic visual analysis data set when the hand-designed features are fused with activations from VGG16 and ResNet50 networks. Current research of aesthetic assessment for images assumes one aesthetic score or one aesthetic label for an image, ignoring the relations of multiple aesthetic-related attributes. However, most images can be described by multiple aesthetic attributes simultaneously. Therefore, in this paper, we propose multiple aesthetic attribute prediction and classification by modeling relations among aesthetic attributes through Bayesian Networks (BN). In order to realize continuous aesthetic attribute prediction, each aesthetic attribute is represented by a three-node BN, including the discretized aesthetic attribute label, the predicted aesthetic attribute score, and the measurement of the aesthetic attribute score. In addition, the relations among multiple aesthetic attributes are modeled by another discrete BN, whose structure and conditional probabilities are learned from the training data. The attribute measurements are obtained by an existing image-driven regression method. With the learned BN, we infer the true discrete label and continuous score for each attribute by combining the relations among attributes with the previously obtained measurements. Experiments on the Memorability dataset show the superiority of our proposed approach to current image-driven methods for both multiple continuous aesthetic attribute score prediction and multiple discrete aesthetic attribute label classification, indicating the effectiveness of the captured relations for aesthetic quality assessment. In this paper we investigate the aesthetic image classification problem, also known as automatically classifying an image into low or high aesthetic quality, which is quite a challenging problem. Considering both the local and global information of images is quite important for image aesthetic quality assessment. Currently, a powerful inception module is proposed which shows very high performance in object classification. We have the observation that the inception module has the ability of considering both the local and global features in nature. Thus, in this paper, we propose a novel DCNN structure codenamed ILGNet for image aesthetics classification, which introduces the Inception module and connects intermediate Local layers to the Global layer for the output. In addition, the ILGNet is derived from part of the GoogLeNet. Thus, we can easily use a pre-trained image classification GoogleLeNet model on the ImageNet dataset and fine tune our connected local and global layer on the large scale aesthetics assessment AVA dataset. The experimental results show that the proposed ILGNet outperforms the state of the art results in image aesthetics assessment in the AVA benchmark. The time cost of both training and test of the ILGNet are significantly less than those of full GoogLeNet with only a little reduction of the classification accuracy. Our ILGNet can achieve similar classification accuracy as that of 2 3 GoogLeNet, whose computational cost is nearly twice of ours. This makes the aesthetic assessment model more easily to be integrated into mobile and embedded systems.