Humans comprehend a natural scene at a single glance; painters and other visual artists, through their abstract representations, stressed this capacity to the limit. The performance of computer vision solutions matched that of humans in many problems of visual recognition. In this paper we address the problem of recognizing the genre (subject) in digitized paintings using Convolutional Neural Networks (CNN) as part of the more general dealing with abstract and or artistic representation of scenes. Initially we establish the state of the art performance by training a CNN from scratch. In the next level of evaluation, we identify aspects that hinder the CNNs' recognition, such as artistic abstraction. Further, we test various domain adaptation methods that could enhance the subject recognition capabilities of the CNNs. The evaluation is performed on a database of 80,000 annotated digitized paintings, which is tentatively extended with artistic photographs, either original or stylized, in order to emulate artistic representations. Surprisingly, the most efficient domain adaptation is not the neural style transfer. Finally, the paper provides an experiment-based assessment of the abstraction level that CNNs are able to achieve.
The process of transferring knowledge from natural photography to art objects has been previously addressed beyond the recent transfer from ImageNet to WikiArt @cite_28 ; yet their solution is general as it is common to use ImageNet pre-trained CNNs on smaller databases. 3D object reconstruction can be augmented if information from old paintings is available @cite_34 . Classifiers (deep CNNs) trained on real data are able to locate objects such as cars, cows and cathedrals @cite_23 if the artistic rendering is not very abstract. The problem of detecting recognizing objects in any type of data, regardless if it is real or artistic, was named cross-depiction by @cite_7 ; however the problem is noted as being particular difficult and in the light of dedicated benchmarks @cite_10 , the results show plenty of space for improvement.
@cite_7 @cite_28 @cite_23 @cite_34 @cite_10
2397563792 2161381512 2520857446 2615562997
We consider the problem of recognizing objects in collections of art works, in view of automatically labeling, searching and organizing databases of art works. To avoid manually labelling objects, we introduce a framework for transferring a convolutional neural network (CNN), trained on available large collections of labelled natural images, to the context of drawings. We retrain both the top and the bottom layer of the network, responsible for the high-level classiication output and the low-level features detection respectively, by transforming natural images into drawings. We apply this procedure to the drawings in the Jan Brueghel Wiki, and show the transferred CNN learns a discriminative metric on drawings and achieves good recognition accuracy. We also discuss why standard descriptor-based methods is problematic in the context of drawings. Convolutional neural networks (CNN) have recently shown outstanding image classification performance in the large- scale visual recognition challenge (ILSVRC2012). The suc- cess of CNNs is attributed to their ability to learn rich mid- level image representations as opposed to hand-designed low-level features used in other image classification meth- ods. Learning CNNs, however, amounts to estimating mil- lions of parameters and requires a very large number of annotated image samples. This property currently prevents application of CNNs to problems with limited training data. In this work we show how image representations learned with CNNs on large-scale annotated datasets can be effi- ciently transferred to other visual recognition tasks with limited amount of training data. We design a method to reuse layers trained on the ImageNet dataset to compute mid-level image representation for images in the PASCAL VOC dataset. We show that despite differences in image statistics and tasks in the two datasets, the transferred rep- resentation leads to significantly improved results for object and action classification, outperforming the current state of the art on Pascal VOC 2007 and 2012 datasets. We also show promising results for object and action localization. The objective of this work is to recognize object categories in paintings, such as cars, cows and cathedrals. We achieve this by training classifiers from natural images of the objects. We make the following contributions: (i) we measure the extent of the domain shift problem for image-level classifiers trained on natural images vs paintings, for a variety of CNN architectures; (ii) we demonstrate that classification-by-detection (i.e. learning classifiers for regions rather than the entire image) recognizes (and locates) a wide range of small objects in paintings that are not picked up by image-level classifiers, and combining these two methods improves performance; and (iii) we develop a system that learns a region-level classifier on-the-fly for an object category of a user’s choosing, which is then applied to over 60 million object regions across 210,000 paintings to retrieve localised instances of that category. Humans are capable of perceiving a natural scene at a glance and painters, through their representations, push this capacity to the limit by abstraction. Yet we still do it... What about machines? In this paper we address the problem of recognizing the theme (scene type) in digitized paintings. The approach is based on the Convolutional Neural Network framework and the chosen architecture is the recent Residual Network. In the first level of evaluation, we determine the recognition rate of a CNN given a database of 80,000 annotated digitized paintings. In the second level we evaluate the impact of extending the training database with photographs directly and via two domain adaptation functions and thus we are able to assess the abstraction level that CNN is capable to achieve.