Automatic art analysis has been mostly focused on classifying artworks into different artistic styles. However, understanding an artistic representation involves more complex processes, such as identifying the elements in the scene or recognizing author influences. We present SemArt, a multi-modal dataset for semantic art understanding. SemArt is a collection of fine-art painting images in which each image is associated to a number of attributes and a textual artistic comment, such as those that appear in art catalogues or museum collections. To evaluate semantic art understanding, we envisage the Text2Art challenge, a multi-modal retrieval task where relevant paintings are retrieved according to an artistic text, and vice versa. We also propose several models for encoding visual and textual artistic representations into a common semantic space. Our best approach is able to find the correct image within the top 10 ranked images in the 45.5 of the test samples. Moreover, our models show remarkable levels of art understanding when compared against human evaluation.
With the digitalization of large collections of fine-art paintings and the emergence of publicly available online art catalogs such as WikiArt http: www.wikiart.org or the Web Gallery of Art https: www.wga.hu , computer vision researchers become interested in analyzing fine-art paintings automatically. Early work @cite_19 @cite_22 @cite_0 @cite_23 proposes methods based on handcrafted visual features to identify an author and or a specific style in a piece of art. Datasets used in these kinds of approaches, such as PRINTART @cite_0 and Painting-91 @cite_23 , are rather small, with 988 and 4,266 painting images, respectively. Mensink and Van Gemert introduce in @cite_21 the large-scale Rijksmuseum dataset for multi-class prediction, consisting on 112,039 images from artistic objects, although only 3,593 are from fine-art paintings. With the success of convolutional neural networks (CNN) in large-scale image classification @cite_1 , deep features from CNNs replace handcrafted image representations in many computer vision applications, including painting image classification @cite_17 @cite_5 @cite_25 @cite_13 @cite_7 @cite_15 , and larger datasets are made publicly available @cite_5 @cite_15 . In these methods, paintings are fed into a CNN to predict its artistic style or author by studying its visual aesthetics.
@cite_22 @cite_7 @cite_21 @cite_1 @cite_0 @cite_19 @cite_23 @cite_5 @cite_15 @cite_13 @cite_25 @cite_17
2615562997 2397563792 2766086266 2765362515
Humans are capable of perceiving a natural scene at a glance and painters, through their representations, push this capacity to the limit by abstraction. Yet we still do it... What about machines? In this paper we address the problem of recognizing the theme (scene type) in digitized paintings. The approach is based on the Convolutional Neural Network framework and the chosen architecture is the recent Residual Network. In the first level of evaluation, we determine the recognition rate of a CNN given a database of 80,000 annotated digitized paintings. In the second level we evaluate the impact of extending the training database with photographs directly and via two domain adaptation functions and thus we are able to assess the abstraction level that CNN is capable to achieve. We consider the problem of recognizing objects in collections of art works, in view of automatically labeling, searching and organizing databases of art works. To avoid manually labelling objects, we introduce a framework for transferring a convolutional neural network (CNN), trained on available large collections of labelled natural images, to the context of drawings. We retrain both the top and the bottom layer of the network, responsible for the high-level classiication output and the low-level features detection respectively, by transforming natural images into drawings. We apply this procedure to the drawings in the Jan Brueghel Wiki, and show the transferred CNN learns a discriminative metric on drawings and achieves good recognition accuracy. We also discuss why standard descriptor-based methods is problematic in the context of drawings. This paper aims to generate a better representation of visual arts, which plays a key role in visual arts analysis works. Museums and galleries have a large number of artworks in the database, hiring art experts to do analysis works (e.g., classification, annotation) is time consuming and expensive and the analytic results are not stable because the results highly depend on the experiences of art experts. The problem of generating better representation of visual arts is of great interests to us because of its application potentials and interesting research challenges---both content information and each unique style information within one artwork should be summarized and learned when generating the representation. For example, by studying a vast number of artworks, art experts summary and enhance the knowledge of unique characteristics of each visual arts to do visual arts analytic works, it is non-trivial for computer. In this paper, we present a unified framework, called DeepArt, to learn joint representations that can simultaneously capture contents and style of visual arts. This framework learns unique characteristics of visual arts directly from a large-scale visual arts dataset, it is more flexible and accurate than traditional handcraft approaches. We also introduce Art500k, a large-scale visual arts dataset containing over 500,000 artworks, which are annotated with detailed labels of artist, art movement, genre, etc. Extensive empirical studies and evaluations are reported based on our framework and Art500k and all those reports demonstrate the superiority of our framework and usefulness of Art500k. A practical system for visual arts retrieval and annotation is implemented based on our framework and dataset. Code, data and system are publicly available at http: deepart.ece.ust.hk. Compared with normal modalities, the representations of paintings are much more complex due to its large intra-class and small inter-class variation. This poses more difficulties in the task of authorship identification. In this paper, we propose a multi-task multi-range (MTMR) representation framework and try to resolve this issue in two ways. First, we investigate how to improve the representation through multi-task learning. Specifically, we attempt to optimize authorship identification with subtly correlated identification tasks such as style, genre and date. Second, in order to make the representation more comprehensive and reduce the information loss from image scaling, we propose a multi-range structure which is composed of local, regional and global representations. Experiments on the two most representative large-scale painting datasets, Rijksmuseum Challenge and Wikiart, have shown that our method significantly outperforms the existing methods. To give better understanding and provide more effective predictions, we utilize random forest as the feature ranking method to analyze the importance of different features and apply external knowledge matching to further examine the predictions. Moreover, the framework's effects of identifying the authorship are visualized on the paintings' artist-characteristic regions and t-SNE is further applied to perform artist-based cluster analysis. Extensive validation has demonstrated that the proposed framework yields superior performance in the chanllenging task of painting authorship identification.