Many vision problems require matching images of object instances across different domains. These include fine-grained sketch-based image retrieval (FG-SBIR) and Person Re-identification (person ReID). Existing approaches attempt to learn a joint embedding space where images from different domains can be directly compared. In most cases, this space is defined by the output of the final layer of a deep neural network (DNN), which primarily contains features of a high semantic level. In this paper, we argue that both high and mid-level features are relevant for cross-domain instance matching (CDIM). Importantly, mid-level features already exist in earlier layers of the DNN. They just need to be extracted, represented, and fused properly with the final layer. Based on this simple but powerful idea, we propose a unified framework for CDIM. Instantiating our framework for FG-SBIR and ReID, we show that our simple models can easily beat the state-of-the-art models, which are often equipped with much more elaborate architectures.
Fine-grained SBIR With the growth of touch-screen devices, the problem of fine-grained image retrieval (FG-SBIR) has received increasing interest recently @cite_35 @cite_4 @cite_32 @cite_15 . It was first proposed in @cite_35 where a sketch is used to retrieve a gallery photo containing an object in the same pose or viewpoint, rather than the same black object instance per se . The instance-level FG-SBIR problem was first formalized and addressed in @cite_4 targeting an online shopping application, a user sketches a specific shoe to search for it among all the shoes available in an online shopping website. In this scenario, the pose or viewpoint is aligned between the query sketch and the gallery photos and the gallery contains only object photos of the same category as the query sketch. Differently @cite_32 studies FG-SBIR in the presence of objects of multiple categories. Both works employ a deep triplet ranking model to learn a domain-invariant representation shared by both the photo and sketch domains. Only the final CNN layers are used as the representation; thus primarily high-level semantic features are included.
@cite_35 @cite_15 @cite_4 @cite_32
2293340461 2467281799 2466618734
(c) 2014. The copyright of this document resides with its authors. It may be distributed unchanged freely in print or electronic forms. We investigate the problem of fine-grained sketch-based image retrieval (SBIR), where free-hand human sketches are used as queries to perform instance-level retrieval of images. This is an extremely challenging task because (i) visual comparisons not only need to be fine-grained but also executed cross-domain, (ii) free-hand (finger) sketches are highly abstract, making fine-grained matching harder, and most importantly (iii) annotated cross-domain sketch-photo datasets required for training are scarce, challenging many state-of-the-art machine learning techniques. In this paper, for the first time, we address all these challenges, providing a step towards the capabilities that would underpin a commercial sketch-based image retrieval application. We introduce a new database of 1,432 sketchphoto pairs from two categories with 32,000 fine-grained triplet ranking annotations. We then develop a deep tripletranking model for instance-level SBIR with a novel data augmentation and staged pre-training strategy to alleviate the issue of insufficient fine-grained training data. Extensive experiments are carried out to contribute a variety of insights into the challenges of data sufficiency and over-fitting avoidance when training deep networks for finegrained cross-domain ranking tasks. We present the Sketchy database, the first large-scale collection of sketch-photo pairs. We ask crowd workers to sketch particular photographic objects sampled from 125 categories and acquire 75,471 sketches of 12,500 objects. The Sketchy database gives us fine-grained associations between particular photos and sketches, and we use this to train cross-domain convolutional networks which embed sketches and photographs in a common feature space. We use our database as a benchmark for fine-grained retrieval and show that our learned representation significantly outperforms both hand-crafted features as well as deep features trained for sketch or photo classification. Beyond image retrieval, we believe the Sketchy database opens up new opportunities for sketch and image understanding and synthesis.