Fine grained sketch based image retrieval: The role of part aware attributes
We study the problem of fine grained sketch based image retrieval. By performing instance level (rather than category level) retrieval, it embodies a timely and practical application, particularly with the ubiquitous availability of touchscreens. Three factors contribute to the challenging nature of the problem: (i) free hand sketches are inherently abstract and iconic, making visual comparisons with photos more difficult, (ii) sketches and photos are in two different visual domains, i.e. black and white lines vs. color pixels, and (iii) fine grained distinctions are especially challenging when executed across domain and abstraction level. To address this, we propose to detect visual attributes at part level, in order to build a new representation that not only captures fine grained characteristics but also traverses across visual domains. More specifically, (i) we propose a dataset with 304 photos and 912 sketches, where each sketch and photo is annotated with its semantic parts and associated part level attributes, and with the help of this dataset, we investigate (ii) how strongly supervised deformable part based models can be learned that subsequently enable automatic detection of part level attributes, and (iii) a novel matching framework that synergistically integrates low level features, mid level geometric structure and high level semantic attributes to boost retrieval performance. Extensive experiments conducted on our new dataset demonstrate value of the proposed method.
2016 IEEE Winter Conference on Applications of Computer Vision (WACV)