Deep Learning for Free-Hand Sketch: A Survey <s> Lacking of Data and Annotations <s> We investigate the problem of fine-grained sketch-based image retrieval (SBIR), where free-hand human sketches are used as queries to perform instance-level retrieval of images. This is an extremely challenging task because (i) visual comparisons not only need to be fine-grained but also executed cross-domain, (ii) free-hand (finger) sketches are highly abstract, making fine-grained matching harder, and most importantly (iii) annotated cross-domain sketch-photo datasets required for training are scarce, challenging many state-of-the-art machine learning techniques. In this paper, for the first time, we address all these challenges, providing a step towards the capabilities that would underpin a commercial sketch-based image retrieval application. We introduce a new database of 1,432 sketchphoto pairs from two categories with 32,000 fine-grained triplet ranking annotations. We then develop a deep tripletranking model for instance-level SBIR with a novel data augmentation and staged pre-training strategy to alleviate the issue of insufficient fine-grained training data. Extensive experiments are carried out to contribute a variety of insights into the challenges of data sufficiency and over-fitting avoidance when training deep networks for finegrained cross-domain ranking tasks. <s> BIB001 </s> Deep Learning for Free-Hand Sketch: A Survey <s> Lacking of Data and Annotations <s> We present sketch-rnn, a recurrent neural network (RNN) able to construct stroke-based drawings of common objects. The model is trained on thousands of crude human-drawn images representing hundreds of classes. We outline a framework for conditional and unconditional sketch generation, and describe new robust training methods for generating coherent sketch drawings in a vector format. <s> BIB002
For single-modal sketch tasks, QuickDraw BIB002 is the largest dataset containing more than 50M sketches. However, the sketches in QuickDraw are highly abstract and noisy, due to that QuickDraw online game allows the urses to use only 20 seconds to draw a sketch. On the other hand, only the class labels are provided without any other finegrained annotations (i.e., attributes, groupings), thus, for single-modal tasks, QuickDraw dataset is mainly used for sketch recognition or pre-training. For multi-modal sketch tasks, most existing methods are proposed in supervised manner. As discussed in Section 4.2.2, the cross-modal pairing information only provides weak supervision signals to train the cross-modal model, due to the large domain gap, e.g., sketch-photo gap, sketch-3D gap. Hence, fine-grained/local annotations can bring more opportunities to study the subtle crossmodal alignment and matching, e.g., attribute annotations provided in SketchX Shoe dataset BIB001 . The main application of sketch research is the sketchbased touch-screen interaction that should be capable for any possible users with various drawing styles and abilities. Hence, for both single-modal and multi-modal sketch tasks, sketches selected from any drawing styles and abilities should be studied. However, all the existing sketch datasets are mainly selected by crowd-sourcing and online game. These two approaches are corresponding to two extreme cases: (i) the crowd-sourcing participators draw every stroke deliberately. (ii) the gamers draw too casually, i.e., QuickDraw. This led to the selected sketches are either excessively well-drawn or messily poorly-drawn. Therefore, both single-modal and multi-modal free-hand sketch related tasks need more diverse sketches.