Sketch based image retrieval and synthesis.  
Retrieving images that resemble a human sketch has been extensively studied, including classic methods [1]}, [2]}, [3]}, [4]} that rely on feature descriptors, as well as more recent deep learning methods [5]}, [6]}, [7]}, [8]}, [9]}. The above sketch-based image retrieval (SBIR) techniques have powered sketch-based 3D modeling systems (e.g., Teddy [10]}) as well as image synthesis systems, including Sketch2Photo [11]} and PhotoSketcher [12]}. These seminal works have further inspired deep learning solutions based on image-to-image translation [13]}, [14]}, [15]} such as Scribbler [16]}, SketchyGAN [17]}, SketchyCOCO [18]}, and sketch-based face and hair editing [19]}, [20]}. Other relevant work includes sketch recognition [21]}, [22]} and sketch generation [23]}. Collectively, the above methods have enabled a novice user to synthesize a single natural photograph. In this work, we would like to employ the same intuitive interface for rewriting a generative model. Once done, the resulting model can produce an infinite number of new samples that resemble the input sketch. The models can be further used for random sampling, latent space interpolation, as well as natural photo editing.
