Model fine-tuning.  To train a GAN model of a new dataset, researchers have fine-tuned the weights of a pre-trained generator and discriminator pair using transfer learning [1]}, [2]}. The fine-tuning can improve upon training from scratch [3]}, but it can also easily overfit on the new training data. To avoid overfitting, several groups propose to limit the changes in model weights: Batch Statistic Adaptation preserves all weights except batch statistics [4]}; Freeze-D freezes layers of the discriminator [5]}; AdaFM preserves generator layers [6]}; MineGAN keeps weights while altering latent-sampling [7]}; and Elastic Weight Consolidation [8]} preserves weights based on Fisher information [9]}. The other technique is data augmentation, which has been proven effective for small-scale datasets [10]}, [11]}, [12]}, [13]}. Different from the above works, our goal is not to fine-tune weights to learn a model of sketches. Instead, we aim to enable a user to create a new model of realistic images that builds upon the color, texture, and details of a pre-trained GAN, guided by the user-specified object shapes and poses from sketches.
