Few-Shot Adaptation of Generative Adversarial Networks

Generative Adversarial Networks (GANs) have shown remarkable performance in image synthesis tasks, but typically require a large number of training samples to achieve high-quality synthesis. This paper proposes a simple and effective method, Few-Shot GAN (FSGAN), for adapting GANs in few-shot settings (less than 100 images). FSGAN repurposes component analysis techniques and learns to adapt the singular values of the pre-trained weights while freezing the corresponding singular vectors. This provides a highly expressive parameter space for adaptation while constraining changes to the pretrained weights. We validate our method in a challenging few-shot setting of 5-100 images in the target domain. We show that our method has significant visual quality gains compared with existing GAN adaptation methods. We report qualitative and quantitative results showing the effectiveness of our method. We additionally highlight a problem for few-shot synthesis in the standard quantitative metric used by data-efficient image synthesis works. Code and additional results are available at http://e-271.github.io/few-shot-gan.


INTRODUCTION
Recent years have witnessed rapid progress in Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) with improvements in architecture designs (Radford et al., 2015;Karras et al., 2018;Zhang et al., 2019;Karras et al., 2019a), training techniques (Salimans et al., 2016;Karras et al., 2018;Miyato et al., 2018), and loss functions Gulrajani et al., 2017). Training these models, however, typically requires large, diverse datasets in a target visual domain. While there have been significant advancements in improving training stability (Karras et al., 2018;Miyato et al., 2018), adversarial optimization remains challenging because the optimal solutions lie at saddle points rather than a minimum of a loss function (Yadav et al., 2017). Additionally, GAN-based models may suffer from the inadequate generation of rare modes in the training data because they optimize a mode-seeking loss rather than the mode-covering loss of standard likelihood maximization (Poole et al., 2016). These difficulties of training GANs become even more severe when the number of training examples is scarce. In the low-data regime (e.g., less than 1,000 samples), GANs frequently suffer from memorization or instability, leading to a lack of diversity or poor visual quality.
Several recent efforts have been devoted to improving the sample efficiency of GANs through transfer learning. The most straightforward approaches are finetuning a pre-trained generator and discriminator on the samples in the target domain (Wang et al., 2018;Mo et al., 2020). When the number of training examples is severely limited, however, finetuning the network weights often leads to poor results, particularly when the source and target domains are distant. Instead of finetuning the entire network weights, the method in (Noguchi & Harada, 2019) focuses on adapting batch norm statistics, constraining the optimization problem to a smaller set of parameters. The authors report that this method achieves better results using MLE-based optimization but fails for GAN-based optimization. Although their quality outperforms GAN-based methods in the low-shot setting, the images are blurry and lack details due to maximum likelihood optimization. Invertible flow-based models have shown promising results in data-efficient adaptation (Gambardella et al., 2019), but require compute-and memory-intensive architectures with high-dimensional latent spaces.
In this paper, we propose a method for adapting a pre-trained GAN to generate novel, high-quality sample images with a small number of training images from a new target domain (Figure 1). To accomplish this, we restrict the space of trainable parameters to a small number of highly-expressive parameters that modulate orthogonal features of the pre-trained weight space. Our method first 1 arXiv:2010.11943v1 [cs.CV] 22 Oct 2020 Figure 1: Few-shot image generation. Our method generates novel and high-quality samples in a new domain with a small amount of training data. (Top) Diverse random samples from adapting a FFHQ-pretrained StyleGAN2 to toddler images from the CelebA dataset (with only 30 images) using our method. (Bottom) Smooth latent space interpolation between two random seeds shows that our method produces novel samples instead of simply memorizing the 30 images. Please see the supplementary video for more results.
applies singular value decomposition (SVD) to the network weights of a pretrained GAN (generator + discriminator). We then adapts the singular values using GAN optimization on the target few-shot domain, with fixed left/right singular vectors. We show that varying singular values in the weight space corresponds to semantically meaningful changes of the synthesized image while preserving natural structure. Compared with methods that finetune all weights of the GAN (Wang et al., 2018), individual layers (Mo et al., 2020), or only adapt batch norm statistics (Noguchi & Harada, 2019), our method demonstrates higher image quality after adaptation. We additionally highlight problems with the standard evaluation practice in the low-shot GAN setting.

BACKGROUND
Generative Adversarial Networks (GANs) GANs (Goodfellow et al., 2014) use adversarial training to learn a mapping of random noise to the distribution of an image dataset, allowing for sampling of novel images. GANs optimize a competitive objective where a generator G(Z) maximizes the classification error of a discriminator D(X) trained to distinguish real data p(X) from fake data G(Z). The GAN (Goodfellow et al., 2014) objective is expressed formally as: Recent research reformulated this objective to address instability problems Heusel et al., 2017a;Gulrajani et al., 2017). Improved architecture and training has led to remarkable performance in synthesis (Karras et al., 2020;Brock et al., 2019). Compared to pixel-reconstruction losses (Kingma & Welling, 2014;Higgins et al., 2017;Bojanowski et al., 2018) GANs typically produce sharper images, although strong priors over the latent space can offer competitive quality (Razavi et al., 2019). A high-quality generation has relied on large datasets of high-quality images (10K+) that may be expensive or infeasible to collect in many scenarios. Additionally, GANs can suffer from a lack of diversity, even when large training sets are used because the objective does not penalize the absence of outlier modes (Poole et al., 2016). Data-efficient GAN methods are, therefore, of great utility.
Sample-efficient Image Synthesis Sample-efficient image synthesis methods encourage diverse and high-quality generation in the low-data regime, most commonly through pretraining (Wang et al., 2018;Noguchi & Harada, 2019) or simultaneous training (Yamaguchi et al., 2019) on large image datasets. The main differences among these methods lie in the choice of learnable parameters used for adaptation. Examples include adapting all weights of the generator and discriminator Wang et al. (2018), freezing only lower layers of the discriminator Mo et al. (2020), or changing only channelwise batch statistics Noguchi & Harada (2019). Flow-based methods (Gambardella et al., 2019) show promising results in few-shot adaptation, but their architecture is compute-and memory-intensive and requires latent space of the same dimensionality as the data. Our method uses a smaller but more expressive set of parameters ( Figure 2), resulting in more natural adapted samples.
(b) FSGAN singular value adaptation. One-shot Image Re-synthesis. Recent work in one-shot image synthesis has demonstrated highquality and diverse results by modeling the internal distribution of features from a single image without pretraining (Shaham et al., 2019;Shocher et al., 2019). Our work differs as we transfer external knowledge from a pretrained GAN to a new domain and, therefore, can generate drastically more diverse samples.
Singular Value Decomposition (SVD). SVD factorizes any matrix M ∈ R m×n into unitary matrices U ∈ R m×m ,V ∈ R n×n and diagonal matrix Σ such that M = UΣV , where U,V contain the left and right singular vectors respectively and Σ contains the singular values along the diagonal entries. SVD can be interpreted as a decomposition of a linear transformation x − → Mx into three separate transformations: a rotation/reflection U, followed by rescaling Σ, followed be another rotation/reflection V . The transformation defined by the maximum singular value σ 0 = Σ (1,1) and its corresponding normalized singular vectors represent the maximal axis of variation in the matrix M. This interpretation is commonly used in data science for dimensionality reduction via PCA (Kwak, 2008). PCA can be obtained via SVD on a column-normalized matrix (Golub & Reinsch, 1971). SVD is also used for a wide number of other applications, including regularization (Sedghi et al., 2018), and quantification of entanglement (Martyn et al., 2020), and has also been used to build theoretical background for semantic development in neural networks (Saxe et al., 2019). The work most closely related to ours is GANSpace (Härkönen et al., 2020) for image synthesis editing. GANSpace applies PCA within the latent feature space of a pretrained GAN to discover semantically-interpretable directions for image editing in the latent space. In contrast, our work performs SVD on the weight space of a GAN to discover meaningful directions for domain adaptation. Performing SVD on the weight space enables two critical differences between our work and Härkönen et al. (2020): (i) we edit the entire output distribution rather than one image, and (ii) rather than manual editing, we adapt to a new domain.

OVERVIEW
Our goal is to improve GAN finetuning on small image domains by discovering a more effective and constrained parameter space for adapting the pretrained weights. We are inspired by prior work in GAN adaptation showing that constraining the space of trainable parameters can lead to improved performance on target domain (Rebuffi et al., 2017;Mo et al., 2020;Noguchi & Harada, 2019). In contrast to identifying the parameter space within the model architecture, we propose to discover a parameter space based on the pretrained weights. Specifically, we apply singular value decomposition to the pretrained weights and uncover a basis representing orthogonal directions of maximum variance in the weight space. To explore the interpretation of the SVD representation, we visualize the top three singular values of synthesis and style layers of StyleGAN2 (Karras et al., 2020). We observe that varying the singular values corresponds to natural and semantically-meaningful changes in the output image as shown in Figure 3. Changing the singular values can be interpreted as changing the entanglement between orthogonal factors of variation in the data (singular vectors), providing an style 4 conv 8×8 conv 1024×1024 Original 10σ 0 10σ 1 10σ 2 5σ 0 5σ 1 5σ 2 10σ 0 10σ 1 10σ 2 Figure 3: Effects of singular values. We visualize FSGAN's adaptation space by magnifying the top 3 singular values σ 0 , σ 1 , σ 2 from SVD performed on style and conv layers of a StyleGAN2 (Karras et al., 2019a; 2020) pretrained on FFHQ. In mapping layer 4 (style 4 ), the leading σ s change the age, skin tone, and head pose. In synthesis layer 2 (conv 8×8 ), face dimensions are modified in term of face height/size/width. In synthesis layer 9 (conv 1024×1024 ), the face appearance changes in finer pixel stats such as saturation, contrast, and color balance.
expressive parameterization of the pretrained weights, which we leverage for adaptation as described in the following section.

ADAPTATION PROCEDURE
Our method first performs SVD on both the generator and discriminator of a pretrained GAN and adapts the singular values to a new domain using standard GAN training objectives. A generator layer G ( ) or a discriminator layer D ( ) may consist of either 2D (c in × c out ) fully-connected weights or 4D (k × k × c in × c out ) convolutional filter weights. We apply SVD separately at every layer of the generator G ( ) and discriminator D ( ) . Next, we describe the decomposition process for a single layer of pretrained weights W ( ) 0 . For fully-connected layer W ( ) 0 , we can apply SVD directly on the weight matrix. For 4D convolution weights W ( ) 0 ∈ R k×k×c in ×c out this is not feasible because SVD operates only on a 2D matrix. We therefore reshape the 4D tensor by flattening across the spatial and input feature channels before performing SVD to obtain a 2D matrix W ( ) 0 ∈ R k 2 c in ×c out . Our intuition is that the spatial-feature relationship in the pretrained model should be preserved during the adaptation. We apply SVD over each set of flattened convolutional weights or fully convolution weights to obtain the decomposition: After decomposing the pretrained weights, we perform domain adaptation by freezing pretrained left/right singular vectors in (U 0 ,V 0 ) ( ) and optimizing the singular values Σ = λ Σ 0 using a standard GAN objective to obtain transferred weights ( Figure 2): Effectively, our GAN domain adaptation aims to find a new set of singular values in each layer of a pretrained model so that the generated outputs match the distribution of the target domain.
During forward propagation, we reconstruct weights W Σ using the finetuned singular values at each convolution or fully-connected layer of the generator and discriminator before applying the operation.

TRAINING & INFERENCE
Our experiments use the StyleGAN2 (Karras et al., 2020) training framework, which optimizes a logistic GAN loss (Equation 1) with latent space gradient regularization and a discriminator gradient Train set (10-shot): Figure 4: Problem with FID as a few-shot metric. TGAN (Wang et al., 2018) adaptation from English characters to 10-shot Kannada characters (Bottom) (De Campos et al., 2009). The adaptation process is illustrated by interpolating two random latent vectors at different timesteps (t=20 means 20K images seen during training). We measure FID against a 2K-image Kannada set, from which the 10 images was sampled. The interpolation shows larger timesteps (t) tend to memorize the 10-image training set while yielding lower FID, revealing that FID favors overfitting and is not suitable for the few-shot setting.
penalty. We retrain the singular values Σ for a fixed number of timesteps (20K images or 16K for 5-shot). We find limiting the training time is essential for quality and diversity in the low-shot setting, as longer training often leads to overfitting or quality degradation (examples in Figure 4 & 7). Like Noguchi & Harada (2019), we use the truncation trick (Brock et al., 2019) during inference, but our method works with a less-restrictive truncation parameter of ψ = 0.8, which enables more diversity in the generated images.

EVALUATION IN FEW-SHOT SYNTHESIS
A common adverse outcome in few-shot image generation is overfitting to the target set, such that all generated images look similar to the training data. Evaluation metrics should reflect the diversity of generated images, so that memorization is penalized. The standard evaluation practice used in prior low-shot GAN adaptation work (Wang et al., 2018;Noguchi & Harada, 2019;Mo et al., 2020) is to estimate FID (Heusel et al., 2017b) using a large held-out test set with 1K+ images, from which the low-shot training set was sampled. Standard GAN evaluation typically measures FID with respect to the training set, but in the low-shot setting, this is not desirable because the generator may simply memorize the training set. However, we find that even when measuring FID against a held-out test set, this evaluation still favors overfitted or poor-quality models, as shown in Figure 4. FID between real and fake images is calculated as the Frechet distance between perceptual features p r (X) and p f (Z): where it is assumed features are Gaussian i.e., p f (Z) = N(µ f ,C f ) and p r (X) = N(µ r ,C r ). In the fewshot setting, our n-shot training set T = (x 1 , x 2 , ..., x n ) is sampled from our test set p r (X). Assuming T is chosen at random, its sample mean and varianceμ,σ 2 are unbiased estimators of µ r ,C r . Therefore if the generator memorizes T , its statistics approximate µ r ,C r . This artificially decreases the FID of an overfit model (Figure 4). Consequently, we suggest that FID should be supplemented with additional metrics and extensive qualitative results in the low-shot setting. In high-data settings, a very large number of parameters would be required to memorize the images, so this problem is less likely to occur. Based on these observations, throughout our evaluation, we limit training timesteps rather than select the step with the best FID as we find the latter approach gives more inferior qualitative results.
To address the limitations of standard metrics for GAN evaluation, we also report sharpness (Kumar et al., 2012) and face quality index (Hernandez-Ortega et al., 2019) for human face transfer.

SETTINGS
We adapt a pretrained model to a new target domain using only 5-100 target images, as we focus on scenarios with 1-2 orders fewer number of training samples than standard data-efficient GAN adaptation methods (Wang et al., 2018;Mo et al., 2020;Noguchi & Harada, 2019). As discussed in Section 3.4, we find that the FID score is unsuitable in the low-shot regime due to overfitting bias. However, we still report the FID scores of our experiments for completeness. In addition, we report additional quality metrics and extensive qualitative results.  Datasets. We used FFHQ (Karras et al., 2019a) and LSUN Churches (Yu et al., 2015) pretrained checkpoints from StyleGAN2 (Karras et al., 2019b), and transferred to few-shot single-ID CelebA (30 or 31 images) (Liu et al., 2015), Portraits (5-100 images) (Lee et al., 2018), Anime ID "Rem" (25 images) 2 , and Van Gogh landscapes (25 images) (Zhu et al., 2017). We evaluate FID against a large test set (10K for CelebA) following the evaluation method of Wang et al. (2018). We also evaluate face quality index (Hernandez-Ortega et al., 2019) and image sharpness (Kumar et al., 2012) for face domain adaptation, using 1000 images from each method generated using identical seeds. Full few-shot target sets are shown in Figures 5 & 6, and we will make all few-shot sets available online.  Figure 6: Far-domain adaptation (Photo→Art). Comparing FSGAN with alternative GAN adaptation methods in the photo-to-art setting. (a) FSGAN more effectively alters building layouts and adds landscape in the foreground to match the Van Gogh paintings, maintaining better spatial coherency. (b) FSGAN adopts features from the Portraits dataset (hats, beards, artistic backgrounds), while other methods primarily alter image textures. (c) FSGAN transforms natural hair and facial features to imitate the anime target while retaining spatial consistency. Note the occurrence of pink hair in our generated images, which does not exist in the few-shot target but is visually consistent.

NEAR-DOMAIN ADAPTATION
We first show a near domain transfer setting (adapting FFHQ to single-ID CelebA datatset (Liu et al., 2015)). As both source and target domains contain faces, the pretrained model has useful features for the transfer domain. Figure 5 shows that existing GAN adaptation methods produce artifacts around the eyes/chin and low overall structural consistency. In contrast, our method generates more natural face images with characteristics similar to the training samples (e.g., the head size, position of the faces). Comparing Figure 5 and Table 1 shows that the FID correlates poorly with qualitative evaluation for this setting. In light of this, we report additional metrics of face quality (Hernandez-Ortega et al., 2019) and sharpness (Kumar et al., 2012). On these metrics, our method achieves competitive performance across adaptation settings.

FAR-DOMAIN ADAPTATION
We show far-domain 25-shot transfer, where we define "far" as differing significantly in the distribution of image features such as textures, proportions, and semantics. 1) LSUN Churches→ Van Gogh paintings: The two domains differ in the foreground, building shapes, and textural styles. 2) FFHQ→Art portraits: The main differences between the two domains are low-level styles and facial features. 3) FFHQ→Anime Rem ID: A challenging setting with exaggerated facial proportions and lack of texture details. Figure 6 shows visual comparisons with three state-of-the-art methods. We 5-shot 15-shot 50-shot 100-shot find that the proposed FSGAN can adapt the model to produce more dramatic changes to match the target distributions in terms of semantics, proportions, and textures while maintaining image quality.

N-SHOT SETTINGS
We test the sensitivity of both FSGAN (ours) and FreezeD (Mo et al., 2020) to differing n-shot settings and show the results in Figure 7. We find that FSGAN is more robust to n-shot setting compared to FreezeD. To show this better, we compare two variations of FreezeD. The first FreezeD variant (FD) is limited in timesteps (20K images / 16K on 5-shot) to match FSGAN and the results reported in Figures 5 & 6. Limiting timesteps prevents degradation that occurs at later iterations in the few-shot settings. However, the time-limited FD produces low quality and limited adaptation of textures and semantic features. The second FreezeD variant (FD-FT) is trained for longer (60K images) to demonstrate (1) degradation in fewer n-shot and (2) improvements in quality/adaptation in higher n-shot as seen in (Mo et al., 2020). In contrast, our method (FSGAN) effectively transfers semantic features while preserving quality across all n-shot settings tested in Figure 7. We note variance across n-shot settings for all methods as the data distribution changes.

CONCLUSIONS
We presented Few-Shot GAN, a simple yet effective method for adapting a pre-trained GAN based model to a new target domain where the number of training images is scarce. Our core idea lies in factorizing the weights of convolutional/fully-connected layers in a pretrained model using SVD to identify a semantically meaningful parameter space for adaptation. Our strategy preserves the capability of generating diverse and realistic samples while provides the flexibility for adapting the model to a target domain with few examples. We demonstrate the effectiveness of the proposed method with close-domain and far-domain adaptation experiments and across various n-shot settings. We show favorable results compared with existing data-efficient GAN adaptation methods.