We propose a non-adversarial feature matching-based approach to train generative models. Our approach, Generative Feature Matching Networks (GFMN), leverages pretrained neural networks such as autoencoders and ConvNet classifiers to perform feature extraction. We perform an extensive number of experiments with different challenging datasets, including ImageNet. Our experimental results demonstrate that, due to the expressiveness of the features from pretrained ImageNet classifiers, even by just matching first order statistics, our approach can achieve state-of-the-art results for challenging benchmarks such as CIFAR10 and STL10. One of the key research focus in unsupervised learning is the training of generative methods that can model the observed data distribution. Good progress has been made in recent years with the advent of new approaches such as generative adversarial networks (GANs) BID10 and variational autoencoders (VAE) BID19 which use deep neural networks as building blocks. Both methods have advantages and disadvantages, and a significant number of recent works focus on addressing their issues BID20 BID5 . While the main disadvantage of VAEs is the generation of blurred images, the main issue with GANs is the training instability due to the adversarial learning.Feature matching has been explored to improve the stability of GANs BID41 . The key idea in feature matching GANs (FM-GANs) is to use the discriminator network as a feature extractor, and guide the generator to generate data that matches the feature statistics of the real data. Concretely, the objective function of the generator in FM-GAN consists in minimizing the mean squared error of the average features of a minibatch of generated data and a minibatch of real data. The features are extracted from one single layer of the discriminator. FM-GAN is somewhat similar to methods that use maximum mean discrepancy (MMD) BID11 BID12 . However, while in FM-GAN the objective is to match the mean of the extracted features, in MMD-based generative models BID9 , one normally aims to match all the moments of the two distributions using a Gaussian kernel. Although MMD-based generative models have strong theoretical guarantees, these models normally perform much worse than GANs on challenging benchmarks BID23 .In this work, we focus on answering the following research question: can we train effective generative models by performing feature matching on features extracted from a pretrained neural networks? In other words, we would like to know if adversarial training of the feature extractor together with the generator is a requirement for training effective generators. Towards answering this question, we propose Generative Feature Matching Networks (GFMN), a new feature matching-based approach to train generative models that uses features from pretrained neural networks, breaking away from the problematic min/max game completely. Some interesting properties of the proposed method include: (1) the loss function is directly correlated to the generated image quality; (2) mode collapsing is not an issue; (3) the same pretrained feature extractor can be used across different datasets; and (4) both supervised (classifiers) and unsupervised (autoencoder) models can be used as feature extractors.We perform an extensive number of experiments with different challenging datasets, including ILSVRC2012 (henceforth Imagenet) BID37 . We demonstrate that , due to the expressiveness of the features from pretrained Imagenet classifiers, even by just matching first order statistics, our approach can achieve state-of-the-art results for challenging benchmarks such as CIFAR10 and STL10. Moreover, we show that the same feature extractor is effective across different datasets. The main contributions of this work can be summarized as follows: (1) We propose a new effective feature matching-based approach to train generative models that does not use adversarial learning, have stable training and achieves state-of-the-art results; (2) We propose an ADAM-based moving average method that allows effective training with small minibatches; (3) Our extensive quantitative and qualitative experimental results demonstrate that pretrained autoencoders and deep convolutional net (DCNN) classifiers can be effectively used as feature extractors for the purpose of learning generative models. This work is driven towards answering the question of whether one can train effective generative models by performing feature matching on features extracted from pretrained neural networks. The goal is to avoid adversarial training, breaking away from the problematic min/max game completely. According to our experimental results, the answer to our research question is yes. We achieve successful non-adversarial training of generative feature matching networks by introducing different key ingredients: (1) a more robust way to compute the moving average of the mean features by using ADAM optimizer, which allows us to use small minibatches; (2) the use of features from all layers of pretrained neural networks; (3) the use of features from multiple neural networks at the same time (VGG19 + Resnet18); and (4) the initialization of the generator network.Our quantitative results in Tab. 3 show that GFMN achieves better or similar results compared to the state-of-the-art Spectral GAN (SN-GAN) BID30 for both CIFAR10 and STL10. This is an impressive result for a non-adversarial feature matching-based approach that uses pretrained cross-domain feature extractors and has stable training. When compared to other MMD approaches BID23 3.47±.03 GMMN+AE BID23 3.94±.04 VAE BID27 (Berthelot et al., 2017) 5.62 MMD GAN BID23 6.17±.07 MMD dist GAN BID2 6.39±.04 40.2 / -WGAN BID30 6.41±.11 42.6 / -7.57±.10 64.2 MMDrq GAN BID2 6.51±.03 39.9 / -WGAN-GP BID30 6.68±.06 40.2 / -8.42±.13 55.1 / -McGAN 6.97±.10 SN-GANs BID30 7.58±.12 25.5 / -8.79±.14 43.2 / -MoLM-1024 BID36 7.55±.08 25.0 / 20.3 MoLM-1536 BID36 7.90±.10 23.3 / 18.9 BID9 BID23 BID2 BID36 , GFMN presents important distinctions (some of them already listed in Sec. 3) which make it an attractive alternative. Compared to GMMN and GMMN+AE , we can see in TAB2 that GFMN achieves far better results. In Appendix A.10, we also show a qualitative comparison between GFMN and GMMN results. The main reason why GFMN results are significantly better than GMMN is because GFMN uses a strong, robust kernel function (a pretrained DCNN), which, together with our AMA trick, allows the training with small minibatches. On the other hand, the Gaussian kernel used in GMMN requires a very large minibatch size in order to work well, which is impractical due to memory limitations and computational cost. Compared to recent adversarial MMD methods (MMD GAN) BID23 BID2 ) GFMN also presents significantly better results while avoiding the problematic min/max game. GFMN achieves similar results to the Method of Learned Moments (MoLM) BID36 , while using a much smaller number of features to perform matching. The best performing model from BID36 , MoLM-1536, uses around 42 million moments to train the CIFAR10 generator, while our best GFMN model uses around 850 thousand moments/features only, almost 50x less. In other words, MoLM-1536 can be used in large-scale environments only, while GFMN can be used in single GPU environments.One may argue that the best results from GFMN are obtained with feature extractors that were trained in a supervised manner (classifiers). However, there are two important points to note: (1) we use a cross domain feature extractor and do not use labels from the target datasets (CIFAR10, STL10, MNIST, CelebA); (2) since the accuracy of the classifier does not seem to be the most important factor for generating good features (VGG19 classifier produces better features although it is less accurate than Resnet18, see Appendix A.3); we are confident that GFMN will also achieve state-of-the-art results when trained with features from classifiers trained using unsupervised methods such as the one recently proposed by BID4 . In this work, we introduced GFMN, an effective non-adversarial approach to train generative models. GFMNs are demonstrated to achieve state-of-the-art results while avoiding the challenge of defining and training an adversarial discriminator. Our feature extractors can be easily obtained and provide for a robust and stable training of our generators. Some interesting open questions include: what type of feature extractors other than classifiers and auto-encoders are good for GFMN? What architecture designs are better suited for the purpose of feature extraction in GFMN?A APPENDIX
A new non-adversarial feature matching-based approach to train generative models that achieves state-of-the-art results.
{"src": "scitldr-AIC_train", "paper_id": "Syfz6sC9tQ"}