Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy <s> Wasserstein GAN (WGAN) <s> We investigate the properties of a metric between two distributions, the Earth Mover's Distance (EMD), for content-based image retrieval. The EMD is based on the minimal cost that must be paid to transform one distribution into the other, in a precise sense, and was first proposed for certain vision problems by Peleg, Werman, and Rom. For image retrieval, we combine this idea with a representation scheme for distributions that is based on vector quantization. This combination leads to an image comparison framework that often accounts for perceptual similarity better than other previously proposed methods. The EMD is based on a solution to the transportation problem from linear optimization, for which efficient algorithms are available, and also allows naturally for partial matching. It is more robust than histogram matching techniques, in that it can operate on variable-length representations of the distributions that avoid quantization and other binning problems typical of histograms. When used to compare distributions with the same overall mass, the EMD is a true metric. In this paper we focus on applications to color and texture, and we compare the retrieval performance of the EMD with that of other distances. <s> BIB001 </s> Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy <s> Wasserstein GAN (WGAN) <s> Recent advances in deep learning have shown exciting promise in filling large holes in natural images with semantically plausible and context aware details, impacting fundamental image manipulation tasks such as object removal. While these learning-based methods are significantly more effective in capturing high-level features than prior techniques, they can only handle very low-resolution inputs due to memory limitations and difficulty in training. Even for slightly larger images, the inpainted regions would appear blurry and unpleasant boundaries become visible. We propose a multi-scale neural patch synthesis approach based on joint optimization of image content and texture constraints, which not only preserves contextual structures but also produces high-frequency details by matching and adapting patches with the most similar mid-layer feature correlations of a deep classification network. We evaluate our method on the ImageNet and Paris Streetview datasets and achieved state-of-the-art inpainting accuracy. We show our approach produces sharper and more coherent results than prior methods, especially for high-resolution images. <s> BIB002
WGAN [88] has successfully solved the two problems for the original GAN by using the Earth mover (EM) or Wasserstein-1 BIB001 distance as the loss measure for optimization. The EM distance is defined as γ∈ (pr,pg) (a) JS divergence changes with distance. (b) Gradient JS divergence changes with distance. where (p r , p g ) denotes the set of all joint distributions and γ(x, y) whose marginals are p r and p g . Compared with KL and JS divergence, EM is able to reflect distance even when p r and p g do not overlap and it is also continuous and thus able to provide meaningful gradient for training the generator. Figure 18 illustrates the gradient of WGAN comparing to the original GAN. It is noticeable that WGAN has a smooth gradient for training the generator spanning the complete space. However, the infimum in equation (9) is intractable but the creators demonstrate that instead the Wasserstein distance can be estimated as where f w can be realized by D but has some constraints (for details the interested reader can refer to the original work [88] ) and z is the input noise for G. So w here is the parameters in D and D aims to maximize equation BIB002 in order to make the optimization distance equivalent to Wasserstein distance. When D is optimized, equation (9) will become the Wasserstein distance and G aims to minimize it. So the loss for G is An important difference between WGAN and the original GAN is the function of D. The D in the original work is used as a binary classifier but D used in WGAN is to fit the Wasserstein distance, which is a regression task. Thus, the sigmoid in the last layer of D is removed in the WGAN.