Generating complex discrete distributions remains as one of the challenging problems in machine learning. Existing techniques for generating complex distributions with high degrees of freedom depend on standard generative models like Generative Adversarial Networks (GAN), Wasserstein GAN, and associated variations. Such models are based on an optimization involving the distance between two continuous distributions. We introduce a Discrete Wasserstein GAN (DWGAN) model which is based on a dual formulation of the Wasserstein distance between two discrete distributions. We derive a novel training algorithm and corresponding network architecture based on the formulation. Experimental results are provided for both synthetic discrete data, and real discretized data from MNIST handwritten digits. Generative Adversarial Networks (GAN) BID3 have gained significant attention in the field of machine learning. The goal of GAN models is to learn how to generate data based on a collection of training samples. The GAN provides a unique training procedure by treating the learning optimization as a two player game between a generator network and discriminator network. Since the learning process involves optimization over two different networks simultaneously, the GAN is hard to train, often times unstable BID11 . Newly developed models such as the Wasserstein GAN aim to improve the training process by leveraging the Wasserstein distance in optimization, as opposed to the Kullback-Leibler or Jensen-Shannon divergences utilized by the original GAN.A source of interest in generative models arises from natural language processing. In natural language applications, a generative model is necessary to learn complex distributions of text documents. Although both the GAN and Wasserstein GAN approximate a distance between two continuous distributions, and use a continuous sample distance, prior research efforts BID4 BID12 BID10 have applied the models to discrete probability distributions advocating for a few modifications. However, using a continuous sample distance for the discrete case may lead to discrepancies. More precisely, as will be demonstrated via explicit examples, a small continuous distance does not necessarily imply a small discrete distance. This observation has potentially serious ramifications for generating accurate natural language text and sentences using GAN models.To address the above issues, we propose a Discrete Wasserstein GAN (DWGAN) which is directly based on a dual formulation of the Wasserstein distance between two discrete distributions. A principal challenge is to enforce the dual constraints in the corresponding optimization. We derive a novel training algorithm and corresponding network architecture as one possible solution. We proposed the Discrete Wasserstein GAN (DWGAN) which approximates the Wasserstein distance between two discrete distributions. We derived a novel training algorithm and corresponding network architecture for a dual formulation to the problem, and presented promising experimental results. Our future work focuses on exploring techniques to improve the stability of the training process, and applying our model to other datasets such as for natural language processing.
We propose a Discrete Wasserstein GAN (DWGAN) model which is based on a dual formulation of the Wasserstein distance between two discrete distributions.
{"src": "scitldr-AIC_train", "paper_id": "Bkv76ilDz"}