Unsupervised image-to-image translation aims to learn a mapping between several visual domains by using unpaired training pairs. Recent studies have shown remarkable success in image-to-image translation for multiple domains but they suffer from two main limitations: they are either built from several two-domain mappings that are required to be learned independently and/or they generate low-diversity results, a phenomenon known as model collapse. To overcome these limitations, we propose a method named GMM-UNIT based on a content-attribute disentangled representation, where the attribute space is fitted with a GMM. Each GMM component represents a domain, and this simple assumption has two prominent advantages. First, the dimension of the attribute space does not grow linearly with the number of domains, as it is the case in the literature. Second, the continuous domain encoding allows for interpolation between domains and for extrapolation to unseen domains. Additionally, we show how GMM-UNIT can be constrained down to different methods in the literature, meaning that GMM-UNIT is a unifying framework for unsupervised image-to-image translation. Translating images from one domain into another is a challenging task that has significant influence on many real-world applications where data are expensive, or impossible to obtain and to annotate. Image-to-Image translation models have indeed been used to increase the resolution of images (Dong et al., 2014) , fill missing parts (Pathak et al., 2016) , transfer styles (Gatys et al., 2016) , synthesize new images from labels (Liu et al., 2017) , and help domain adaptation (Bousmalis et al., 2017; Murez et al., 2018) . In many of these scenarios, it is desirable to have a model mapping one image to multiple domains, while providing visual diversity (i.e. a day scene ↔ night scene in different seasons). However, the existing models can either map an image to multiple stochastic results in a single domain, or map in the same model multiple domains in a deterministic fashion. In other words, most of the methods in the literature are either multi-domain or multi-modal. Several reasons have hampered a stochastic translation of images to multiple domains. On the one hand, most of the Generative Adversarial Network (GAN) models assume a deterministic mapping (Choi et al., 2018; Pumarola et al., 2018; Zhu et al., 2017a) , thus failing at modelling the correct distribution of the data . On the other hand, approaches based on Variational Auto-Encoders (VAEs) usually assume a shared and common zero-mean unit-variance normally distributed space Zhu et al., 2017b) , limiting to two-domain translations. In this paper, we propose a novel image-to-image translation model that disentangles the visual content from the domain attributes. The attribute latent space is assumed to follow a Gaussian mixture model (GMM), thus naming the method: GMM-UNIT (see Figure 1 ). This simple assumption allows four key properties: mode-diversity thanks to the stochastic nature of the probabilistic latent model, multi-domain translation since the domains are represented as clusters in the same attribute spaces, scalability because the domain-attribute duality allows modeling a very large number of domains without increasing the dimensionality of the attribute space, and few/zero-shot generation since the continuity of the attribute representation allows interpolating between domains and extrapolating to unseen domains with very few or almost no observed data from these domains. The code and models will be made publicly available. : GMM-UNIT working principle. The content is extracted from the input image (left, purple box), while the attribute (turquoise box) can be either sampled (top images) or extracted from a reference image (bottom images). Either way, the generator (blue box) is trained to output realistic images belonging to the domain encoded in the attribute vector. This is possible thanks to the disentangled attribute-content latent representation of GMM-UNIT and the generalisation properties associated to Gaussian mixture modeling. In this paper, we present a novel image-to-image translation model that maps images to multiple domains and provides a stochastic translation. GMM-UNIT disentangles the content of an image from its attributes and represents the attribute space with a GMM, which allows us to have a continuous encoding of domains. This has two main advantages: first, it avoids the linear growth of the dimension of the attribute space with the number of domains. Second, GMM-UNIT allows for interpolation across-domains and the translation of images into previously unseen domains. We conduct extensive experiments in three different tasks, namely two-domain translation, multidomain translation and multi-attribute multi-domain translation. We show that GMM-UNIT achieves quality and diversity superior to state of the art, most of the times with fewer parameters. Future work includes the possibility to thoroughly learn the mean vectors of the GMM from the data and extending the experiments to a higher number of GMM components per domain.
GMM-UNIT is an image-to-image translation model that maps an image to multiple domains in a stochastic fashion.
{"src": "scitldr-AIC_train", "paper_id": "HkeFQgrFDr"}