Unpaired image-to-image translation among category domains has achieved remarkable success in past decades. Recent studies mainly focus on two challenges. For one thing, such translation is inherently multimodal due to variations of domain-specific information (e.g., the domain of house cat has multiple fine-grained subcategories). For another, existing multimodal approaches have limitations in handling more than two domains, i.e. they have to independently build one model for every pair of domains. To address these problems, we propose the Hierarchical Image-to-image Translation (HIT) method which jointly formulates the multimodal and multi-domain problem in a semantic hierarchy structure, and can further control the uncertainty of multimodal. Specifically, we regard the domain-specific variations as the result of the multi-granularity property of domains, and one can control the granularity of the multimodal translation by dividing a domain with large variations into multiple subdomains which capture local and fine-grained variations. With the assumption of Gaussian prior, variations of domains are modeled in a common space such that translations can further be done among multiple domains within one model. To learn such complicated space, we propose to leverage the inclusion relation among domains to constrain distributions of parent and children to be nested. Experiments on several datasets validate the promising results and competitive performance against state-of-the-arts. Image-to-image translation is the process of mapping images from one domain to another, during which changing the domain-specific aspect and preserving the domain-irrelevant information. It has wide applications in computer vision and computer graphics Isola et al. (2017) ; Ledig et al. (2017) ; Zhu et al. (2017a) ; Liu et al. (2017) ; such as mapping photographs to edges/segments, colorization, super-resolution, inpainting, attribute and category transfer, style transfer, etc. In this work, we focus on the task of attribute and category transfer, i.e. a set of images sharing the same attribute or category label is defined as a domain 1 . Such task has achieved significant development and impressive results in terms of image quality in recent years, benefiting from the improvement of generative adversarial nets (GANs) Goodfellow et al. (2014) ; Mirza & Osindero (2014) . Representative methods include pix2pix Isola et al. (2017) , UNIT Liu et al. (2017) , CycleGAN Zhu et al. (2017a) , DiscoGAN Kim et al. (2017) , DualGAN Kim et al. (2017) and DTN Taigman et al. (2017) . More recently the study of this task mainly focus on two challenges. The first is the ability of involving translation among several domains into one model. It is quite a practical need for users. Using most methods, we have to train a separate model for each pair of domains, which is obviously inefficient. To deal with such problem, StarGAN Choi et al. (2018) leverages one generator to transform an image to any domain by taking both the image and the target domain label as conditional input supervised by an auxiliary domain classifier. Another challenge is the multimodal problem, which is early addressed by BicycleGAN Zhu et al. (2017b) . Most techniques including aforementioned StarGAN can only give a single determinate output in target domain given an image from source domain. However, for many translation task, the mapping is naturally multimodal. As shown in Fig.1 , a cat could have many possible appearances such as being a Husky, a Samoyed or other dogs when translated to the dog domain. To address Figure 1: An illustration of a hierarchy structure and the distribution relationship in a 2D space among categories in such hierarchy. Multi-domain translation is shown in the horizontal direction (blue dashed arrow) while multimodal translation is indicated in the vertical direction (red dashed arrow). Since one child category is a special case of its parent, in the distribution space it is a conditional distribution of its parent, leading to the nested relationship between them. this issue, recent works including BicycleGAN Zhu et al. (2017b) , MUNIT Huang et al. (2018) and DRIT Lee et al. (2018) model a continuous and multivariant distribution independently for each domain to represent the variations of domain-specific information, and they have achieved diverse and high-quality results for several two-domain translation tasks. In this paper , we aim at involving the abilities of both multi-domain and multimodal translation into one model. As shown in Fig.1 , it is noted that categories have the natural hierarchical relationships. For instance, the cat, dog and bird are three special children of the animal category since they share some common visual attributes. Furthermore, in the dog domain, some samples are named as husky and some of them are called samoyed due to the appearance variations of being the dog. Of course, one can continue to divide the husky to be finer-grained categories based on the variations of certain visual attributes. Such hierarchical relationships widely exist among categories in real world since it is a natural way for our human to understand objects according to our needs in that time. We go back to the image translation task, the multi-domain and multimodal issues can be understood from horizontal and vertical views respectively. From the horizontal view as the blue dashed arrow indicates, multi-domain translation is the transformation in a flat level among categories. From the vertical view as the red dashed arrow indicates, multimodal translation further considers variations within target category, i.e. the multimodal issue is actually due to the multi-granularity property of categories. In the extreme case, every instance is a variation mode of the domain-specific information. Inspired by these observations, we propose a Hierarchical Image-to-image Translation (HIT) method which translates object images among both multiple category domains in a same hierarchy level and their children domains. To this end, our method models the variations of all domains in forms of multiple continuous and multivariant Gaussian distributions in a common space. This is different from previous methods which model the same Gaussian distribution for two domains in independent spaces and thus can not work with only one generator. Due to the hierarchical relationships among domains, distribution of a child domain is the conditional one of its parent domain. Take such principle into consideration, distributions of domains should be nested between a parent and its children, as a 2D illustration shown in Fig.1 . To effectively supervise the learning of such distributions space, we further improve the traditional conditional GAN framework to possess the hierarchical discriminability via a hierarchical classifier. Experiments on several categories and attributes datasets validate the competitive performance of HIT against state-of-the-arts. In this paper we propose the Hierarchical Image-to-image Translation (HIT) method which incorporates multi-domain and multimodal translation into one model. Experiments on three datasets especially on CelebA show that the proposed method can well achieve such granularity controlled translation objectives, i.e. the variation modes of outputs can be specified owe to the nested distributions. However, current work has a limitation, i.e. the assumption of single Gaussian for each category domain. On one hand, though Gaussian distribution prior is a good approximation for many data, it may not be applicable when scale of available training data is small but variations within domain are large such as the used hierarchical data on ImageNet and ShapeNet in this paper. On the other hand, the parent distributions should be mixture of Gaussians given multiple single Gaussians of its children. This issue would lead to sparse sampling around the centers of parent distributions and poor nested results if samples are not enough to fulfill the whole space. We have made efforts to the idea of mixture of Gaussians and found that it is hard to compute the KL divergence between two mixture of Gaussians which does not have an analytical solution. Besides, the re-parameterize trick for distribution sampling during SGD optimization can not be transferred to the case of mixture of Gaussians. A better assumption to realize the nested relationships among parent-children distributions is a promising direction for our future research.
Granularity controled multi-domain and multimodal image to image translation method
{"src": "scitldr-AIC_train", "paper_id": "HkljrTEKvr"}