Conditional generative adversarial networks (cGAN) have led to large improvements in the task of conditional image generation, which lies at the heart of computer vision. The major focus so far has been on performance improvement, while there has been little effort in making cGAN more robust to noise. The regression (of the generator) might lead to arbitrarily large errors in the output, which makes cGAN unreliable for real-world applications. In this work, we introduce a novel conditional GAN model, called RoCGAN, which leverages structure in the target space of the model to address the issue. Our model augments the generator with an unsupervised pathway, which promotes the outputs of the generator to span the target manifold even in the presence of intense noise. We prove that RoCGAN share similar theoretical properties as GAN and experimentally verify that our model outperforms existing state-of-the-art cGAN architectures by a large margin in a variety of domains including images from natural scenes and faces. Image-to-image translation and more generally conditional image generation lie at the heart of computer vision. Conditional Generative Adversarial Networks (cGAN) (Mirza & Osindero, 2014) have become a dominant approach in the field, e.g. in dense 1 regression (Isola et al., 2017; Pathak et al., 2016; Ledig et al., 2017; BID1 Liu et al., 2017; Miyato & Koyama, 2018; Yu et al., 2018; Tulyakov et al., 2018) . They accept a source signal as input, e.g. prior information in the form of an image or text, and map it to the target signal (image). The mapping of cGAN does not constrain the output to the target manifold, thus the output can be arbitrarily off the target manifold (Vidal et al., 2017) . This is a critical problem both for academic and commercial applications. To utilize cGAN or similar methods as a production technology, we need to study their generalization even in the face of intense noise.Similarly to regression, classification also suffers from sensitivity to noise and lack of output constraints. One notable line of research consists in complementing supervision with unsupervised learning modules. The unsupervised module forms a new pathway that is trained with the same, or different data samples. The unsupervised pathway enables the network to explore the structure that is not present in the labelled training set, while implicitly constraining the output. The addition of the unsupervised module is only required during the training stage and results in no additional computational cost during inference. Rasmus et al. (2015) and Zhang et al. (2016) modified the original bottom-up (encoder) network to include top-down (decoder) modules during training. However, in dense regression both bottom-up and top-down modules exist by default, and such methods are thus not trivial to extend to regression tasks.Motivated by the combination of supervised and unsupervised pathways, we propose a novel conditional GAN which includes implicit constraints in the latent subspaces. We coin this new model 'Robust Conditional GAN' (RoCGAN). In the original cGAN the generator accepts a source signal and maps it to the target domain. In our work, we (implicitly) constrain the decoder to generate samples that span only the target manifold. We replace the original generator, i.e. encoder-decoder, with a two pathway module (see FIG0 ). The first pathway, similarly to the cGAN generator, performs regression while the second is an autoencoder in the target domain (unsupervised pathway). The two pathways share a similar network structure, i.e. each one includes an encoder-decoder network. The weights of the two decoders are shared which promotes the latent representations of the two pathways to be semantically similar. Intuitively, this can be thought of as constraining the output of our dense regression to span the target subspace. The unsupervised pathway enables the utilization of all the samples in the target domain even in the absence of a corresponding input sample. During inference, the unsupervised pathway is no longer required, therefore the testing complexity remains the same as in cGAN. (a) The source signal is embedded into a low-dimensional, latent subspace, which is then mapped to the target subspace. The lack of constraints might result in outcomes that are arbitrarily off the target manifold. (b) On the other hand, in RoCGAN, steps 1b and 2b learn an autoencoder in the target manifold and by sharing the weights of the decoder, we restrict the output of the regression (step 2a). All figures in this work are best viewed in color.In the following sections, we introduce our novel RoCGAN and study their (theoretical) properties. We prove that RoCGAN share similar theoretical properties with the original GAN, i.e. convergence and optimal discriminator. An experiment with synthetic data is designed to visualize the target subspaces and assess our intuition. We experimentally scrutinize the sensitivity of the hyper-parameters and evaluate our model in the face of intense noise. Moreover, thorough experimentation with both images from natural scenes and human faces is conducted in two different tasks. We compare our model with both the state-of-the-art cGAN and the recent method of Rick Chang et al. (2017) . The experimental results demonstrate that RoCGAN outperform the baseline by a large margin in all cases.Our contributions can be summarized as following:• We introduce RoCGAN that leverages structure in the target space. The goal is to promote robustness in dense regression tasks.• We scrutinize the model performance under (extreme) noise and adversarial perturbations.To the authors' knowledge, this robustness analysis has not been studied previously for dense regression.• We conduct a thorough experimental analysis for two different tasks. We outline how RoCGAN can be used in a semi-supervised learning task, how it performs with lateral connections from encoder to decoder.Notation: Given a set of N samples, s (n) denotes the n th conditional label, e.g. a prior image; y (n) denotes the respective target image. Unless explicitly mentioned otherwise || · || will declare an 1 norm. The symbols L * define loss terms, while λ * denote regularization hyper-parameters optimized on the validation set. We introduce the Robust Conditional GAN (RoCGAN) model, a new conditional GAN capable of leveraging unsupervised data to learn better latent representations, even in the face of large amount of noise. RoCGAN's generator is composed of two pathways. The first pathway (reg pathway), performs the regression from the source to the target domain. The new, added pathway (AE pathway) is an autoencoder in the target domain. By adding weight sharing between the two decoders, we implicitly constrain the reg pathway to output images that span the target manifold. In this following sections (of the appendix) we include additional insights, a theoretical analysis along with additional experiments. The sections are organized as following:• In sec. B we validate our intuition for the RoCGAN constraints through the linear equivalent.• A theoretical analysis is provided in sec. C.• We implement different networks in sec. D to assess whether the performance gain can be attributed to a single architecture.• An ablation study is conducted in sec. E comparing the hyper-parameter sensitivity and the robustness in the face of extreme noise.The FIG3 , 7, 8 include all the outputs of the synthetic experiment of the main paper. As a reminder, the output vector is [x + 2y + 4, e x + 1, x + y + 3, x + 2] with x, y ∈ [−1, 1].
We introduce a new type of conditional GAN, which aims to leverage structure in the target space of the generator. We augment the generator with a new, unsupervised pathway to learn the target structure. 
{"src": "scitldr-AIC_train", "paper_id": "Byg0DsCqYQ"}