We propose a fully-convolutional conditional generative model, the latent transformation neural network (LTNN), capable of view synthesis using a light-weight neural network suited for real-time applications. In contrast to existing conditional
 generative models which incorporate conditioning information via concatenation, we introduce a dedicated network component, the conditional transformation unit (CTU), designed to learn the latent space transformations corresponding to specified target views. In addition, a consistency loss term is defined to guide the network toward learning the desired latent space mappings, a task-divided decoder is constructed to refine the quality of generated views, and an adaptive discriminator is introduced to improve the adversarial training process. The generality of the proposed methodology is demonstrated on a collection of three diverse tasks: multi-view reconstruction on real hand depth images, view synthesis of real and synthetic faces, and the rotation of rigid objects. The proposed model is shown to exceed state-of-the-art results in each category while simultaneously achieving a reduction in the computational demand required for inference by 30% on average. Generative models have been shown to provide effective frameworks for representing complex, structured datasets and generating realistic samples from underlying data distributions BID8 . This concept has also been extended to form conditional models capable of sampling from conditional distributions in order to allow certain properties of the generated data to be controlled or selected BID20 . These generative models are designed to sample from broad classes of the data distribution, however, and are not suitable for inference tasks which require identity preservation of the input data. Models have also been proposed which incorporate encoding components to overcome this by learning to map input data to an associated latent space representation within a generative framework BID18 . The resulting inference models allow for the defining structure/features of inputs to be preserved while specified target properties are adjusted through conditioning BID34 . Conventional conditional models have largely relied on rather simple methods, such as concatenation, for implementing this conditioning process; however, BID21 have shown that utilizing the conditioning information in a less trivial, more methodical manner has the potential to significantly improve the performance of conditional generative models. In this work, we provide a general framework for effectively performing inference with conditional generative models by strategically controlling the interaction between conditioning information and latent representations within a generative inference model.In this framework, a conditional transformation unit (CTU), Φ, is introduced to provide a means for navigating the underlying manifold structure of the latent space. The CTU is realized in the form of a collection of convolutional layers which are designed to approximate the latent space operators defined by mapping encoded inputs to the encoded representations of specified targets (see FIG7 ). This is enforced by introducing a consistency loss term to guide the CTU mappings during training. In addition, a conditional discriminator unit (CDU), Ψ, also realized as a collection of convolutional layers, is included in the network's discriminator. This CDU is designed to improve the network's ability to identify and eliminate transformation specific artifacts in the network's predictions.The network has also been equipped with RGB balance parameters consisting of three values {θ R , θ G , θ B } designed to give the network the ability to quickly adjust the global color balance of FIG7 : The conditional transformation unit Φ constructs a collection of mappings {Φ k } in the latent space which produce high-level attribute changes to the decoded outputs. Conditioning information is used to select the appropriate convolutional weights ω k for the specified transformation; the encoding l x of the original input image x is transformed to l y k = Φ k (l x ) = conv(l x , ω k ) and provides an approximation to the encoding l y k of the attribute-modified target image y k .the images it produces to better align with that of the true data distribution. In this way, the network is easily able to remove unnatural hues and focus on estimating local pixel values by adjusting the three RGB parameters rather than correcting each pixel individually. In addition, we introduce a novel estimation strategy for efficiently learning shape and color properties simultaneously; a task-divided decoder is designed to produce a coarse pixel-value map along with a refinement map in order to split the network's overall task into distinct, dedicated network components. In this work, we have introduced an effective, general framework for incorporating conditioning information into inference-based generative models. We have proposed a modular approach to incorporating conditioning information using CTUs and a consistency loss term, defined an efficient task-divided decoder setup for deconstructing the data generation process into managable subtasks, and shown that a context-aware discriminator can be used to improve the performance of the adversarial training process. The performance of this framework has been assessed on a diverse range of tasks and shown to outperform state-of-the-art methods. At the bottle-neck between the encoder and decoder, a conditional transformation unit (CTU) is applied to map the 2×2 latent features directly to the transformed 2×2 latent features on the right. This CTU is implemented as a convolutional layer with filter weights selected based on the conditioning information provided to the network. The noise vector z ∈ R 4 from normal distribution N (0, 1) is concatenated to the transformed 2×2 features and passed to the decoder for the face attributes task only. The 32×32 features near the end of the decoder component are processed by two independent convolution transpose layers: one corresponding to the value estimation map and the other corresponding to the refinement map. The channels of the value estimation map are rescaled by the RGB balance parameters, and the Hadamard product is taken with the refinement map to produce the final network output. For the ALOI data experiment, we have followed the IterGAN Galama & Mensink (2018) encoder and decoder structure, and for the stereo face dataset BID5 experiment, we have added an additional Block v1 layer in the encoder and decoder to utilize the full 128×128×3 resolution images.The encoder incorporates two main block layers, as defined in Figure A. 2, which are designed to provide efficient feature extraction; these blocks follow a similar design to that proposed by , but include dense connections between blocks, as introduced by BID10 . We normalize the output of each network layer using the batch normalization method as described in BID12 . For the decoder, we have opted for a minimalist design, inspired by the work of BID24 . Standard convolutional layers with 3 × 3 filters and same padding are used through the penultimate decoding layer, and transpose convolutional layers with 5 × 5 filters and same padding are used to produce the value-estimation and refinement maps. All parameters have been initialized using the variance scaling initialization method described in BID9 .Our method has been implemented and developed using the TensorFlow framework. The models have been trained using stochastic gradient descent (SGD) and the ADAM optimizer BID15 with initial parameters: learning rate = 0.005, β 1 = 0.9, and β 2 = 0.999 (as defined in the TensorFlow API r1.6 documentation for tf.train.AdamOptimizer). , along with loss function hyperparameters: λ = 0.8, ρ = 0.2, γ = 0.0002, and κ = 0.00005 (as introduced in FORMULA8 ). The discriminator is updated once every two encoder/decoder updates, and one-sided label smoothing BID28 has been used to improve stability of the discriminator training procedure. All datasets have also been normalized to the interval [0, 1] for training. Once the total number of output channels, N out , is specified, the remaining N out − N in output channels are allocated to the non-identity filters (where N in denotes the number of input channels). For the Block v1 layer at the start of the proposed LTNN model, for example, the input is a single grayscale image with N in = 1 channel and the specified number of output channels is N out = 32. One of the 32 channels is accounted for by the identity component, and the remaining 31 channels are the three non-identity filters. When the remaining channel count is not divisible by 3 we allocate the remainder of the output channels to the single 3 × 3 convolutional layer. Swish activation functions are used for each filter, however the filters with multiple convolutional layers (i.e. the right two filters in the Block v1 diagram) do not use activation functions for the intermediate 3 × 3 convolutional layers (i.e. those after the 1 × 1 layers and before the final 3 × 3 layers).
We introduce an effective, general framework for incorporating conditioning information into inference-based generative models.
{"src": "scitldr-AIC_train", "paper_id": "S1g6zn09tm"}