We exploit a recently derived inversion scheme for arbitrary deep neural networks to develop a new semi-supervised learning framework that applies to a wide range of systems and problems.  
 The approach reaches current state-of-the-art methods on MNIST and provides reasonable performances on SVHN and CIFAR10. Through the introduced method, residual networks are for the first time applied to semi-supervised tasks. Experiments with one-dimensional signals highlight the generality of the method. Importantly, our approach is simple, efficient, and requires no change in the deep network architecture. Deep neural networks (DNNs) have made great strides recently in a wide range of difficult machine perception tasks. They consist of parametric functionals f Θ with internal parameters Θ. However, those systems are still trained in a fully supervised fashion using a large set of labeled data, which is tedious and costly to acquire.Semi-supervised learning relaxes this requirement by leaning Θ based on two datasets: a labeled set D of N training data pairs and an unlabeled set D u of N u training inputs. Unlabeled training data is useful for learning as unlabeled inputs provide information on the statistical distribution of the data that can both guide the learning required to classify the supervised dataset and characterize the unlabeled samples in D u hence improve generalization. Limited progress has been made on semi-supervised learning algorithms for DNNs BID15 ; BID16 BID14 , but today's methods suffer from a range of drawbacks, including training instability, lack of topology generalization, and computational complexity.In this paper, we take two steps forward in semi-supervised learning for DNNs. First, we introduce an universal methodology to equip any deep neural net with an inverse that enables input reconstruction. Second, we introduce a new semi-supervised learning approach whose loss function features an additional term based on this aforementioned inverse guiding weight updates such that information contained in unlabeled data are incorporated into the learning process. Our key insight is that the defined and general inverse function can be easily derived and computed; thus for unlabeled data points we can both compute and minimize the error between the input signal and the estimate provided by applying the inverse function to the network output without extra cost or change in the used model. The simplicity of this approach, coupled with its universal applicability promise to significantly advance the purview of semi-supervised and unsupervised learning. We have presented a well-justified inversion scheme for deep neural networks with an application to semi-supervised learning. By demonstrating the ability of the method to best current state-of-theart results on MNIST with different possible topologies support the portability of the technique as well as its potential. These results open up many questions in this yet undeveloped area of DNN inversion, input reconstruction, and their impact on learning and stability.Among the possible extensions, one can develop the reconstruction loss into a per-layer reconstruction loss. Doing so, there is the possibility to weight each layer penalty bringing flexibility as well as meaningful reconstruction. Define the per layer loss as DISPLAYFORM0 with DISPLAYFORM1 Doing so, one can adopt a strategy in favor of high reconstruction objective for inner layers, close to the final latent representation z (L) in order to lessen the reconstruction cost for layers closer to the input X n . In fact, inputs of standard dataset are usually noisy, with background, and the object of interest only contains a small energy with respect to the total energy of X n . Another extension would be to update the weighting while performing learning. Hence, if we denote by t the position in time such as the current epoch or batch, we now have the previous loss becoming DISPLAYFORM2 (13) One approach would be to impose some deterministic policy based on heuristic such as favoring reconstruction at the beginning to then switch to classification and entropy minimization. Finer approaches could rely on explicit optimization schemes for those coefficients. One way to perform this, would be to optimize the loss weighting coefficients α, β, γ ( ) after each batch or epoch by backpropagation on the updates weights. Define DISPLAYFORM3 as a generic iterative update based on a given policy such as gradient descent. One can thus adopt the following update strategy for the hyper-parameters as DISPLAYFORM4 and so for all hyper-parameters. Another approach would be to use adversarial training to update those hyper-parameters where both update cooperate trying to accelerate learning.EBGAN BID18 ) are GANs where the discriminant network D measures the energy of a given input X. D is formulated such as generated data produce high energy and real data produce lower energy. Same authors propose the use of an auto-encoder to compute such energy function. We plan to replace this autoencoder using our proposed method to reconstruct X and compute the energy; hence D(X) = R(X) and only one-half the parameters will be needed for D.Finally, our approach opens the possibility of performing unsupervised tasks such as clustering. In fact, by setting α = 0, we are in a fully unsupervised framework. Moreover, β can push the mapping f Θ to produce a low-entropy, clustered, representation or rather simply to produce optimal reconstruction. Even in a fully unsupervised and reconstruction case (α = 0, β = 1), the proposed framework is not similar to a deep-autoencoder for two main reasons. First, there is no greedy (per layer) reconstruction loss, only the final output is considered in the reconstruction loss. Second, while in both case there is parameter sharing, in our case there is also "activation" sharing that corresponds to the states (spline) that were used in the forward pass that will also be used for the backward one. In a deep autoencoder, the backward activation states are induced by the backward projection and will most likely not be equal to the forward ones.
We exploit an inversion scheme for arbitrary deep neural networks to develop a new semi-supervised learning framework applicable to many topologies.
{"src": "scitldr-AIC_train", "paper_id": "B1i7ezW0-"}