A Survey on Deep Learning in Medical Image Analysis <s> Restricted Boltzmann Machines (RBMs) and <s> Complexity theory of circuits strongly suggests that deep architectures can be much more efficient (sometimes exponentially) than shallow architectures, in terms of computational elements required to represent some functions. Deep multi-layer neural networks have many levels of non-linearities allowing them to compactly represent highly non-linear and highly-varying functions. However, until recently it was not clear how to train such deep networks, since gradient-based optimization starting from random initialization appears to often get stuck in poor solutions. Hinton et al. recently introduced a greedy layer-wise unsupervised learning algorithm for Deep Belief Networks (DBN), a generative model with many layers of hidden causal variables. In the context of the above optimization problem, we study this algorithm empirically and explore variants to better understand its success and extend it to cases where the inputs are continuous or where the structure of the input distribution is not revealing enough about the variable to be predicted in a supervised task. Our experiments also confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization. <s> BIB001 </s> A Survey on Deep Learning in Medical Image Analysis <s> Restricted Boltzmann Machines (RBMs) and <s> We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to ½ everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples. <s> BIB002
Deep Belief Networks (DBNs) RBMs  are a type of Markov Random Field (MRF), constituting an input layer or visible layer x = (x 1 , x 2 , . . . , x N ) and a hidden layer h = (h 1 , h 2 , . . . , h M ) that carries the latent feature representation. The connections between the nodes are bidirectional, so given an input vector x one can obtain the latent feature representation h and also vice versa. As such, the RBM is a generative model, and we can sample from it and generate new data points. In analogy to physical systems, an energy function is defined for a particular state (x, h) of input and hidden units: with c and b bias terms. The probability of the 'state' of the system is defined by passing the energy to an exponential and normalizing: Computing the partition function Z is generally intractable. However, conditional inference in the form of computing h conditioned on v or vice versa is tractable and results in a simple formula: Since the network is symmetric, a similar expression holds for P(x i |h). DBNs BIB001 are essentially SAEs where the AE layers are replaced by RBMs. Training of the individual layers is, again, done in an unsupervised manner. Final fine-tuning is performed by adding a linear classifier to the top layer of the DBN and performing a supervised optimization. 2.6.3. Variational Auto-Encoders and Generative Adverserial Networks Recently, two novel unsupervised architectures were introduced: the variational auto-encoder (VAE)  ) and the generative adversarial network (GAN) BIB002 . There are no peer-reviewed papers applying these methods to medical images yet, but applications in natural images are promising. We will elaborate on their potential in the discussion.