Auto-encoders are commonly used for unsupervised representation learning and for pre-training deeper neural networks.
 When its activation function is linear and the encoding dimension (width of hidden layer) is smaller than the input dimension, it is well known that auto-encoder is optimized to learn the principal components of the data distribution (Oja1982).
 However, when the activation is nonlinear and when the width is larger than the input dimension (overcomplete), auto-encoder behaves differently from PCA, and in fact is known to perform well empirically for sparse coding problems. 

 We provide a theoretical explanation for this empirically observed phenomenon, when rectified-linear unit (ReLu) is adopted as the activation function and the hidden-layer width is set to be large.
 In this case, we show that, with significant probability, initializing the weight matrix of an auto-encoder by sampling from a spherical Gaussian distribution followed by stochastic gradient descent (SGD) training converges towards the ground-truth representation for a class of sparse dictionary learning models.
 In addition, we can show that, conditioning on convergence, the expected convergence rate is O(1/t), where t is the number of updates.
 Our analysis quantifies how increasing hidden layer width helps the training performance when random initialization is used, and how the norm of network weights influence the speed of SGD convergence. d . An auto-encoder can be decomposed into two parts, encoder and decoder. The encoder can be viewed as a composition function s e • a e : R d → R n ; function a e : R d → R n is defined as a e (x) := W e x + b e with W e ∈ R n×d , b e ∈ R n W e and b e are the network weights and bias associated with the encoder. s e is a coordinate-wise activation function defined as s e (y) j := s(y j ) where s : R → R is typically a nonlinear functionThe decoder takes the output of encoder and maps it back to R d . Let x e := s e (a e (x)). The decoding function, which we denote asx, is defined aŝ DISPLAYFORM0 where (W d , b d ) and s d are the network parameters and the activation function associated with the decoder respectively.Suppose the activation functions are fixed before training. One can viewx as a reconstruction of the original signal/data using the hidden representation parameterized by (W e , b e ) and (W d , b d ). The goal of training an auto-encoder is to learn the "right" network parameters, (W e , b e , W d , b d ), so that x has low reconstruction error.Weight tying A folklore knowledge when training auto-encoders is that, it usually works better if one sets W d = W T e . This trick is called "weight tying", which is viewed as a trick of regularization, since it reduces the total number of free parameters. With tied weights, the classical auto-encoder is simplified asx(s e (a e (x))) = s d (W T s e (W x + b e ) + b d )In the rest of the manuscript, we focus on weight-tied auto-encoder with the following specific architecture:x W,b (x) = W T s ReLu (a(x )) = W T s ReLu (W x + b) with s ReLu ( y) i := max{0, y i }Here we abuse notation to usex W,b to denote the encoder-decoder function parametrized by weights W and bias b. In the deep learning community, s ReLu is commonly referred to as the rectified-linear (ReLu) activation.Reconstruction error A classic measure of reconstruction error used by auto-encoders is the expected squared loss. Assuming that the data fed to the auto-encoder is i.i.d distributed according to an unknown distribution, i.e., x ∼ p(x ), the population expected squared loss is defined as DISPLAYFORM1 Learning a "good representation" thus translates to adjusting the parameters (W, b) to minimize the squared loss function. The implicit hope is that the squared loss will provide information about what is a good representation. In other words, we have a certain level of belief that the squared loss characterizes what kind of network parameters are close to the parameters of the latent distribution p(x) . This unwarranted belief leads to two natural questions that motivated our theoretical investigation:• Does the global minimum (or any of global minima, if more than one) of L(W, b) correspond to the latent model parameters of distribution p(x)?• From an optimization perspective, since L(W, b) is non-convex in W and is shown to have exponentially many local minima Safran & Shamir (2016) , one would expect a local algorithm like stochastic gradient descent, which is the go-to algorithm in practice for optimizing L(W, b), to be stuck in local minima and only find sub-optimal solutions. Then how should we explain the practical observation that auto-encoders trained with SGD often yield good representation?Stochastic-gradient based training Stochastic gradient descent (SGD) is a scalable variant of gradient descent commonly used in deep learning. At every time step t, the algorithm evaluates a stochastic gradient g(·) of the population loss function with respect to the network parameters using back propagation by sampling one or a mini-batch of data points. The weight and bias update has the following generic form DISPLAYFORM2 where η t w and η t b are the learning rates for updating W and b respectively, typically set to be a small number or a decaying function of time t. The unbiased gradient estimate g(W t ) and g(b t ) can be obtained by differentiating the empirical loss function defined on a single or a mini-batch of size m, Then the stochastic or mini-batch gradient descent update can be written as DISPLAYFORM3 DISPLAYFORM4 n (width of hidden layer)Max-norm regularization A common trick called "max-norm regularization" Srivastava et al. (2014) or "weight clipping" is used in training deep neural networks. 1 In particular, after each step of stochastic gradient descent, the updated weights is forced to satisfy DISPLAYFORM5 for some constant c. This means the row norm of the weights can never exceed the prefixed constant c. In practice, whenever W i, 2 > c, the max-norm constraint is enforced by projecting the weights back to a ball of radius c.
theoretical analysis of nonlinear wide autoencoder
{"src": "scitldr-AIC_train", "paper_id": "HyiRazbRb"}