We provide a novel perspective on the forward pass through a block of layers in a deep network. In particular, we show that a forward pass through a standard dropout layer followed by a linear layer and a non-linear activation is equivalent to optimizing a convex objective with a single iteration of a $\tau$-nice Proximal Stochastic Gradient method. We further show that replacing standard Bernoulli dropout with additive dropout is equivalent to optimizing the same convex objective with a variance-reduced proximal method. By expressing both fully-connected and convolutional layers as special cases of a high-order tensor product, we unify the underlying convex optimization problem in the tensor setting and derive a formula for the Lipschitz constant $L$ used to determine the optimal step size of the above proximal methods. We conduct experiments with standard convolutional networks applied to the CIFAR-10 and CIFAR-100 datasets and show that replacing a block of layers with multiple iterations of the corresponding solver, with step size set via $L$, consistently improves classification accuracy. Deep learning has revolutionized computer vision and natural language processing and is increasingly applied throughout science and engineering BID20 . This has motivated the mathematical analysis of various aspects of deep networks, such as the capacity and uniqueness of their representations BID28 BID24 and their global training convergence properties BID10 . However, a complete characterization of deep networks remains elusive. For example, Bernoulli dropout layers are known to improve generalization BID29 , but a thorough theoretical understanding of their behavior remains an open problem. While basic dropout layers have proven to be effective, there are many other types of dropout with various desirable properties BID22 . This raises many questions. Can the fundamental block of layers that consists of a dropout layer followed by a linear transformation and a non-linear activation be further improved for better generalization? Can the choice of dropout layer be made independently from the linear transformation and non-linear activation? Are there systematic ways to propose new types of dropout?We attempt to address some of these questions by establishing a strong connection between the forward pass through a block of layers in a deep network and the solution of convex optimization problems of the following form: DISPLAYFORM0 Note that when f i (a i x) = 1 2 (a i x − y i ) 2 and g(x) = x 2 2 , Eq. (1) is standard ridge regression. When g(x) = x 1 , Eq. (1) has the form of LASSO regression.We show that a block of layers that consists of dropout followed by a linear transformation (fullyconnected or convolutional) and a non-linear activation has close connections to applying stochastic solvers to (1). Interestingly , the choice of the stochastic optimization algorithm gives rise to commonly used dropout layers, such as Bernoulli and additive dropout, and to a family of other types of dropout layers that have not been explored before. As a special case, when the block in question does not include dropout, the stochastic algorithm reduces to a deterministic one.Our contributions can be summarized as follows. (i) We show that a forward pass through a block that consists of Bernoulli dropout followed by a linear transformation and a non-linear activation is equivalent to a single iteration of τ -nice Proximal Stochastic Gradient, Prox-SG BID34 when it is applied to an instance of (1). We provide various conditions on g that recover (either exactly or approximately) common non-linearities used in practice. (ii) We show that the same block with an additive dropout instead of Bernoulli dropout is equivalent to a single iteration of mS2GD BID16 ) -a mini-batching form of variance-reduced SGD BID12 ) -applied to an instance of (1). (iii) By expressing both fully-connected and convolutional layers (referred to as linear throughout) as special cases of a high-order tensor product BID2 , we derive a formula for the Lipschitz constant L of ∇F (x). As a consequence, we can compute the optimal step size for the stochastic solvers that correspond to blocks of layers. We note that concurrent work BID26 used a different analysis strategy to derive an equivalent result for computing the singular values of convolutional layers. (iv) We validate our theoretical analysis experimentally by replacing blocks of layers in standard image classification networks with corresponding solvers and show that this improves the accuracy of the models. We have presented equivalences between layers in deep networks and stochastic solvers, and have shown that this can be leveraged to improve accuracy. The presented relationships open many doors for future work. For instance, our framework shows an intimate relation between a dropout layer and the sampling S from the set [n 1 ] in a stochastic algorithm. As a consequence, one can borrow theory from the stochastic optimization literature to propose new types of dropout layers. For example, consider a serial importance sampling strategy with Prox-SG to solve (5) BID37 BID34 , where serial sampling is the sampling that satisfies Prob (i ∈ S, j ∈ S) = 0. A serial importance sampling S from the set of functions f i ( X ) is the sampling such that Prob DISPLAYFORM0 i.e. each function from the set [n 1 ] is sampled with a probability proportional to the norm of the gradient of the function. This sampling strategy is the optimal serial sampling S that maximizes the rate of convergence solving (5) BID37 . From a deep layer perspective, performing Prox-SG with importance sampling for a single iteration is equivalent to a forward pass through the same block of layers with a new dropout layer. Such a dropout layer will keep each input activation with a non-uniform probability proportional to the norm of the gradient. This is in contrast to BerDropout p where all input activations are kept with an equal probability 1 − p. Other types of dropout arise when considering non-serial importance sampling where |S| = τ > 1.In summary, we have presented equivalences between stochastic solvers on a particular class of convex optimization problems and a forward pass through a dropout layer followed by a linear layer and a non-linear activation. Inspired by these equivalences, we have demonstrated empirically on multiple datasets and network architectures that replacing such network blocks with their corresponding stochastic solvers improves the accuracy of the model. We hope that the presented framework will contribute to a principled understanding of the theory and practice of deep network architectures.A LEAKY RELU AS A PROXIMAL OPERATOR Proof. The proximal operator is defined as Prox g (a) = arg min DISPLAYFORM1 Note that the problem is both convex and smooth. The optimality conditions are given by: DISPLAYFORM2 Since the problem is separable in coordinates, we have: DISPLAYFORM3 The Leaky ReLU is defined as DISPLAYFORM4 which shows that Prox g is a generalized form of the Leaky ReLU with a shift of λ and a slope α = Proof. The proximal operator is defined as Prox g (a) = arg min DISPLAYFORM5 Note that the function g(x) is elementwise separable, convex, and smooth. By equating the gradient to zero and taking the positive solution of the resulting quadratic polynomial, we arrive at the closedform solution: DISPLAYFORM6 where denotes elementwise multiplication. It is easy to see that this operator is close to zero for x i << 0, and close to x i for x i >> 0, with a smooth transition for small |x i |.Note that the function Prox g (a) approximates the activation SoftPlus = log(1 + exp (a)) very well. An illustrative example is shown in FIG2 .
A framework that links deep network layers to stochastic optimization algorithms; can be used to improve model accuracy and inform network design.
{"src": "scitldr-AIC_train", "paper_id": "ryxxCiRqYX"}