To provide principled ways of designing proper Deep Neural Network (DNN) models, it is essential to understand the loss surface of DNNs under realistic assumptions. We introduce interesting aspects for understanding the local minima and overall structure of the loss surface. The parameter domain of the loss surface can be decomposed into regions in which activation values (zero or one for rectified linear units) are consistent. We found that, in each region, the loss surface have properties similar to that of linear neural networks where every local minimum is a global minimum. This means that every differentiable local minimum is the global minimum of the corresponding region. We prove that for a neural network with one hidden layer using rectified linear units under realistic assumptions. There are poor regions that lead to poor local minima, and we explain why such regions exist even in the overparameterized DNNs. Deep Neural Networks (DNNs) have achieved state-of-the-art performances in computer vision, natural language processing, and other areas of machine learning . One of the most promising features of DNNs is its significant expressive power. The expressiveness of DNNs even surpass shallow networks as a network with few layers need exponential number of nodes to have similar expressive power (Telgarsky, 2016) . The DNNs are getting even deeper after the vanishing gradient problem has been solved by using rectified linear units (ReLUs) BID12 . Nowadays, RELU has become the most popular activation function for hidden layers. Leveraging this kind of activation functions, depth of DNNs has increased to more than 100 layers BID7 .Another problem of training DNNs is that parameters can encounter pathological curvatures of the loss surfaces prolonging training time. Some of the pathological curvatures such as narrow valleys would cause unnecessary vibrations. To avoid these obstacles, various optimization methods were introduced (Tieleman & Hinton, 2012; BID9 . These methods utilize the first and second order moments of the gradients to preserve the historical trends. The gradient descent methods also have a problem of getting stuck in a poor local minimum. The poor local minima do exist (Swirszcz et al., 2016) in DNNs, but recent works showed that errors at the local minima are as low as that of global minima with high probability BID4 BID2 BID8 BID14 Soudry & Hoffer, 2017) .In case of linear DNNs in which activation function does not exist, every local minimum is a global minimum and other critical points are saddle points BID8 . Although these beneficial properties do not hold in general DNNs, we conjecture that it holds in each region of parameters where the activation values for each data point are the same as shown in FIG0 . We prove this for a simple network. The activation values of a node can be different between data points as shown in FIG0 , so it is hard to apply proof techniques used for linear DNNs. The whole parameter space is a disjoint union of these regions, so we call it loss surface decomposition.Using the concepts of loss surface decomposition, we explain why poor local minima do exist even in large networks. There are poor local minima where gradient flow disappears when using the ReLU (Swirszcz et al., 2016) . We introduce another kind of poor local minima where the loss is same as that of linear regression. To be more general, we prove that for each local minimum in a network, there exists a local minimum of the same loss in the larger network that is constructed by adding a node to that network. DISPLAYFORM0 T . In each region , activation values are the same. There are six nonempty regions. The parameters on the boundaries hit the non-differentiable point of the rectified linear unit. We conjecture that the loss surface is a disjoint union of activation regions where every local minimum is a subglobal minimum. Using the concept of loss surface decomposition, we studied the existence of poor local minima and experimentally investigated losses of subglobal minima. However, the structure of non-differentiable local minima is not yet well understood yet. These non-differentiable points exist within the boundaries of the activation regions which can be obstacles when using gradient descent methods. Further work is needed to extend knowledge about the local minima, activation regions, their boundaries. Let θ ∈ R A be a differentiable point, so it is not in the boundaries of the activation regions. This implies that w T j x i + b j = 0 for all parameters. Without loss of generality, we assume w T j x i + b j < 0. Then there exist > 0 such that w T j x i + b j + < 0. This implies that small changes in the parameters for any direction does not change the activation region. Since L f (θ) and L g A (θ) are equivalent in the region R A , the local curvatures of these two function around the θ are also the same. Thus, the θ is a local minimum (saddle point) in L f (θ) if and only if it is a local minimum (saddle point) in L g A (θ). DISPLAYFORM0 is a linear transformation of p j , q j , and c, the DISPLAYFORM1 2 is convex in terms of p j , q j , and c. Summation of convex functions is convex, so the lemma holds.A.3 PROOF OF THEOREM 2.5(1) Assume that activation values are not all zeros, and then consider the following Hessian matrix evaluated from v j and b j for some non-zero activation values a ij > 0: DISPLAYFORM2 Let v j = 0 and b j = 0, then two eigenvalues of the Hessian matrix are as follows: DISPLAYFORM3 There exist c > 0 such that g A (x i , θ) > y i for all i. If we choose such c, then DISPLAYFORM4 ∂vj ∂bj > 0 which implies that two eigenvalues are positive and negative. Since the Hessian matrix is not positive semidefinite nor negative semidefinite, the function L g A (θ) is non-convex and non-concave.(2 , 3) We organize some of the gradients as follows: DISPLAYFORM5 We select a critical point θ * where ∇ wj L g A (θ * ) = 0, ∇ vj L g A (θ * ) = 0, ∇ bj L g A (θ * ) = 0, and ∇ c L g A (θ * ) = 0 for all j. Case 1) Assume that ∇ pj L g A (θ * ) = 0 and ∇ qj L g A (θ * ) = 0 for all j. These points are global minima, since ∇ c L g A (θ * ) = 0 and L g A (θ) is convex in terms of p j , q j , and c.Case 2) Assume that there exist j such that ∇ pj L g A (θ DISPLAYFORM6 There exist an element w * in w j such that ∇ vj ∇ w * L g A (θ * ) = 0. Consider a Hessian matrix evaluated from w * and v j . Analogous to the proof of (1), this matrix is not positive semidefinite nor negative semidefinite. Thus θ * is a saddle point.Case 3) Assume that there exist j such that ∇ qj L g A (θ * ) = 0. Since ∇ bj L g A (θ * ) = v j ∇ qj L g A (θ * ) = 0, the v j is zero. Analogous to the Case 2, a Hessian matrix evaluated from b j and v j is not positive semidefinite nor negative semidefinite. Thus θ * is a saddle point.As a result, every critical point is a global minimum or a saddle point. Since L g A (θ) is a differentiable function, every local minimum is a critical point. Thus every local minimum is a global minimum.
The loss surface of neural networks is a disjoint union of regions where every local minimum is a global minimum of the corresponding region.
{"src": "scitldr-AIC_train", "paper_id": "SJDYgPgCZ"}