---
abstract: 'In this paper, we propose a new weight initialization method called [*even initialization*]{} for wide and deep nonlinear neural networks with the ReLU activation function. We prove that no poor local minimum exists in the initial loss landscape in the wide and deep nonlinear neural network initialized by the even initialization method that we propose. Specifically, in the initial loss landscape of such a wide and deep ReLU neural network model, the following four statements hold true: 1) the loss function is non-convex and non-concave; 2) every local minimum is a global minimum; 3) every critical point that is not a global minimum is a saddle point; and 4) bad saddle points exist. We also show that the weight values initialized by the even initialization method are contained in those initialized by both of the (often used) standard initialization and He initialization methods.'
author:
- |
    Tohru Nitta\
    National Institute of Advanced Industrial Science and Technology (AIST), Japan\
    `tohru-nitta@aist.go.jp`\
title: Weight Initialization without Local Minima in Deep Nonlinear Neural Networks
---

Introduction {#intro}
============

Hinton et al. (2006) proposed Deep Belief Networks with a learning algorithm that trains one layer at a time. Since that report, deep neural networks have attracted attention extensively because of their human-like intelligence achieved through learning and generalization. To date, deep neural networks have produced outstanding results in the fields of image processing and speech recognition (Mohamed et al., 2009; Seide et al., 2011; Taigman et al., 2014). Moreover, their scope of application has expanded, for example, to the field of machine translation (Sutskever et al., 2014).

In using deep neural networks, finding a good initialization becomes extremely important to achieve good results. Heuristics have been used for weight initialization of neural networks for a long time. For example, a uniform distribution $
U [ -1/\sqrt{n}, 1/\sqrt{n} ]
$ has been often used where $n$ is the number of neurons in the preceding layer. Pre-training might be regarded as a kind of weight initialization methods, which could avoid local minima and plateaus (Bengio et al., 2007). However, several theoretical researches on weight initialization methods have been progressing in recent years. Glorot and Bengio (2010) derived a theoretically sound uniform distribution $
U [\ - \sqrt{6} / \sqrt{n_i + n_{i+1} }, 
          \sqrt{6} /\sqrt{n_i + n_{i+1} } \ ] 
$ for the weight initialization of deep neural networks with an activation function which is symmetric and linear at the origin. He et al. (2015) proposed a weight initialization method (called [*He initialization*]{} here) with a normal distribution (either $N ( 0, 2 / n_i )$ or $N ( 0, 2 / n_{i+1} )$ ) for the neural networks with the ReLU (Rectified Linear Unit) activation function. The above two initialization methods are driven by experiments to monitor activations and back-propagated gradients during learning.

On the other hand, local minima of deep neural networks has been investigated theoretically in recent years. Local minima cause plateaus which have a strong negative influence on learning in deep neural networks. Dauphin et al. (2014) experimentally investigated the distribution of the critical points of a single-layer MLP and demonstrated that the possibility of existence of local minima with large error (i.e., bad or poor local minima) is very small. Choromanska et al. provided a theoretical justification for the work of Dauphin et al. (2014) on a deep neural network with ReLU units using the spherical spin-glass model under seven assumptions (Choromanska, Henaff, Mathieu, Arous & LeCun, 2015). Choromanska et al. also suggested that discarding the seven unrealistic assumptions remains an important open problem (Choromanska, LeCun & Arous, 2015). Kawaguchi (2016) discarded most of these assumptions and proved that the following four statements for a deep nonlinear neural network with only two out of the seven assumptions: 1) the loss function is non-convex and non-concave, 2) every local minimum is a global minimum, 3) every critical point that is not a global minimum is a saddle point, and 4) bad saddle points exist.

In this paper, we propose a new weight initialization method (called [*even initialization*]{}) for wide and deep nonlinear neural networks with the ReLU activation function: weights are initialized independently and identically according to a probability distribution whose probability density function is even on $[ -1/n, 1/n]$ where $n$ is the number of neurons in the layer. Using the research results presented by Kawaguchi (2016), we prove that no poor local minimum exists in the initial loss landscape in the wide and deep nonlinear neural network initialized by the even initialization method that we propose. We also show that the weight values initialized by the even initialization method are contained in those initialized by both of the (often used) standard initialization and He initialization methods.

Weight Initialization without Local Minima {#initialize_weight}
==========================================

In this section, we propose a weight initialization method for a wide and deep neural network model. There is no local minimum in the initial weight space of the wide and deep neural network initialized by the weight initialization method. The weight values initialized by the weight initialization method are contained in those initialized by both of the standard initialization and He initialization methods. In other words, there exists an interval of initial weights where there is no local minimum in both cases of the standard initialization and He initialization methods.

Kawaguchi Model {#kawa-model}
---------------

This subsection presents a description of the deep nonlinear neural network model analyzed by Kawaguchi (2016) (we call it [*Kawaguchi model*]{} here). We will propose a new weight initialization method for the Kawaguchi model in the subsection \[even\_weight\].

First, we consider the following neuron. The net input $U_n$ to a neuron $n$ is defined as: $U_n = \sum_mW_{nm}I_m$, where $W_{nm}$ represents the weight connecting the neurons $n$ and $m$, $I_m$ represents the input signal from the neuron $m$. It is noteworthy that biases are omitted for the sake of simplicity. The output signal is defined as $\varphi(U_n)$ where $\varphi(u) {\stackrel{\rm def}{=}}\max(0, u)$ for any $u \in {\mbox{\boldmath $R$}}$ and is called [*Rectified Linear Unit*]{} ([*ReLU*]{},  denotes the set of real numbers).

The deep nonlinear neural network described in (Kawaguchi, 2016) consists of such neurons described above (Fig. \[fig1\]). The network has $H+2$ layers ($H$ is the number of hidden layers). The activation function $\psi$ of the neuron in the output layer is linear, i.e., $\psi(u)=u$ for any $u\in{\mbox{\boldmath $R$}}$. For any $0 \leq k \leq H+1$, let $d_k$ denote the number of neurons of the $k$-th layer, that is, the width of the $k$-th layer where the 0-th layer is the input layer and the $(H+1)$-th layer is the output layer. Let $d_x = d_0$ and $d_y = d_{H+1}$ for simplicity.

Let $({\mbox{\boldmath $X$}}, {\mbox{\boldmath $Y$}})$ be the training data where ${\mbox{\boldmath $X$}}\in {\mbox{\boldmath $R$}}^{d_x \times m}$ and ${\mbox{\boldmath $Y$}}\in {\mbox{\boldmath $R$}}^{d_y \times m}$ and where $m$ denotes the number of training patterns. We can rewrite the $m$ training data as $\{ ({\mbox{\boldmath $X$}}_i, {\mbox{\boldmath $Y$}}_i) \}_{i=1}^m$ where ${\mbox{\boldmath $X$}}_i \in {\mbox{\boldmath $R$}}^{d_x}$ is the $i$-th input training pattern and ${\mbox{\boldmath $Y$}}_i \in {\mbox{\boldmath $R$}}^{d_y}$ is the $i$-th output training pattern. Let ${\mbox{\boldmath $W$}}_k$ denote the weight matrix between the $(k-1)$-th layer and the $k$-th layer for any $1 \leq k \leq H+1$. Let ${\mbox{\boldmath $\Theta$}}$ denote the one-dimensional vector which consists of all the weight parameters of the deep nonlinear neural network.

Kawaguchi specifically examined a path from an input neuron to an output neuron of the deep nonlinear neural network (Fig. \[fig2\]), and expressed the actual output of output neuron $j$ of the output layer of the deep nonlinear neural network for the $i$-th input training pattern ${\mbox{\boldmath $X$}}_i \in {\mbox{\boldmath $R$}}^{d_x}$ as $$\begin{aligned}
\label{eqn2-1}
\hat{{\mbox{\boldmath $Y$}}}_i({\mbox{\boldmath $\Theta$}}, {\mbox{\boldmath $X$}}_i)_j = q\sum_{p=1}^\Psi [{\mbox{\boldmath $X$}}_i]_{(j,p)} [{\mbox{\boldmath $Z$}}_i]_{(j,p)} 
\prod_{k=1}^{H+1} w_{(j,p)}^{(k)} \in {\mbox{\boldmath $R$}}\end{aligned}$$ where $\Psi$ represents the total number of paths from the input layer to output neuron $j$, $[{\mbox{\boldmath $X$}}_i]_{(j,p)} \in {\mbox{\boldmath $R$}}$ denotes the component of the $i$-th input training pattern ${\mbox{\boldmath $X$}}_i \in {\mbox{\boldmath $R$}}^{d_x}$ that is used in the $p$-th path to the $j$-th output neuron. Also, $[{\mbox{\boldmath $Z$}}_i]_{(j,p)} \in \{0, 1\}$ represents whether the $p$-th path to the output neuron $j$ is active or not for each training pattern $i$ as a result of ReLU activation. $[{\mbox{\boldmath $Z$}}_i]_{(j,p)} = 1$ means that the path is active, and $[{\mbox{\boldmath $Z$}}_i]_{(j,p)} = 0$ means that the path is inactive. $w_{(j,p)}^{(k)} \in {\mbox{\boldmath $R$}}$ is the component of the weight matrix ${\mbox{\boldmath $W$}}_k \in {\mbox{\boldmath $R$}}^{d_k \times d_{k-1} }$ that is used in the $p$-th path to the output neuron $j$.

The objective of the training is to find the parameters which minimize the error function defined as $$\label{eqn2-2}
L({\mbox{\boldmath $\Theta$}}) = \frac{1}{2} \sum_{i=1}^m 
E_{{\mbox{\boldmath $Z$}}} \Vert \hat{{\mbox{\boldmath $Y$}}}_i({\mbox{\boldmath $\Theta$}}, {\mbox{\boldmath $X$}}_i) - {\mbox{\boldmath $Y$}}_i \Vert^2$$ where $\Vert \cdot \Vert$ is the Euclidean norm, that is, $\Vert {\mbox{\boldmath $u$}}\Vert = \sqrt{u_1^2 + \cdots + u_N^2}$ for a vector ${\mbox{\boldmath $u$}}= (u_1 \cdots u_N)^T \in {\mbox{\boldmath $R$}}^N$, and $\hat{{\mbox{\boldmath $Y$}}}_i({\mbox{\boldmath $\Theta$}}, {\mbox{\boldmath $X$}}_i) \in {\mbox{\boldmath $R$}}^{d_y}$ is the actual output of the output layer of the deep nonlinear neural network for the $i$-th training pattern ${\mbox{\boldmath $X$}}_i$. The expectation in Eq. (\[eqn2-2\]) is made with respect to random vector ${\mbox{\boldmath $Z$}}= \{ [{\mbox{\boldmath $Z$}}_i]_{(j,p)} \}$.

The Kawaguchi model has been analyzed based on the two assumptions.

[**A1p-m**]{} [ *$P([{\mbox{\boldmath $Z$}}_i]_{(j,p)} = 1) = \rho$ for all $i$ and $(j,p)$ where $\rho \in {\mbox{\boldmath $R$}}$ is a constant. That is, $[{\mbox{\boldmath $Z$}}_i]_{(j,p)}$ is a Bernoulli random variable.*]{}

[**A5u-m**]{} [ *${\mbox{\boldmath $Z$}}$ is independent of the input ${\mbox{\boldmath $X$}}$ and the parameter ${\mbox{\boldmath $\Theta$}}$.*]{}

A1p-m and A5u-m are weaker ones of the two assumptions A1p and A5u in (Choromanska, Henaff, Mathieu, Arous & LeCun, 2015), respectively. Assumption A5u-m is used for the proof of Corollary 3.2 presented by Kawaguchi (2016). Strictly speaking, the following assumption A5u-m-1 suffices for the proof instead of the assumption A5u-m described above

[**A5u-m-1**]{} [ *For any $i$ and any $(j,p)$, $[{\mbox{\boldmath $Z$}}_i]_{(j,p)} \in \{0, 1\}$ is independent of the $i$-th input training pattern ${\mbox{\boldmath $X$}}_i \in {\mbox{\boldmath $R$}}^{d_x}$ and the sequence of the weights on the $p$-th path $\{ w_{(j,p)}^{(k)} \in {\mbox{\boldmath $R$}}\}_{k=1}^{H+1}$ where $w_{(j,p)}^{(k)}$ is the weight between the layer $k-1$ and the layer $k$ on the $p$-th path ($k=1, \cdots, H+1$).* ]{}

Actually, according to assumption A5u-m-1, $$\begin{aligned}
\label{eqn2-3}
& & E_{{\mbox{\boldmath $Z$}}} \left[\hat{{\mbox{\boldmath $Y$}}}_i({\mbox{\boldmath $\Theta$}}, {\mbox{\boldmath $X$}}_i)_j \right] \nonumber\\
&=& E_{{\mbox{\boldmath $Z$}}} \left[q\sum_{p=1}^\Psi [{\mbox{\boldmath $X$}}_i]_{(j,p)} [{\mbox{\boldmath $Z$}}_i]_{(j,p)} 
\prod_{k=1}^{H+1} w_{(j,p)}^{(k)} \right] \hspace*{1cm} \mbox{(from Eq.} (\ref{eqn2-1})) \nonumber\\
&=& q\sum_{p=1}^\Psi [{\mbox{\boldmath $X$}}_i]_{(j,p)} E_{{\mbox{\boldmath $Z$}}} \left[ [{\mbox{\boldmath $Z$}}_i]_{(j,p)} \right] 
\prod_{k=1}^{H+1} w_{(j,p)}^{(k)}.  \hspace*{1cm} \mbox{(from the assumption A5u-m-1) }
$$

Even Initialization {#even_weight}
-------------------

This subsection proposes a new weight initialization method for the Kawaguchi model described in the subsection \[kawa-model\]. We assume that the width of the deep nonlinear neural network is sufficiently large, that is, $d_0 (=d_x), d_1, \cdots, d_{H-1}$ are sufficiently large (Fig. \[fig1\]), and that each element of the $i$-th input training pattern ${\mbox{\boldmath $X$}}_i$ takes a value between $-\alpha$ and $\alpha$, that is, ${\mbox{\boldmath $X$}}_i \in {\mbox{\boldmath $I$}}^{d_x}$ for any $i$ where ${\mbox{\boldmath $I$}}=[-\alpha, \alpha]$ and $\alpha$ is a positive real number. Then we propose the following weight initialization scheme.

[**Even Initialization**]{} [ *Elements $\{w_n\}_{n=1}^{d_{k-1}}$ of the weight vector ${\mbox{\boldmath $w$}}= (w_1 \cdots w_{d_{k-1}})^T$ of any hidden neuron in the $k$-th hidden layer are initialized independently and identically according to a probability distribution whose probability density function $f_k: {\mbox{\boldmath $J$}}_k \rightarrow [0, +\infty)$ is an even function where ${\mbox{\boldmath $J$}}_k = [-1/d_{k-1}, 1/d_{k-1}]$, that is, $f_k(-x) = f_k(x)$ for any $x \in {\mbox{\boldmath $J$}}_k$ $( k = 1, \cdots, H$).* ]{} [$\Box$]{}

We designate it as [*even initialization*]{}.

[**Remarks**]{} (a) The expectation of each weight is zero because the probability density function is an even function. (b) The probability distribution in the even initialization can be a normal distribution with zero mean or a uniform distribution. (c) For the weight vectors ${\mbox{\boldmath $w$}}_1=(w_{11} \cdots w_{1M})^T, {\mbox{\boldmath $w$}}_2=(w_{21} \cdots w_{2N})^T$ of any two hidden neurons, the probability distribution which $w_{1k}$ obeys can be different from that which $w_{2l}$ obeys ($k=1, \cdots, M, l=1, \cdots, N$) (Fig. \[fig3\]).

Analysis
--------

This subsection presents an analysis of the initial loss landscape of the Kawaguchi model initialized by the even initialization method described in Section \[even\_weight\].

We prove in the following that the two assumptions A1p-m and A5u-m (A5u-m-1) are satisfied in the Kawaguchi model initialized by the even initialization.

\[thm1\] For any training pattern $i$, any output neuron $j$, and any path $p$ from an input neuron to the output neuron $j$ in the Kawaguchi model initialized by the even initialization method, $$\label{eqn4-1}
P\left([Z_i]_{(j,p)} = 1 \right) = \frac{1}{2^H}$$ where $H$ is the number of hidden layers ($H \geq 1$). [$\Box$]{}

[*Proof*]{}.   Denote by $j_1, \cdots, j_H$ the hidden neurons on path $p$ where $j_k$ is the hidden neuron in the $k$-th hidden layer $(k=1, \cdots, H$) (Fig. \[fig4\]). Then, $$\begin{aligned}
\label{eqn4-2}
P\Big( [Z_i]_{(j,p)} = 1 \Big) 
&=& P\Big( \mbox{Net input to the hidden neuron}\ j_k > 0 \ \ 
(k=1, \cdots, H) \Big) \nonumber\\
&=& P \left( {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0, \left[ \varphi( U_1^{(1)} ) \cdots \varphi(U_{d_1}^{(1)} ) \right] 
{\mbox{\boldmath $w$}}_{j_2} > 0, \right. \nonumber\\ 
& & \left. 
\cdots, \left[ \varphi( U_1^{(H-1)} ) \cdots \varphi(U_{d_{H-1}}^{(H-1)}) \right] 
{\mbox{\boldmath $w$}}_{j_H} > 0 \right)\end{aligned}$$ where ${\mbox{\boldmath $X$}}_i \in {\mbox{\boldmath $I$}}^{d_x}$ is the $i$-th input training pattern, ${\mbox{\boldmath $w$}}_{j_k} \in {\mbox{\boldmath $J$}}_k^{d_{k-1}}$ is the weight vector of the hidden neuron $j_k$ in the hidden layer $k$, and $U_l^{(k)}$ is the net input to the hidden neuron $l$ in the hidden layer $k$ (Fig. \[fig5\]).

We prove by mathematical induction that Eq. (\[eqn4-1\]) holds true.

\[For $H=1$\] This case corresponds to a three-layered neural network. It follows that $$\begin{aligned}
\label{eqn4-3}
P\Big( [Z_i]_{(j,p)} = 1 \Big) &=& P ( {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0 ) \ \ \ 
(\mbox{from Eq.} (\ref{eqn4-2})) \nonumber\\
&=& \frac{1}{2} \end{aligned}$$ where $j_1$ is a hidden neuron on path $p$. We can see below that the last equality of Eq. (\[eqn4-3\]) holds true. For a given input training pattern ${\mbox{\boldmath $X$}}_i \in {\mbox{\boldmath $I$}}^{d_x}$, $\{ {\mbox{\boldmath $w$}}_{j_1} \in {\mbox{\boldmath $J$}}_1^{d_x} \vert \ {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0 \}$ is a half-open hyperspace with a normal vector ${\mbox{\boldmath $X$}}_i$ through the origin in a $d_x$-dimensional Euclidean space (Fig. \[fig6\]). According to the even initialization scheme, the random variables $w_{j_11}, \cdots, w_{j_1d_x}$ which are the components of the weight vector ${\mbox{\boldmath $w$}}_{j1} = (w_{j_11} \cdots w_{j_1d_x})^T$ of the hidden neuron $j_1$ are [*i.i.d.*]{}, and the probability density function $f_1: {\mbox{\boldmath $J$}}_1 \rightarrow [ 0, +\infty )$ of the probability distribution which each weight $w_{j_1k}$ obeys is an even function ($k = 1, \cdots, d_x$). Consequently, $P\left({\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0 \right) = P\left({\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} \leq 0 \right)$, which means $P\left({\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0 \right) = 1/2$.

\[For $H$\] This case corresponding to a deep nonlinear neural network with $H$ hidden layers, we show that if the case of $H-1$ holds, then case $H$ also holds. Assuming that the case of $H-1$ holds, then $$\begin{aligned}
\label{eqn4-4}
& &P\Big( [Z_i]_{(j,p)} = 1 \Big) \nonumber \\
&=& P \left( {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0, \left[ \varphi( U_1^{(1)} ) \cdots \varphi(U_{d_1}^{(1)} ) \right] 
{\mbox{\boldmath $w$}}_{j_2} > 0, \right. \nonumber\\ 
& & \left. \cdots, \left[ \varphi( U_1^{(H-1)} ) \cdots \varphi(U_{d_{H-1}}^{(H-1)}) \right] 
{\mbox{\boldmath $w$}}_{j_H} > 0 \right) \ \ 
(\mbox{from Eq.} (\ref{eqn4-2})) \nonumber\\
&=& P \left( {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0, \left[ \varphi( U_1^{(1)} ) \cdots \varphi(U_{d_1}^{(1)} ) \right] 
{\mbox{\boldmath $w$}}_{j_2} > 0, \right. \nonumber\\ 
& & \left. \cdots, \left[ \varphi( U_1^{(H-1)} ) \cdots \varphi(U_{d_{H-1}}^{(H-1)}) \right] 
{\mbox{\boldmath $w$}}_{j_H} > 0 \ \ \Big\vert \ {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0 \right) 
\cdot P\left( {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0 \right) \nonumber\\
& & + P \left( {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0, \left[ \varphi( U_1^{(1)} ) \cdots \varphi(U_{d_1}^{(1)} ) \right] 
{\mbox{\boldmath $w$}}_{j_2} > 0, \right. \nonumber\\ 
& & \left. \cdots, \left[ \varphi( U_1^{(H-1)} ) \cdots \varphi(U_{d_{H-1}}^{(H-1)}) \right] 
{\mbox{\boldmath $w$}}_{j_H} > 0 \ \  \Big\vert \ {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} \leq 0 \right) 
\cdot P\left( {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} \leq 0 \right) \nonumber\\
&=& P \left( \left[ \varphi( U_1^{(1)} ) \cdots \varphi(U_{d_1}^{(1)} ) \right] 
{\mbox{\boldmath $w$}}_{j_2} > 0, \cdots, \left[ \varphi( U_1^{(H-1)} ) \cdots \varphi(U_{d_{H-1}}^{(H-1)}) \right] 
{\mbox{\boldmath $w$}}_{j_H} > 0 \right. \nonumber\\
& & \hspace*{2cm} \left. \Big\vert \ {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0 \right) 
\cdot P\left( {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0 \right). $$ Here, the first term of the right-hand-side of Eq. (\[eqn4-4\]) represents the probability that the path passing through the $H-1$ hidden neurons $j_2, \cdots, j_H$ for the input training pattern $[ \varphi( U_1^{(1)} ) \cdots \varphi(U_{d_1}^{(1)} ) ]^T \in {\mbox{\boldmath $R$}}^{d_1}$ such that $U_{j_1}^{(1)} = {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0$ is active. Also, for any $1 \leq s \leq d_1$, by denoting ${\mbox{\boldmath $X$}}_i = (x_1 \cdots x_{d_0})^T$ and ${\mbox{\boldmath $w$}}_s = (w_1 \cdots w_{d_0})$ for the sake of simplicity, $\vert \varphi(U_s^{(1)}) \vert \leq \vert U_s^{(1)} \vert 
=\vert {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_s \vert 
\leq \sum_{t=1}^{d_0} \vert x_t \vert \vert w_t \vert 
\leq \alpha \sum_{t=1}^{d_0} (1/d_0) = \alpha$. Hence, according to the assumption of mathematical induction, the first term of the right-hand-side of Eq. (\[eqn4-4\]) is equal to $(1/2)^{H-1}$. In addition, the second term of the right-hand-side of Eq. (\[eqn4-4\]) is equal to 1/2 from Eq.(\[eqn4-3\]). Therefore, $$\begin{aligned}
\label{eqn4-5}
P\Big( [Z_i]_{(j,p)} = 1 \Big) &=& \left(\frac{1}{2}\right)^{H-1} \cdot \frac{1}{2} \nonumber\\
&=& \frac{1}{2^H}, \end{aligned}$$ which means that the case of $H$ indeed holds. Therefore, by mathematical induction, Eq. (\[eqn4-1\]) holds for any $H \geq 1$. Theorem \[thm1\] states that assumption A1p-m holds. Because $$\begin{aligned}
\lim_{H \rightarrow +\infty} P \left( [Z_i]_{(j,p)} = 1 \right) 
&=& \lim_{H \rightarrow +\infty} \frac{1}{2^H} \nonumber\\
&=& 0, \end{aligned}$$ the probability that path $p$ is active decreases exponentially. It converges to zero as the number of hidden layers $H$ increases. The probability that path $p$ is active decreases by half when a hidden layer is added.

\[thm2\] For any training pattern $i$, any output neuron $j$, and any path $p$ from an input neuron to the output neuron $j$ in the Kawaguchi model initialized by the even initialization method, random variable $[Z_i]_{(j,p)}$ is independent of the sequence of weights on path $p$. [$\Box$]{}

[*Proof*]{}.   Take training pattern $i$, output neuron $j$, and path $p$ from an input neuron to output neuron $j$ of the Kawaguchi model arbitrarily and fix them. Denote by $j_0, \cdots, j_H$ the neurons on path $p$ where $j_k$ is the neuron in the $k$-th layer $(k=0, \cdots, H$). Let also $w_{j_1 j_0}, \cdots, w_{j_{H+1} j_H }$ denote $H+1$ weights on path $p$ where $w_{j_{k+1} j_k }$ is the weight between the layer $k$ and the layer $k+1$ on path $p$ ($k=0, \cdots, H$) (Fig. \[fig7\]). Then, for any $\lambda_1, \cdots, \lambda_{H+1} \in {\mbox{\boldmath $R$}}$, $$\begin{aligned}
\label{eqn4-6}
& & \hspace*{-0.3cm} P \left( [Z_i]_{(j,p)} = 1 \ \Big\vert \ w_{j_1 j_0} = \lambda_1, 
\cdots, w_{j_{H+1} j_H } = \lambda_{H+1} \right) \nonumber\\
&=& \hspace*{-0.3cm} P \left( {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0, \left[ \varphi( U_1^{(1)} ) \cdots \varphi(U_{d_1}^{(1)} ) \right] {\mbox{\boldmath $w$}}_{j_2} > 0, \cdots, \left[ \varphi( U_1^{(H-1)} ) \cdots \varphi(U_{d_{H-1}}^{(H-1)}) \right] 
{\mbox{\boldmath $w$}}_{j_H} > 0 \right. \nonumber \\
& & \hspace*{5.5cm} \left. \Big\vert \ w_{ j_1 j_0 } = \lambda_1, \cdots, 
w_{ j_{H} j_{H-1} } = \lambda_{H} \right) \ \ (\mbox{from Eq.} (\ref{eqn4-2})) \nonumber\\
& & (\mbox{$w_{j_{H+1} j_H} = \lambda_{H+1}$ is removed because it is independent of 
$[Z_i]_{(j,p)}$}) \nonumber\\
&\fallingdotseq& \hspace*{-0.3cm} \frac{1}{2^H}. 
$$ We prove below by mathematical induction that the approximate equality $(\fallingdotseq)$ in Eq. (\[eqn4-6\]) holds true.

\[For $H=1$\] This case corresponds to a three-layered neural network. For the sake of simplicity, we let ${\mbox{\boldmath $X$}}_i = (x_1 \cdots x_{j_0} \cdots x_{d_0})^T$ and ${\mbox{\boldmath $w$}}_{j_1} = (w_1 \cdots w_{j_0} \cdots w_{d_0})^T$ where $w_{j_0} = w_{j_1 j_0}$. Then, $$\begin{aligned}
\label{eqn4-7}
& & P \left( {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0 \ \Big\vert \ w_{j_1 j_0} = \lambda_1\right) \nonumber\\
&=& P \left(x_1 w_1 + \cdots + x_{j_0} w_{j_1 j_0} + \cdots + x_{d_0} w_{d_0} > 0 \ 
\Big \vert \ w_{j_1 j_0} = \lambda_1\right) \nonumber\\
&=& P \left(x_1 w_1 + \cdots + x_{j_0} \lambda_1 + \cdots + x_{d_0} w_{d_0} > 0 \ 
\Big \vert \ w_{j_1 j_0} = \lambda_1\right) \nonumber\\
&=& P \Big(x_1 w_1 + \cdots + x_{j_0} \lambda_1 + \cdots + x_{d_0} w_{d_0} > 0 \ \Big) 
\nonumber\\
& & (\mbox{$w_{j_1 j_0} = \lambda_{1}$ is removed because it is independent of 
the other weights}) \nonumber\\
&=& P \Big(x_1 w_1 + \cdots + x_{j_0 - 1} w_{j_0 -1} + x_{j_0 + 1} w_{j_0 + 1} 
+ \cdots + x_{d_0} w_{d_0} > -  x_{j_0} \lambda_1 \ \Big) \nonumber\\
&\fallingdotseq& 
P \Big(x_1 w_1 + \cdots + x_{j_0 - 1} w_{j_0 -1} + x_{j_0 + 1} w_{j_0 + 1} 
+ \cdots + x_{d_0} w_{d_0} > 0 \ \Big) \nonumber\\
& & (\mbox{because $\vert-  x_{j_0} \lambda_1 \vert = \vert x_{j_0} \vert 
\vert \lambda_1 \vert \leq \alpha \cdot (1/d_0)$ and 
the number of input neurons $d_0$ is} \nonumber\\
& & \mbox{sufficiently large}) \nonumber\\
&=& \frac{1}{2}.  \ \ (\mbox{for the same reason that the last equality 
of Eq. (\ref{eqn4-3}) holds true.}) 
$$

\[For $H$\] This case corresponding to a deep nonlinear neural network with $H$ hidden layers, we show that if the case of $H-1$ holds, then case $H$ also holds. Assuming that the case of $H-1$ holds, then $$\begin{aligned}
\label{eqn4-8}
& & P \left( {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0, \left[ \varphi( U_1^{(1)} ) \cdots \varphi(U_{d_1}^{(1)} ) \right] {\mbox{\boldmath $w$}}_{j_2} > 0, 
\cdots, \left[ \varphi( U_1^{(H-1)} ) \cdots \varphi(U_{d_{H-1}}^{(H-1)}) \right] 
{\mbox{\boldmath $w$}}_{j_H} > 0 \right. \nonumber\\
& & \hspace*{8.0cm} \left. \Big\vert \ w_{j_1 j_0} = \lambda_1, 
\cdots, w_{j_H j_{H-1} } = \lambda_H \right) \nonumber\\
&=& P \left( \left[ \varphi( U_1^{(1)} ) \cdots \varphi(U_{d_1}^{(1)} ) \right] 
{\mbox{\boldmath $w$}}_{j_2} > 0, \cdots, \left[ \varphi( U_1^{(H-1)} ) \cdots \varphi(U_{d_{H-1}}^{(H-1)}) \right] 
{\mbox{\boldmath $w$}}_{j_H} > 0 \ \ \right. \nonumber\\
& & \hspace*{2cm} \left. 
\Big\vert \ {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0, w_{j_1 j_0} = \lambda_1, 
\cdots, w_{j_H j_{H-1} } = \lambda_H \right) \cdot \nonumber\\
& & \hspace*{2cm} P\left( {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0 \ \Big\vert \ w_{j_1 j_0} = \lambda_1, 
\cdots, w_{j_H j_{H-1} } = \lambda_H \right) \nonumber\\
&=& P \left( \left[ \varphi( U_1^{(1)} ) \cdots \varphi(U_{d_1}^{(1)} ) \right] 
{\mbox{\boldmath $w$}}_{j_2} > 0, \cdots, \left[ \varphi( U_1^{(H-1)} ) \cdots \varphi(U_{d_{H-1}}^{(H-1)}) \right] 
{\mbox{\boldmath $w$}}_{j_H} > 0 \right. \nonumber\\
& & \hspace*{0.5cm} \left. \Big\vert \ {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0, 
\ w_{j_1 j_0} = \lambda_1, \cdots, w_{j_H j_{H-1} } = \lambda_H \right)  \cdot P\left( {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0 \ \Big\vert \ w_{j_1 j_0} = \lambda_1 \right). \\
& & (\mbox{$w_{j_2 j_1} = \lambda_2, \cdots, w_{j_H j_{H-1}} = \lambda_H$ are removed 
because they are independent of ${\mbox{\boldmath $w$}}_{j_1}$ } ) \nonumber\end{aligned}$$ Here, the first term of the right-hand-side of Eq. (\[eqn4-8\]) represents the probability that the path passing through the $H-1$ hidden neurons $j_2, \cdots, j_H$ for the input training pattern $[ \varphi( U_1^{(1)} ) \cdots \varphi(U_{d_1}^{(1)} ) ]^T \in {\mbox{\boldmath $R$}}^{d_1}$ such that $U_{j_1}^{(1)} = {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0$ and $w_{j_1 j_0} = \lambda_1$ is active. Also, for any $1 \leq s \leq d_1$, by denoting ${\mbox{\boldmath $X$}}_i = (x_1 \cdots x_{d_0})^T$ and ${\mbox{\boldmath $w$}}_s = (w_1 \cdots w_{d_0})$ for the sake of simplicity, $\vert \varphi(U_s^{(1)}) \vert \leq \vert U_s^{(1)} \vert 
=\vert {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_s \vert 
\leq \sum_{t=1}^{d_0} \vert x_t \vert \vert w_t \vert 
\leq \alpha \sum_{t=1}^{d_0} (1/d_0) = \alpha$. Hence, according to the assumption of mathematical induction, the first term of the right-hand-side of Eq. (\[eqn4-8\]) is nearly equal to $(1/2)^{H-1}$. In addition, the second term of the right-hand-side of Eq. (\[eqn4-8\]) is nearly equal to 1/2 from Eq.(\[eqn4-7\]). So, $$\begin{aligned}
\label{eqn4-9}
&P& \hspace*{-0.3cm} \left( {\mbox{\boldmath $X$}}_i^T {\mbox{\boldmath $w$}}_{j_1} > 0, \left[ \varphi( U_1^{(1)} ) 
\cdots \varphi(U_{d_1}^{(1)} ) \right] {\mbox{\boldmath $w$}}_{j_2} > 0, 
\cdots, \left[ \varphi( U_1^{(H-1)} ) \cdots \varphi(U_{d_{H-1}}^{(H-1)}) \right] 
{\mbox{\boldmath $w$}}_{j_H} > 0 \right. \nonumber\\
& & \hspace*{7.0cm} \left. \Big\vert \ w_{j_1 j_0} = \lambda_1, 
\cdots, w_{j_H j_{H-1} } = \lambda_H \right) \nonumber\\
&\fallingdotseq& \left(\frac{1}{2}\right)^{H-1} \cdot \frac{1}{2} \nonumber\\
&=& \frac{1}{2^H}, \end{aligned}$$ which means that the case of $H$ indeed holds. Thus, by mathematical induction, Eq. (\[eqn4-6\]) holds for any $H \geq 1$. Therefore, $$\begin{aligned}
\label{eqn4-10}
& & P \left( [Z_i]_{(j,p)} = 1 \ \Big\vert \ w_{j_1 j_0} = \lambda_1, 
\cdots, w_{j_{H+1} j_H } = \lambda_{H+1} \right) \nonumber\\
&\fallingdotseq& \frac{1}{2^H} \hspace*{1cm} \mbox{(from Eq. (\ref{eqn4-6}))}\nonumber\\
&=& P \left( [Z_i]_{(j,p)} = 1 \right). \hspace*{1cm} \mbox{(from Theorem \ref{thm1})}\end{aligned}$$ This completes the proof.

\[thm3\] For any training pattern $i$, any output neuron $j$, and any path $p$ from an input neuron to the output neuron $j$ in the Kawaguchi model initialized by the even initialization method, random variable $[Z_i]_{(j,p)}$ is independent of the input training singnal ${\mbox{\boldmath $X$}}_i$. [$\Box$]{}

[*Proof*]{}.   Take training pattern $i$, output neuron $j$, path $p$ from an input neuron to output neuron $j$ of the deep nonlinear neural network in the Kawaguchi model initialized by the even initialization method, and ${\mbox{\boldmath $\mu$}}\in {\mbox{\boldmath $R$}}^{d_x}$ arbitrarily and fix them. Then, in the same mode of the proof of Theorem \[thm1\], it is apparent that $$\label{eqn4-12}
P\left([Z_i]_{(j,p)} = 1 \ \big\vert \ {\mbox{\boldmath $X$}}_i = {\mbox{\boldmath $\mu$}}\right) = \frac{1}{2^H}.$$ Therefore, it follows that $$\begin{aligned}
\label{eqn4-13}
P\left([Z_i]_{(j,p)} = 1 \right) &=& \frac{1}{2^H}  \hspace*{1.0cm} (\mbox{from Theorem} \ref{thm1} ) \nonumber\\
&=& P\left([Z_i]_{(j,p)} = 1 \ \ \big\vert \ \ {\mbox{\boldmath $X$}}_i = {\mbox{\boldmath $\mu$}}\right) \hspace*{0.5cm} (\mbox{from Eq.} (\ref{eqn4-12}) ) \end{aligned}$$ holds true. Eq. (\[eqn4-13\]) completes the proof. Theorem \[thm2\] and Theorem \[thm3\] state that assumption A5u-m-1 holds in the initial loss landscape of the Kawaguchi model initialized by the even initialization method.

Thus, it follows from Theorem \[thm1\] that assumptions A1p-m and A5u-m (A5u-m-1) hold in the initial loss landscape of the Kawaguchi model initialized by the even initialization method, both of which were introduced by Kawaguchi (2016). Therefore, no poor local minimum exists in the initial loss landscape of the Kawaguchi model initialized by the even initialization method according to Corollary 3.2 presented by Kawaguchi (2016). Specifically, the following four statements hold true: 1) the loss function is non-convex and non-concave; 2) every local minimum is a global minimum; 3) every critical point that is not a global minimum is a saddle point; and 4) bad saddle points exist.

Discussion and Related Work {#discuss}
===========================

In this section, we compare the even initialization proposed in the subsection \[even\_weight\] with the existing methods. First, let us briefly review several existing methods on the initialization for weights in neural networks.

The following uniform distribution has been often used for setting the initial value for a weight of a neuron: $$\label{standard-initialization}
U \left[ -\frac{1}{\sqrt{n}}, \frac{1}{\sqrt{n}} \right]$$ where $n$ is the number of neurons in the preceding layer (we call it [*standard initialization*]{} here). To the best of the authors’ knowledge, it is a heuristics.

In contrast, the following two weight initialization methods have been derived theoretically. Glorot and Bengio (2010) proposed the [*normalized initialization*]{}: the initial value of each weight between the layer $i$ and the layer $i+1$ is set according to the uniform distribution $$\label{glorot-initialization}
U \left[ -\frac{ \sqrt{6} }{ \sqrt{n_i + n_{i+1} }}, 
             \frac{ \sqrt{6} }{ \sqrt{n_i + n_{i+1} }} \right]$$ independently where $n_i$ is the number of neurons in the layer $i$. Eq. (\[glorot-initialization\]) was derived by keeping the variance of the output vector in each layer equal and keeping the variance of the back-propagated gradients equal, respectively, for the purpose of avoiding their saturation under the assumption that the variances of the input signals are all the same. The object of the normalized initialization is neural networks with symmetric activation function $f$ such that $\partial f(x)/\partial x \vert_{x=0} = 1$ (for example, sigmoid function and $\tanh$ function). Thus, the ReLU activation function is out of scope.

He et al. (2015) proposed a weight initialization method for neural networks with the ReLU activation function: the initial value of each weight between the layer $i$ and the layer $i+1$ is set according to either of the normal distributions $$\label{He-initialization}
N \left( 0, \frac{2}{ n_i } \right) \ \ \mbox{or} \ \ N \left( 0, \frac{2}{ n_{i+1} } \right)$$ independently where $n_i$ is the number of neurons in the layer $i$. Eq. (\[He-initialization\]) was derived by keeping the variance of the net input vector in each layer equal and keeping the variance of the back-propagated gradients equal, respectively, for the purpose of avoiding their saturation.

The normalized initialization is inappropriate for a comparison object of the even initialization because it is invalid for neural networks with the ReLU activation function as described above. Then we compare below the even initialization method with the standard initialization and He initialization. A schematic relationship among the standard initialization, He initialization and the even initialization is shown in Fig. \[fig8\]. The interval of the initial values of weights initialized by the even initialization is contained in both of those of the standard initialization and He initialization because $1/n \leq 1/\sqrt{n}$ holds true for any $n=1,2, \cdots$ where $n$ is the number of neurons in the preceding layer, and the probability distribution which weights obey in the even initialization can be a normal distribution with zero mean or a uniform distribution. Conversely, there exists an interval which does not have any local minimum in both of the intervals of initial weight values of the standard initialization and He initialization. As a numerical example, $[-3\sqrt{2}/\sqrt{n}, 3\sqrt{2}/\sqrt{n}] \fallingdotseq [ -0.424, 0.424 ]$ (He initialization), $[-1/\sqrt{n}, 1/\sqrt{n}] = [ -0.1, 0.1 ]$ (standard initialization) and $[-1/n, 1/n] = [ -0.01, 0.01 ]$ (even initialization) when $n=100$: the number of neurons in the preceding layer is 100.

Conclusions {#concl}
===========

We introduced a weight initialization scheme called [*even initialization*]{} for the wide and deep ReLU neural network model, and proved using the research results presented by Kawaguchi (2016) that no poor local minimum exists in the initial loss landscape in the wide and deep nonlinear neural network initialized by the even initialization method. We also elucidated that the weight values initialized by the even initialization method are contained in those initialized by both of the standard initialization and He initialization methods. We clarified the essential property of the even initialization theoretically. Applying the even initialization to large-scale real-world problems is a future topic.

[**References**]{}

Bengio, Y., Lamblin, P., Popovici, D., and Larochelle, H. Greedy layer-wise training of deep networks. In [*Advances in Neural Information Processing Systems 19 (NIPS’06)*]{}, (B. Schölkopf, J. Platt, and T. Hoffman, eds.), 153-160, 2007.

Choromanska, A., Henaff, M., Mathieu, M., Arous, G. B., and LeCun, Y. The loss surfaces of multilayer networks. In [*Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics*]{}, 192-204, 2015.

Choromanska, A., LeCun, Y., and Arous, G. B. Open problem: the landscape of the loss surfaces of multilayer networks. In [*Proceedings of The 28th Conference on Learning Theory*]{}, 1756-1760, 2015.

Dauphin, Y. N., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., and Bengio, Y. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In [*Advances in Neural Information Processing Systems*]{}, 2933-2941, 2014.

Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In [*International Conference on Artificial Intelligence and Statistics*]{}, 249-256, 2010.

He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In [*Proceedings of the IEEE International Conference on Computer Vision*]{}, 1026-1034, 2015.

Hinton, G. E., Osindero, S., and Teh, Y. A fast learning algorithm for deep belief nets. [*Neural Computation*]{}, 18: 1527-1554, 2006.

Kawaguchi, K. Deep learning without poor local minima. In [*Advances in Neural Information Processing Systems 29*]{}, 2016.

Mohamed, A-R, Dahl, G. E., and Hinton, G. E. Deep belief network for phone recognition. In [*NIPS Workshop on Deep Learning for Speech Recognition and Related Applications*]{}, 2009.

Seide, F., Li, G., and Yu, D. Conversational speech transcription using context-dependent deep neural networks. In [*Proc. Interspeech*]{}, 437-440, 2011.

Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In [*Advances in Neural Information Processing Systems*]{}, 3104-3112, 2014.

Taigman, Y., Yang. M., Ranzoto, M., and Wolf, L. Deepface: Closing the gap to human-level performance in face verification. In [*Proc. Conference on Computer Vision and Pattern Recognition*]{}, 1701-1708, 2014.
