Arora et al. [1]} investigate the importance of depth on deep neural networks with ReLU activation function.
First, they investigate neural networks with one-dimensional input and one-dimensional output. They prove that given any natural numbers \(k \ge 1\)  and \(w \ge 2\) , there exists a family of functions that can be represented by a ReLU neural network with \(k\)  hidden layers each of width \(w\) . However, to represent this family of functions, a network with \(k^{\prime } < k\)  hidden layers requires at least \(\frac{1}{2}k^{\prime }w^{\frac{k}{k^{\prime }}}-1\)  hidden neurons.
Then, they investigate ReLU neural networks with \(d\)  input dimensions. They prove that, given natural numbers \(k, m,d \ge 1\)  and \(w \ge 2\) ,
there exists a family of \(\mathbb {R}^d \rightarrow \mathbb {R}\)  functions that can be represented by a ReLU network with \(k+1\)  hidden layers and \(2m+wk\)  neurons. This family of functions is constructed using the zonotope theory from the polyhedral theory [2]}. However, to represent this family of functions, the minimum number of hidden neurons that a ReLU neural network with \(k^{\prime }\le k\)  hidden layers requires is
\(\max \lbrace \frac{1}{2}(k^{\prime }w^{\frac{k}{k^{\prime }d}})(m-1)^{(1-\frac{1}{d})\frac{1}{k^{\prime }}} - 1, k^{\prime }(\frac{w^{\frac{k}{k^{\prime }}}}{d^{\frac{1}{k^{\prime }}}})\rbrace .\) 
