Moreover, to explore the role of layer width in expressive capacity quantitatively, Lu et al. [1]} raise the dual question of depth efficiency. That is, whether there exists wide, shallow ReLU networks that cannot be approximated by any narrow, deep neural network whose size is not substantially increased.
Denote by \(F_{\mathcal {A}}: \mathbb {R}^d \rightarrow \mathbb {R}\)  the function represented by a ReLU neural network \(\mathcal {A}\)  whose depth \(h=3\)  and whose width of each layer is \(2k^2\) , where \(k\)  is an integer such that \(k \ge d+4\) . Let \(F_{\mathcal {B}}: \mathbb {R}^d \rightarrow \mathbb {R}\)  be the function represented by a ReLU neural network \(\mathcal {B}\)  whose depth \(h \le k+2\)  and whose width of each hidden layer \(m_i \le k^{3/2}\) , where the parameter values of \(\mathcal {B}\)  are bounded by \([-b, b]\) . For any constant \(b>0\) , there is \(\epsilon >0\)  such that \(F_{\mathcal {A}}\)  can never be approximated by \(F_{\mathcal {B}}\)  to \(\epsilon \)  \(L^1\) -distance. That is,
\(\nonumber \int _{\mathbb {R}^d} |F_{\mathcal {A}}(x) - F_{\mathcal {B}}(x)| dx \ge \epsilon .\) 
