Neural networks are powerful tools in the area of machine learning, especially deep learning. And it is observed that we can normally get better performance when we increase the number of parameters in neural networks, such as the width (the dimensions of parameter matrices each layer) and the depth (the number of layers). By intuition, when the number of parameters grows neural networks will have the potential to overfit the training data, resulting in a larger generalization error. However, it is surprising that neural networks as deep as \(1,000\)  layers can still possess remarkable generalization ability [1]}. This inspires researchers to study the representation power of neural networks.
