Several studies explore the bottleneck of model size (i.e., depth, width) in the expressive capacity. That is, when the model size may become a bottleneck that restricts the expressive capacity. For example, Hanin and Sellke [1]} and Lu et al. [2]} discover that any deep ReLU network with width constrained by the input dimensionality has very limited expressive power.
Serra et al. [3]} find that smaller widths in the first few layers of a deep ReLU network cause a bottleneck on the expressive power.
Kileel et al. [4]} identify a bottleneck of layer width of deep polynomial networks. In a deep polynomial network, if there is a very narrow layer, no matter how wide the other layers are, the network can never correspond to a convex functional space. A convex functional space benefits the optimization and makes the model easier to train.
Research on the bottleneck of expressive capacity may help to tackle many other problems, such as model design, model selection, model compression, and pruning.
