Hanin and Rolnick [1]} investigate the high-capacity low-reality phenomenon in fully connected neural networks with piecewise linear activation functions, especially ReLU. They propose two effective complexity measures using the number of linear regions in the input space and the volume of boundaries between these linear regions.
