In addition to the network architectures listed above, non-linear neural networks with fixed width and large but finite depth are successful and practically more popular [1]}, [2]}. It is well-documented that for a fixed number of parameters, going deeper allows the models to capture richer structures [3]}, [4]}. However, the theoretical foundations for such networks remain widely open due to their complex training landscape.
