Last and, possibly, least, we would be remiss to ignore the question of depth in a paper with deep in its title. Yet, while many analyses in this paper are applicable to multi-layered networks, it is the width that drives most of the observed phenomena and intuitions. Despite recent efforts, the importance of depth is still not well-understood. Properties of deep architectures point to the limitations of simple parameter counting – increasing the depth of an architecture appears to have very different effects from increasing the width, even if the total number of trainable parameters is the same. In particular, while wider networks are generally observed to perform better than more narrow architectures ([1]}, even with optimal early stopping [2]}), the same is not true with respect to the depth, where very deep architectures can be inferior [3]}. One line of inquiry is interpreting depth recursively. Indeed,
in certain settings increasing the depth manifests similarly to iterating a map given by a shallow network [4]}. Furthermore, fixed points of such iterations have been proposed as an alternative to deep networks with some success [5]}.
More weight for this point of view is provided by the fact that tangent kernels of infinitely wide architectures satisfy a recursive relationship with respect to their depth [6]}.
