In addition, while in early days the answer to improving performance was often to build ever larger networks, this approach has more recently been shown to not scale well because of (i) overfitting [1]}, [2]}, [3]} and (ii) gradient vanishing [4]}  — (i) When there is too much overfitting, this results in poor generalization in the test set. For example, Laurence et al. ([3]}, in their Fig.2) showed that, generally, as the network size increases, training error decreases but this ultimately leads to overfitting. As a consequence, test errors exhibit a "U" shaped behavior as a function of network size. (ii) The gradient vanishing problem limits the number of parameters and layers for scaling up the size of networks. Residual networks [6]} are famous for easing the training of networks that are substantially deeper. They create a gradient highway for reducing the effect of gradient vanishing. However, residual networks still cannot avoid the problem entirely. For example, in Tab. 6 of the original residual network paper [6]}, performance of a residual network with 1202 layers is lower than that of residual networks with 32, 44, 56, 110 layers. We find similar scaling issues as detailed in our Results. This prompts us to explore alternatives for boosting performance with more parameters rather than naively expanding the size of a single network.
