DiracNets: Training Very Deep Neural Networks Without Skip Connections
Deep neural networks with skip connections, such as ResNet, show excellent performance in various image classification benchmarks. It is though observed that the initial motivation behind them training deeper networks does not actually hold true, and the benefits come from increased capacity, rather than from depth. Motivated by this, and inspired from ResNet, we propose a simple Dirac weight parameterization, which allows us to train very deep plain networks without explicit skip connections, and achieve nearly the same performance. This parameterization has a minor computational cost at training time and no cost at all at inference, as both Dirac parameterization and batch normalization can be folded into convolutional filters, so that network becomes a simple chain of convolution ReLU pairs. We are able to match ResNet 1001 accuracy on CIFAR 10 with 28 layer wider plain DiracNet, and closely match ResNets on ImageNet. Our parameterization also mostly eliminates the need of careful initialization in residual and non residual networks. The code and models for our experiments are available at this https URL
ArXiv