@inproceedings{
bordelon2023depthwise,
title={Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit},
author={Blake Bordelon and Lorenzo Noci and Mufan Li and Boris Hanin and Cengiz Pehlevan},
booktitle={NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning},
year={2023},
url={https://openreview.net/forum?id=6pfCFDPhy6}
}
We study residual networks with a residual branch scale of $1/\sqrt{\text{depth}}$ in combination with the $\mu$P parameterization.
We provide experiments demonstrating that residual architectures including convolutional ResNets and Vision Transformers trained with this parameterization exhibit transfer of optimal hyperparameters across width and depth on CIFAR-10 and ImageNet. 
Furthermore, using recent developments in the dynamical mean field theory (DMFT) description of neural network learning dynamics,  we show that this parameterization of ResNets admits a well-defined feature learning joint infinite-width and infinite-depth limit and show convergence of finite-size network dynamics towards this limit.
Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit
Blake Bordelon Lorenzo Noci Mufan Li Boris Hanin Cengiz Pehlevan