Large neural network models have become a mainstay of natural language processing and computer vision, yet their initialization and learning rates are set in a largely heuristic fashion, potentially varying from paper to paper and one model size to the next.
The-Parameterization (P)  offers a potential solution to these challenges, yielding scaling rules for model initialization and learning rates, and reportedly enabling zero-shot hyperparameter transfer from small to large models in a variety of cases. Despite the evident promise, theP scaling rules are not yet widely adopted, perhaps due to higher implementation complexity, many variations, or complex theoretical background.
This work investigatesP empirically, focusing on the ubiquitous transformer architecture, and aims to answer a simple question: does-Transfer yield optimal learning rates in practice?
Studying models with up to 10B parameters and training budgets of up to 190B tokens, we find-Transfer works as intended for the majority of important cases, yet also identify a few cases where it may not.