@inproceedings{
schaeffer2023divergence,
title={Divergence at the Interpolation Threshold: Identifying, Interpreting \& Ablating the Sources of a Deep Learning Puzzle},
author={Rylan Schaeffer and Zachary Robertson and Akhilan Boopathy and Mikail Khona and Ila Fiete and Andrey Gromov and Sanmi Koyejo},
booktitle={NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning},
year={2023},
url={https://openreview.net/forum?id=6V11PG7w5l}
}
Machine learning models misbehave, often in unexpected ways. One prominent misbehavior is when the test loss diverges at the interpolation threshold, perhaps best known from its distinctive appearance in double descent. While considerable theoretical effort has gone into understanding generalization of overparameterized models, less effort has been made at understanding why the test loss misbehaves at the interpolation threshold. Moreover, analytically solvable models in this area employ a range of assumptions and use complex techniques from random matrix theory, statistical mechanics, and kernel methods, making it difficult to assess when and why test error might diverge. In this work, we analytically study the simplest supervised model - ordinary linear regression - and show intuitively and rigorously when and why a divergence occurs at the interpolation threshold using basic linear algebra. We identify three interpretable factors that, when all present, cause the divergence. We demonstrate on real data that linear models' test losses diverge at the interpolation threshold and that the divergence disappears when we ablate any one of the three identified factors.
Divergence at the Interpolation Threshold: Identifying, Interpreting     Ablating the Sources of a Deep Learning Puzzle
Rylan Schaeffer Zachary Robertson Akhilan Boopathy Mikail Khona Ila Fiete Andrey Gromov Sanmi Koyejo