@inproceedings{
jin2024learning,
title={Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate},
author={Can Jin and Tong Che and Hongwu Peng and Yiyuan Li and Dimitris N. Metaxas and Marco Pavone},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=B1FOes6cyq}
}
Generalization remains a central challenge in machine learning. In this work, we propose *Learning from Teaching* (**LoT**), a novel regularization technique for deep neural networks to enhance generalization. Inspired by the human ability to capture concise and abstract patterns, we hypothesize that generalizable correlations are expected to be easier to imitate. LoT operationalizes this concept to improve the generalization of the main model with auxiliary student learners. The student learners are trained by the main model and, in turn, provide feedback to help the main model capture more generalizable and imitable correlations. Our experimental results across several domains, including Computer Vision, Natural Language Processing, and methodologies like Reinforcement Learning, demonstrate that the introduction of LoT brings significant benefits compared to training models on the original dataset. The results suggest the effectiveness and efficiency of LoT in identifying generalizable information at the right scales while discarding spurious data correlations, thus making LoT a valuable addition to current machine learning. Code is available at https://github.com/jincan333/LoT.
Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate
Can Jin Tong Che Hongwu Peng Yiyuan Li Dimitris N. Metaxas Marco Pavone