@inproceedings{
lotfi2023nonvacuous,
title={Non-Vacuous Generalization Bounds for Large Language Models},
author={Sanae Lotfi and Marc Finzi and Yilun Kuang and Tim Rudner and Micah Goldblum and Andrew Wilson},
booktitle={NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning},
year={2023},
url={https://openreview.net/forum?id=7QRaAfbium}
}
Modern language models can contain billions of parameters, raising the question of whether they can generalize beyond the training data or simply regurgitate their training corpora. We provide the first non-vacuous generalization bounds for pretrained large language models (LLMs), indicating that language models are capable of discovering regularities that generalize to unseen data. In particular, we derive a compression bound that is valid for the unbounded log-likelihood loss, and we extend the bound to handle subsampling, accelerating bound computation on massive datasets. To achieve the extreme level of compression required for non-vacuous generalization bounds, we devise SubLoRA, a low-dimensional non-linear parameterization. Using this approach, we find that larger models have better generalization bounds and are more compressible than smaller models.
Non-Vacuous Generalization Bounds for Large Language Models
Sanae Lotfi Marc Finzi Yilun Kuang Tim Rudner Micah Goldblum Andrew Wilson