@inproceedings{
zhang2024a,
title={A Study on the Calibration of In-context Learning},
author={Hanlin Zhang and YiFan Zhang and Yaodong Yu and Dhruv Madeka and Dean Foster and Eric P. Xing and Himabindu Lakkaraju and Sham M. Kakade},
booktitle={I Can't Believe It's Not Better Workshop: Failure Modes in the Age of Foundation Models},
year={2024},
url={https://openreview.net/forum?id=FWTqwlHBC5}
}
Modern auto-regressive models are trained to minimize log loss by predicting the next token. As a result, they are expected to get calibrated answers when framing problems as next-token prediction tasks. We study this for in-context learning (ICL), a widely used way to adapt frozen large language models (LLMs) via crafting prompts and investigate the trade-offs between performance and calibration on a wide range of natural language understanding and reasoning tasks. We conduct extensive experiments to show that such trade-offs may get worse as we increase model size, incorporate more ICL examples, and fine-tune models using instruction or dialog tuning on carefully curated datasets. Furthermore, we find that common recalibration techniques that are widely effective such as temperature scaling may provide limited gains for calibration errors, suggesting that new methods may be required for settings where models are expected to be reliable.
A Study on the Calibration of In-context Learning
Hanlin Zhang YiFan Zhang Yaodong Yu Dhruv Madeka Dean Foster Eric P. Xing Himabindu Lakkaraju Sham M. Kakade