@inproceedings{
fisch2024stratified,
title={Stratified Prediction-Powered Inference for Effective Hybrid Evaluation of Language Models},
author={Adam Fisch and Joshua Maynez and R. Alex Hofer and Bhuwan Dhingra and Amir Globerson and William W. Cohen},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=8CBcdDQFDQ}
}
Prediction-powered inference (PPI) is a method that improves statistical estimates based on limited human-labeled data.  PPI achieves this by combining small amounts of human-labeled data with larger amounts of data labeled by a reasonably accurate---but potentially biased---automatic system, in a way that results in tighter confidence intervals for certain parameters of interest (e.g., the mean performance of a language model). In this paper, we propose a method called Stratified Prediction-Powered Inference (StratPPI), in which we show that the basic PPI estimates can be considerably improved by employing simple data stratification strategies. Without making any assumptions on the underlying automatic labeling system or data distribution, we derive an algorithm for computing provably valid confidence intervals for parameters of any dimensionality that is based on stratified sampling. In particular, we show both theoretically and empirically that, with appropriate choices of stratification and sample allocation, our approach can provide substantially tighter confidence intervals than  unstratified  approaches. Specifically, StratPPI  is expected to improve in cases where the performance of the autorater varies across different conditional distributions of the target data.
Stratified Prediction-Powered Inference for Effective Hybrid Evaluation of Language Models
Adam Fisch Joshua Maynez R. Alex Hofer Bhuwan Dhingra Amir Globerson William W. Cohen