@inproceedings{
zhang2023labelbench,
title={LabelBench: A Comprehensive Framework for Benchmarking Adaptive Label-Efficient Learning},
author={Jifan Zhang and Yifang Chen and Gregory Canal and Arnav Mohanty Das and Gantavya Bhatt and Yinglun Zhu and Stephen Mussmann and Simon Shaolei Du and Jeff Bilmes and Kevin Jamieson and Robert D Nowak},
booktitle={NeurIPS 2023 Workshop on Adaptive Experimental Design and Active Learning in the Real World},
year={2023},
url={https://openreview.net/forum?id=4X33gHxHf1}
}
Labeled data are critical to modern machine learning applications, but obtaining labels can be expensive. To mitigate this cost, machine learning methods, such as transfer learning, semi-supervised learning and active learning, aim to be $\text{\textit{label-efficient}}$: achieving high predictive performance from relatively few labeled examples. While obtaining the best label-efficiency in practice often requires combinations of these techniques, existing benchmark and evaluation frameworks do not capture a concerted combination of all such techniques. This paper addresses this deficiency by introducing LabelBench, a new computationally-efficient framework for joint evaluation of multiple label-efficient learning techniques. As an application of LabelBench, we introduce a novel benchmark of state-of-the-art active learning methods in combination with semi-supervised learning for fine-tuning pretrained vision transformers. Our benchmark demonstrates significantly better label-efficiencies than previously reported in active learning. LabelBench's modular codebase is open-sourced for the broader community to contribute label-efficient learning methods and benchmarks. The repository can be found at: https://github.com/EfficientTraining/LabelBench.
LabelBench: A Comprehensive Framework for Benchmarking Adaptive Label-Efficient Learning
Jifan Zhang Yifang Chen Gregory Canal Arnav Mohanty Das Gantavya Bhatt Yinglun Zhu Stephen Mussmann Simon Shaolei Du Jeff Bilmes Kevin Jamieson Robert D Nowak