@inproceedings{
huang2024the,
title={The {ALCHE}mist: Automated Labeling 500x {CHE}aper than {LLM} Data Annotators},
author={Tzu-Heng Huang and Catherine Cao and Vaishnavi Bhargava and Frederic Sala},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=T0glCBw28a}
}
Large pretrained models can be used as annotators, helping replace or augment crowdworkers and enabling distilling generalist models into smaller specialist models. Unfortunately, this comes at a cost: employing top-of-the-line models often requires paying thousands of dollars for API calls, while the resulting datasets are static and challenging to audit. To address these challenges, we propose a simple alternative: rather than directly querying labels from pretrained models, we task models to generate programs that can produce labels. These programs can be stored and applied locally, re-used and extended, and cost orders of magnitude less. Our system, $\textbf{Alchemist}$, obtains comparable to or better performance than large language model-based annotation in a range of tasks for a fraction of the cost: on average, improvements amount to a $\textbf{12.9}$% enhancement while the total labeling costs across all datasets are reduced by a factor of approximately $\textbf{500}\times$.
The ALCHEmist: Automated Labeling 500x CHEaper than LLM Data Annotators
Tzu-Heng Huang Catherine Cao Vaishnavi Bhargava Frederic Sala