TITLE: Towards Coding Social Science Datasets with Language Models 

ABSTRACT: Researchers often rely on humans to code (label, annotate, etc.) large sets
of texts. This kind of human coding forms an important part of social science
research, yet the coding process is both resource intensive and highly variable
from application to application. In some cases, efforts to automate this
process have achieved human-level accuracies, but to achieve this, these
attempts frequently rely on thousands of hand-labeled training examples, which
makes them inapplicable to small-scale research studies and costly for large
ones. Recent advances in a specific kind of artificial intelligence tool -
language models (LMs) - provide a solution to this problem. Work in computer
science makes it clear that LMs are able to classify text, without the cost (in
financial terms and human effort) of alternative methods. To demonstrate the
possibilities of LMs in this area of political science, we use GPT-3, one of
the most advanced LMs, as a synthetic coder and compare it to human coders. We
find that GPT-3 can match the performance of typical human coders and offers
benefits over other machine learning methods of coding text. We find this
across a variety of domains using very different coding procedures. This
provides exciting evidence that language models can serve as a critical advance
in the coding of open-ended texts in a variety of applications.
Researchers often rely on humans to code (label, annotate, etc.) large sets
of texts. This kind of human coding forms an important part of social science
research, yet the coding process is both resource intensive and highly variable
from application to application. In some cases, efforts to automate this
process have achieved human-level accuracies, but to achieve this, these
attempts frequently rely on thousands of hand-labeled training examples, which
makes them inapplicable to small-scale research studies and costly for large
ones. Recent advances in a specific kind of artificial intelligence tool -
language models (LMs) - provide a solution to this problem. Work in computer
science makes it clear that LMs are able to classify text, without the cost (in
financial terms and human effort) of alternative methods. To demonstrate the
possibilities of LMs in this area of political science, we use GPT-3, one of
the most advanced LMs, as a synthetic coder and compare it to human coders. We
find that GPT-3 can match the performance of typical human coders and offers
benefits over other machine learning methods of coding text. We find this
across a variety of domains using very different coding procedures. This
provides exciting evidence that language models can serve as a critical advance
in the coding of open-ended texts in a variety of applications.
Towards Coding Social Science Datasets with Language Models
http://arxiv.org/abs/2306.02177v1