Using cognitive psychology to understand GPT-3
We study GPT-3, a recent large language model, using tools from cognitive
psychology. More specifically, we assess GPT-3's decision-making, information
search, deliberation, and causal reasoning abilities on a battery of canonical
experiments from the literature. We find that much of GPT-3's behavior is
impressive: it solves vignette-based tasks similarly or better than human
subjects, is able to make decent decisions from descriptions, outperforms
humans in a multi-armed bandit task, and shows signatures of model-based
reinforcement learning. Yet we also find that small perturbations to
vignette-based tasks can lead GPT-3 vastly astray, that it shows no signatures
of directed exploration, and that it fails miserably in a causal reasoning
task. These results enrich our understanding of current large language models
and pave the way for future investigations using tools from cognitive
psychology to study increasingly capable and opaque artificial agents.