Large language models (LMs)  offer unprecedented language generation capabilities and exciting opportunities for interaction design.
However, their highly context-dependent capabilities are difficult to grasp and are often subjectively interpreted.
In this paper, we argue that bycurating and analyzing large interaction datasets, the HCI community can foster more incisive examinations of LMs’ generative capabilities.
Exemplifying this approach, we presentCoAuthor, a dataset designed for revealingGPT-3’s capabilities in assisting creative and argumentative writing.CoAuthorcaptures rich interactions between 63 writers and four instances ofGPT-3across 1445 writing sessions.
We demonstrate thatCoAuthorcan address questions aboutGPT-3’s language, ideation, and collaboration capabilities,
and reveal its contribution as a writing “collaborator” under various definitions of good collaboration.
Finally, we discuss how this work may facilitate a more principled discussion around LMs’ promises and pitfalls in relation to interaction design.
The dataset and an interface for replaying the writing sessions are publicly available athttps://coauthor.stanford.edu.