DetectGPT: Zero-Shot Machine-Generated Text Detection
using Probability Curvature
Eric Mitchell1Yoonho Lee1Alexander Khazatsky1C
DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
The increasing fluency and widespread usage of large language models (LLMs)
highlight the desirability of corresponding tools aiding detection of
LLM-generated text. In this paper, we identify a property of the structure of
an LLM's probability function that is useful for such detection. Specifically,
we demonstrate that text sampled from an LLM tends to occupy negative curvature
regions of the model's log probability function. Leveraging this observation,
we then define a new curvature-based criterion for judging if a passage is
generated from a given LLM. This approach, which we call DetectGPT, does not
require training a separate classifier, collecting a dataset of real or
generated passages, or explicitly watermarking generated text. It uses only log
probabilities computed by the model of interest and random perturbations of the
passage from another generic pre-trained language model (e.g., T5). We find
DetectGPT is more discriminative than existing zero-shot methods for model
sample detection, notably improving detection of fake news articles generated
by 20B parameter GPT-NeoX from 0.81 AUROC for the strongest zero-shot baseline
to 0.95 AUROC for DetectGPT. See https://ericmitchell.ai/detectgpt for code,
data, and other project information.
Eric Mitchell Yoonho Lee Alexander Khazatsky Christopher D. Manning Chelsea Finn