ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Data
Large pretrained language models have been performing increasingly well in a
variety of downstream tasks via prompting. However, it remains unclear from
where the model learns the task-specific knowledge, especially in a zero-shot
setup. In this work, we want to find evidence of the model's task-specific
competence from pretraining and are specifically interested in locating a very
small subset of pretraining data that directly supports the model in the task.
We call such a subset supporting data evidence and propose a novel method ORCA
to effectively identify it, by iteratively using gradient information related
to the downstream task. This supporting data evidence offers interesting
insights about the prompted language models: in the tasks of sentiment analysis
and textual entailment, BERT shows a substantial reliance on BookCorpus, the
smaller corpus of BERT's two pretraining corpora, as well as on pretraining
examples that mask out synonyms to the task verbalizers.