2022-5-20
Selection-Inference: Exploiting Large
Language Models for Interpretable Logical
Reasoning
Antonia Creswell1, Murray Sh
Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning
Large language models (LLMs) have been shown to be capable of impressive
few-shot generalisation to new tasks. However, they still tend to perform
poorly on multi-step logical reasoning problems. Here we carry out a
comprehensive evaluation of LLMs on 50 tasks that probe different aspects of
logical reasoning. We show that language models tend to perform fairly well at
single step inference or entailment tasks, but struggle to chain together
multiple reasoning steps to solve more complex problems. In light of this, we
propose a Selection-Inference (SI) framework that exploits pre-trained LLMs as
general processing modules, and alternates between selection and inference to
generate a series of interpretable, casual reasoning steps leading to the final
answer. We show that a 7B parameter LLM used within the SI framework in a
5-shot generalisation setting, with no fine-tuning, yields a performance
improvement of over 100% compared to an equivalent vanilla baseline on a suite
of 10 logical reasoning tasks. The same model in the same setting even
outperforms a significantly larger 280B parameter baseline on the same suite of
tasks. Moreover, answers produced by the SI framework are accompanied by a
causal natural-language-based reasoning trace, which has important implications
for the safety and trustworthiness of the system.
Antonia Creswell Murray Shanahan Irina Higgins