@inproceedings{
du2024a,
title={A Study on Improving Reasoning in Language Models},
author={Yuqing Du and Alexander Havrilla and Sainbayar Sukhbaatar and Pieter Abbeel and Roberta Raileanu},
booktitle={I Can't Believe It's Not Better Workshop: Failure Modes in the Age of Foundation Models},
year={2024},
url={https://openreview.net/forum?id=tCZFmDyPFm}
}
Accurately carrying out complex reasoning is a crucial component of deployable and reliable language models. While current language models can exhibit this capability with few-shot guidance, accurate reasoning is primarily restricted to larger model sizes. In this work, we explore methods for improving the reasoning capabilities of smaller language models which are more deployable than their larger counterparts. Specifically, we look at variations of supervised learning, online reinforcement learning with PPO, and distillation from larger models. Surprisingly, for reasoning tasks such as CommonsenseQA and GSM8K, we find that simple filtered supervised learning often outperforms reward-conditioned supervised learning, and that simpler iterative supervised learning performs on par with online reinforcement learning.
A Study on Improving Reasoning in Language Models
Yuqing Du Alexander Havrilla Sainbayar Sukhbaatar Pieter Abbeel Roberta Raileanu