@inproceedings{
wulfmeier2024imitating,
title={Imitating Language via Scalable Inverse Reinforcement Learning},
author={Markus Wulfmeier and Michael Bloesch and Nino Vieillard and Arun Ahuja and Jorg Bornschein and Sandy Huang and Artem Sokolov and Matt Barnes and Guillaume Desjardins and Alex Bewley and Sarah Maria Elisabeth Bechtle and Jost Tobias Springenberg and Nikola Momchev and Olivier Bachem and Matthieu Geist and Martin Riedmiller},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=5d2eScRiRC}
}
The majority of language model training builds on imitation learning. It covers pretraining, supervised fine-tuning, and affects the starting conditions for reinforcement learning from human feedback (RLHF). The simplicity and scalability of maximum likelihood estimation (MLE) for next token prediction led to its role as predominant paradigm. However, the broader field of imitation learning can more effectively utilize the sequential structure underlying autoregressive generation. We focus on investigating the inverse reinforcement learning (IRL) perspective to imitation, extracting rewards and directly optimizing sequences instead of individual token likelihoods and evaluate its benefits for fine-tuning large language models.  We provide a new angle, reformulating inverse soft-Q-learning as a temporal difference regularized extension of MLE. This creates a principled connection between MLE and IRL and allows trading off added complexity with increased performance and diversity of generations in the supervised fine-tuning (SFT) setting. We find clear advantages for IRL-based imitation, in particular for retaining diversity while maximizing task performance, rendering IRL a strong alternative on fixed SFT datasets even without online data generation. Our analysis of IRL-extracted reward functions further indicates benefits for more robust reward functions via tighter integration of supervised and preference-based LLM post-training.
Imitating Language via Scalable Inverse Reinforcement Learning
Markus Wulfmeier Michael Bloesch Nino Vieillard Arun Ahuja Jorg Bornschein Sandy Huang Artem Sokolov Matt Barnes Guillaume Desjardins Alex Bewley Sarah Maria Elisabeth Bechtle Jost Tobias Springenberg Nikola Momchev Olivier Bachem Matthieu Geist Martin Riedmiller