@inproceedings{
klissarov2023motif,
title={Motif: Intrinsic Motivation from Artificial Intelligence Feedback},
author={Martin Klissarov and Pierluca D'Oro and Shagun Sodhani and Roberta Raileanu and Pierre-Luc Bacon and Pascal Vincent and Amy Zhang and Mikael Henaff},
booktitle={Second Agent Learning in Open-Endedness Workshop},
year={2023},
url={https://openreview.net/forum?id=reHXok5jiQ}
}
Exploring rich environments and evaluating one's actions without prior knowledge is immensely challenging. In this paper, we propose Motif, a general method to interface such prior knowledge from a Large Language Model (LLM) with an agent. Motif is based on the idea of grounding LLMs for decision-making without requiring them to interact with the environment: it elicits preferences from an LLM over pairs of captions to construct an intrinsic reward, which is then used to train agents with reinforcement learning. We evaluate Motif's performance and behavior on the challenging, open-ended and procedurally-generated NetHack game. Surprisingly, by only learning to maximize its intrinsic reward, Motif achieves a higher game score than an algorithm directly trained to maximize the score itself. When combining Motif's intrinsic reward with the environment reward, our method significantly outperforms existing approaches and makes progress on tasks where no advancements have ever been made without demonstrations. Finally, we show that Motif mostly generates intuitive human-aligned behaviors which can be steered easily through prompt modifications, while scaling well with the LLM size and the amount of information given in the prompt.
Motif: Intrinsic Motivation from Artificial Intelligence Feedback
Martin Klissarov Pierluca D'Oro Shagun Sodhani Roberta Raileanu Pierre-Luc Bacon Pascal Vincent Amy Zhang Mikael Henaff