@inproceedings{
li2024reward,
title={Reward Machines for Deep {RL} in Noisy and Uncertain Environments},
author={Andrew C Li and Zizhao Chen and Toryn Q. Klassen and Pashootan Vaezipoor and Rodrigo Toro Icarte and Sheila A. McIlraith},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=Cc0ckJlJF2}
}
Reward Machines provide an automaton-inspired structure for specifying instructions, safety constraints, and other temporally extended reward-worthy behaviour. By exposing the underlying structure of a reward function, they enable the decomposition of an RL task, leading to impressive gains in sample efficiency. Although Reward Machines and similar formal specifications have a rich history of application towards sequential decision-making problems, they critically rely on a ground-truth interpretation of the domain-specific vocabulary that forms the building blocks of the reward function—such ground-truth interpretations are elusive in the real world due in part to partial observability and noisy sensing. In this work, we explore the use of Reward Machines for Deep RL in noisy and uncertain environments. We characterize this problem as a POMDP and propose a suite of RL algorithms that exploit task structure under uncertain interpretation of the domain-specific vocabulary. Through theory and experiments, we expose pitfalls in naive approaches to this problem while simultaneously demonstrating how task structure can be successfully leveraged under noisy interpretations of the vocabulary.
Reward Machines for Deep RL in Noisy and Uncertain Environments
Andrew C Li Zizhao Chen Toryn Q. Klassen Pashootan Vaezipoor Rodrigo Toro Icarte Sheila A. McIlraith