@inproceedings{
zhang2023active,
title={Active Learning for Iterative Offline Reinforcement Learning},
author={Lan Zhang and Luigi Franco Tedesco and Pankaj Rajak and Youcef Zemmouri and Hakan Brunzell},
booktitle={NeurIPS 2023 Workshop on Adaptive Experimental Design and Active Learning in the Real World},
year={2023},
url={https://openreview.net/forum?id=yuJEkWSkTN}
}
Offline Reinforcement Learning (RL) has emerged as a promising approach to address
real-world challenges where online interactions with the environment are limited, risky,
or costly. Although, recent advancements produce high quality policies from offline data,
currently, there is no systematic methodology to continue to improve them without resorting
to online fine-tuning. This paper proposes to repurpose Offline RL to produce a sequence
of improving policies, namely, Iterative Offline Reinforcement Learning (IORL). To produce
such sequence, IORL has to cope with imbalanced offline datasets and to perform controlled
environment exploration. Specifically, we introduce ”Return-based Sampling” as means
to selectively prioritize experience from high-return trajectories and active learning driven
”Dataset Uncertainty Sampling” to probe state-actions inversely proportional to density
in the dataset.We demonstrate that our proposed approach produces policies that achieve
monotonically increasing average returns, from 65.4 to 140.2, in the Atari environment.
Active Learning for Iterative Offline Reinforcement Learning
Lan Zhang Luigi Franco Tedesco Pankaj Rajak Youcef Zemmouri Hakan Brunzell