Non-Markovian Reward Modelling from Trajectory Labels via Interpretable Multiple Instance Learning
We generalise the problem of reward modelling (RM) for reinforcement learning
(RL) to handle non-Markovian rewards. Existing work assumes that human
evaluators observe each step in a trajectory independently when providing
feedback on agent behaviour. In this work, we remove this assumption, extending
RM to include hidden state information that captures temporal dependencies in
human assessment of trajectories. We then show how RM can be approached as a
multiple instance learning (MIL) problem, and develop new MIL models that are
able to capture the time dependencies in labelled trajectories. We demonstrate
on a range of RL tasks that our novel MIL models can reconstruct reward
functions to a high level of accuracy, and that they provide interpretable
learnt hidden information that can be used to train high-performing agent
policies.