Benchmarking Constraint Inference in Inverse Reinforcement Learning
When deploying Reinforcement Learning (RL) agents into a physical system, we
must ensure that these agents are well aware of the underlying constraints. In
many real-world problems, however, the constraints followed by expert agents
(e.g., humans) are often hard to specify mathematically and unknown to the RL
agents. To tackle these issues, Constraint Inverse Reinforcement Learning
(CIRL) considers the formalism of Constrained Markov Decision Processes (CMDPs)
and estimates constraints from expert demonstrations by learning a constraint
function. As an emerging research topic, CIRL does not have common benchmarks,
and previous works tested their algorithms with hand-crafted environments
(e.g., grid worlds). In this paper, we construct a CIRL benchmark in the
context of two major application domains: robot control and autonomous driving.
We design relevant constraints for each environment and empirically study the
ability of different algorithms to recover those constraints based on expert
trajectories that respect those constraints. To handle stochastic dynamics, we
propose a variational approach that infers constraint distributions, and we
demonstrate its performance by comparing it with other CIRL baselines on our
benchmark. The benchmark, including the information for reproducing the
performance of CIRL algorithms, is publicly available at
https://github.com/Guiliang/CIRL-benchmarks-public