Online vs. Offline Adaptive Domain Randomization Benchmark
Physics simulators have shown great promise for conveniently learning
reinforcement learning policies in safe, unconstrained environments. However,
transferring the acquired knowledge to the real world can be challenging due to
the reality gap. To this end, several methods have been recently proposed to
automatically tune simulator parameters with posterior distributions given real
data, for use with domain randomization at training time. These approaches have
been shown to work for various robotic tasks under different settings and
assumptions. Nevertheless, existing literature lacks a thorough comparison of
existing adaptive domain randomization methods with respect to transfer
performance and real-data efficiency. In this work, we present an open
benchmark for both offline and online methods (SimOpt, BayRn, DROID, DROPO), to
shed light on which are most suitable for each setting and task at hand. We
found that online methods are limited by the quality of the currently learned
policy for the next iteration, while offline methods may sometimes fail when
replaying trajectories in simulation with open-loop commands. The code used
will be released at https://github.com/gabrieletiboni/adr-benchmark.