State Representation Learning for Control: An Overview <s> Assessment, comparison and reproducibility in SRL <s> From the Publisher: ::: In Reinforcement Learning, Richard Sutton and Andrew Barto provide a clear and simple account of the key ideas and algorithms of reinforcement learning. Their discussion ranges from the history of the field's intellectual foundations to the most recent developments and applications. The only necessary mathematical background is familiarity with elementary concepts of probability. <s> BIB001 </s> State Representation Learning for Control: An Overview <s> Assessment, comparison and reproducibility in SRL <s> In recent years, significant progress has been made in solving challenging problems across various domains using deep reinforcement learning (RL). Reproducing existing work and accurately judging the improvements offered by novel methods is vital to sustaining this progress. Unfortunately, reproducing results for state-of-the-art deep RL methods is seldom straightforward. In particular, non-determinism in standard benchmark environments, combined with variance intrinsic to the methods, can make reported results tough to interpret. Without significance metrics and tighter standardization of experimental reporting, it is difficult to determine whether improvements over the prior state-of-the-art are meaningful. In this paper, we investigate challenges posed by reproducibility, proper experimental techniques, and reporting procedures. We illustrate the variability in reported metrics and results when comparing against common baselines and suggest guidelines to make future results in deep RL more reproducible. We aim to spur discussion about how to ensure continued progress in the field by minimizing wasted effort stemming from results that are non-reproducible and easily misinterpreted. <s> BIB002
The assessment challenge of SRL is two-sided. First, there is no easy nor certified way for validating a learned representation. Secondly, the lack of common evaluation frameworks makes a fair comparison between approaches difficult. As mentioned in Section 4.3, the most objective method to evaluate the quality of representations is to check if the state representation learned can be used by an RL algorithm to solve a task more efficiently. However, this assessment is uncertain and unstable given the stochasticity of reinforcement learning algorithms BIB002 . Moreover, it is not obvious which RL algorithm is the best choice, and thus, several should be used in the comparison. In practice, a large amount of policy evaluation runs is therefore required in order to provide a robust assessment, which is possible in simulation but is seldom applicable on real robots, given the robots fragility and experimentation time involved. In this case, it is therefore interesting to use several of the other measures presented in section 4.3, that only give partial information on the state representation quality, but are possible to apply for a comparison with a ground truth state. Comparing approaches from published results is also particularly hard because of the high variability of the environments and data used in the different approaches (as illustrated in Table 2 ). This points to the need of an evaluation framework incorporating several tasks and several evaluation metrics similar to the ones proposed for reinforcement learning such as the DeepMind Control Suite BIB001 . Reproducibility guidelines with proper experimental techniques and reporting procedures, as pointed in BIB002 for RL, should also be defined for SRL. In the mean time, as there is not yet an ideal method for state representation assessment, researchers should at least make public their simulation environment (with possible ground truth), and use simulation settings from other approaches to provide fairer comparisons and facilitate the method reproducibility. Furthermore, we strongly encourage authors to entirely describe their experiments, in particular, report their data and models' characteristics.