blueQ-learning is a fundamental approach to RL but we also experimented with other state-of-the-art model-free RL algorithms, such as PPO [1]} and its predecessor, TRPO [2]}. Table REF  summarises our experiments that show there is little improvement to be had. It is worth noting that PPO and TRPO demonstrate much lower standard deviation in final performance across multiple training runs compared to DDQN+PER. This is to be expected as those policy gradient methods are revered for their stability, but the best seed for DDQN+PER consistently outperformed them, albeit marginally, for the networks we tested.
