We run our experiments on RiverSwim [1]}, DeepSea [2]} and sparse MountainCar [3]} environments as these are considered to be hard exploration problems where \(\varepsilon \) -greedy is known to have poor performance. For both RiverSwim and DeepSea experiments, we make use of linear features. The objective here is to compare an exploration method that randomizes the targets in the history (LSVI-PHE) with an exploration method that computes upper confidence bounds given the history (LSVI-UCB) [4]}, [5]}. For the continous control MountainCar environment, we use neural-network as function approximator to implement LSVI-PHE. The objective here is to compare deep RL variant of LSVI-PHE against other popular deep RL algorithms specifically designed to tackle exploration task.
