We further evaluated LSVI-PHE on a continuous control task which requires exploration: sparse reward variant of continuous control MountainCar from OpenAI Gym [1]}. This environment consists of a 2-dimensional continuous state space and a 1-dimensional continuous action space \([-1,1]\) . The agent only receives a reward of \(+1\)  if it reaches the top of the hill and everywhere else it receives a reward of 0. We set the length of the horizon to be 1000 and discount factor \(\gamma = 0.99\) .
