For this setting, we compare four algorithms: LSVI-PHE, DQN with epsilon-greedy exploration, Noisy-Net DQN [1]} and Bootstrapped DQN [2]}. Our experiments are based on the baseline implementations of [3]}. As neural network, we used a multi-layer perceptron with hidden layers fixed to \([32,32]\) . The size of the replay buffer was \(10,000\) . The weights of neural networks were optimized by Adam [4]} with gradient clip 5. We used a batch size of 32. The target network was updated every 100 steps. The best learning rate was chosen from \([10^{-3}, 5\times 10^{-4}, 10^{-4}]\) . For LSVI-PHE, we set \(M = 8\)  and we chose the best value of \(\sigma \)  from \([10^{-4}, 10^{-3}, 10^{-2}]\) . Results are shown in Figure REF .
<FIGURE>