A diagram of the RiverSwim environment is shown in the Appendix. RiverSwim consists of \(\)  states lined up in a chain. The agent begins in the leftmost state \(s_1\)  and has the choice of swimming to the left or to the right at each state. The agent's goal is to maximize its return by trying to reach the rightmost state which has the highest reward. Swimming to the left, with the current, transitions the agent to the left deterministically. Swimming to the right, against the current, stochastically transitions the agent and has relatively high probability of moving right toward the goal state. However, because the current is strong there is a high chance the agent will stay in the current state and a low chance the agent will get swept up in the current and transition to the left.
Thus, smart exploration is required to learn the optimal policy in this environment.
We experiment with the variant of RiverSwim where \(= 12\)  and \(H = 40\) . For this experiment, we swept over the exploration parameters in both LSVI-UCB [1]} and LSVI-PHE and report the best performing run on a 12 state RiverSwim. LSVI-UCB computes confidence widths of the following form \(\beta \Vert \phi (s,a) \Vert _{\Sigma ^{-1}}\)  where \(\phi (s,a) \in \mathbb {R}^d\)  are the features for a given state-action pair and \(\Sigma \in \mathbb {R}^{d \times d}\)  is the empirical covariance matrix. We sweep over \(\beta \)  for LSVI-UCB and \(\sigma ^2\)  for LSVI-PHE, where \(M\)  is chosen according to our theory (Theorem REF ). We sweep over these parameters to speed up learning as choosing the theoretically optimal choices for \(\beta \)  and \(\sigma ^2\)  often leads to a more conservative exploration policy which is slow to learn. As shown in Figure REF , the best performing LSVI-PHE achieves similar performance to the best performing LSVI-UCB on the 12 state RiverSwim environment.
<FIGURE>