For each episode \(k \in [K]\)  an initial state \(x_1^k\)  is chosen randomly.
At each step \(h\)  and episode \(k\)  the agent observes a state \(x_h^k\) , takes action \(a_h^k\) , receives reward \(r_h(x_h^k, a_h^k)\) , and then transitions to its next state drawn from the distribution \(\mathbb {P}_h(\cdot |x_h^k, a_h^k)\) .
The transition dynamics and reward function were chosen to depend on the step \(h\)  for generality and to remain consistent with prior work [1]}.
Note that if dependence on \(h\)  is not required then the transition dynamics and reward function can be set the same for all \(h\)  and all the results shown in this work will still hold.
The episode ends when \(x_{H+1}^k\)  is reached.
