[Proof of Theorem REF ] Consider the Cliff problem of [1]}. There is no hidden context in this problem so \(\delta _{\textrm {off}}(t) = 0\) . Let the learner take the action that puts them at the bottom of the cliff at timestep \(t\)  with probability \(\frac{1}{t+1}\) , giving us \(\epsilon _{\textrm {off}}(t) = \frac{1}{t+1}\) . Note that \(\epsilon _{\textrm {off}}(t)\)  decays to 0 but \(\Sigma _{\textrm {off}}(t)\)  does not as the harmonic series diverges. Once the learner falls off the cliff, they recieve no reward for the rest of the horizon. This means that
\(\frac{1}{T}(J(\pi ^E) - J(\pi )) = \frac{1}{T} \sum _{t=1}^T \frac{T-t}{t+1} = \sum _{t=1}^T \frac{1}{t+1} (1 - \frac{t}{T}) = \sum _{t=1}^T \epsilon _{\textrm {off}}(t) (1 - \frac{t}{T}).\) 
