Surprisal and Bayesian Surprise. The surprisal, or information content, of a random variable is defined as the negative logarithm of its probability distribution. In an MDP, at time step \(t\) , we can define the surprisal with respect to next-step state as \(-\log p(s_{t+1}|s_t,a_t)\) . By using a model with parameters \(\theta \)  to fit the transition dynamics of the task, we can define surprisal in terms of the probability estimated by the model, namely \(- \log p_\theta (s_{t+1}|s_t, a_t)\) . Such surprisal signal has been adopted for exploration in several works [1]}, [2]}. One shortcoming of these methods is that a stochastic transition, e.g. rolling a die, will always incur into high surprisal values, despite the model having observed the same transition several times. This problem has been treated in literature as the `NoisyTV problem' [3]}, [4]}.
