
Our approach requires fitting a model of \(p_E(s_{t+1}|s_t)\) , using a dataset of demonstrations \(D_E\) . We use a normalizing flow model to fit \(p_E\) , a very powerful and expressive type of density estimator. While well-suited for our purpose, these models are known to overfit with little data, leading to poor out of distribution generalization [1]}, [2]}. Since we aim to use this density model as a reward model for an RL optimizer, we also want it to be suitable for policy optimization. This means producing reasonably low probabilities for observations that are far from the expert data and likely encountered during policy optimization, while resulting in a smooth optimization landscape. Given that \(D_E\)  only covers a subset of all possible behaviours that could be encountered during optimization, we have little control on the predicted log-probability density for non-expert behaviour. This results in a noisy, and possibly biased signal for policy optimization outside the support of the training datasetThis same issue is encountered in GAN training [3]}, [4]} and prompted solutions such as label smoothing [5]}, and Wasserstein critics [6]}; too sharp a learning signal leads to poor training signals for the generator. It is also discussed in [7]} in the context of probability distillation..
