To address the issues above, we fit a set of noise conditioned distributions, \(\tilde{p}_{E}(s_{t+1}|s_{t}, h)\)  where \( h \sim \mathrm {Uniform}\left[0, h_{max}\right]\)  represents the noise level – the magnitude of zero-mean noise added to the training data. At training time, we draw a noise level \(h\)  and two zero-mean noise samplese.g. Normal or Cauchy distributed \(\mathbf {\epsilon _s}\)  and \(\mathbf {\epsilon _{s^{\prime }}}\)  for each expert transition \((\mathbf {s_E}, \mathbf {s_E^{\prime }})\) . We set \(\mathbf {\tilde{s}_E } \leftarrow \mathbf {s_E }+ \mathbf {h\epsilon _s}\)  and \(\mathbf {\tilde{s}_E^{\prime }} \leftarrow \mathbf {s_E^{\prime }} + \mathbf {h\epsilon _{s^{\prime }}}\)  and fit our model to maximize \(\log p(\mathbf {\tilde{s}_E^{\prime }} | \mathbf {\tilde{s}_E}, h)\) . At test time, with \(h=0\)  we recover the noise-free fitted distribution \(\tilde{p}_{E}\) , while with \(h=h_{max}\)  we get a distribution closer in shape to the distribution of the additive noise, as it then dominates over \(p_E\) . Any intermediate value for \(h\)  smoothly interpolates between the two. Since the sampled noise is zero-mean, transitions close to the dataset \(D_E\)  will have the highest log-probability, irrespective of the noise level, providing a signal with tunable smoothness that is useful for policy search. In Appendix REF  we show an example of varying the noise level \(h\) . Noise regularization for density estimators has been studied in [1]}, and noise-conditioned normalizing flows have previously been applied to 3D data [2]}. Previous work restricts the noise level to \(h \approx 0\)  at test time however, while we actively use the set of noise levels \(h \in [h_{min}, h_{max}]\)  to control the smoothness of our optimization objective, giving the agent a usable signal at all times during policy optimization.
