As in [1]}, the input to our dynamic model is \(\left((s,a) - \mu _{(s,a)}\right) / \sigma _{(s,a)}\) , where \(\mu _{(s,a)}\)  and \(\sigma _{(s,a)}\)  are respectively coordinate-wise mean and standard deviation of \((s,a)\)  in the offline dataset.
We follow [2]} and [3]} to define the learning target as \(\left(\left(r, \Delta s\right) - \mu _{\left(r, \Delta s\right)}\right) / \sigma _{\left(r, \Delta s\right)}\) , where \(\Delta s\)  denote \(s^{\prime } - s\)  , \(\mu _{\left(r, \Delta s\right)}\)  and \(\sigma _{\left(r, \Delta s\right)}\)  denote the coordinate-wise mean and standard deviation of \(\left(r, \Delta s\right)\)  in the offline dataset.
As in prior work using Gaussian probabilistic ensemble on model-based RL [4]}, [5]}, [1]}, [7]}, [8]}, we use a double-head architecture for our dynamic model, where the two output heads represent the mean and log-standard-deviation of the normal distribution of the predicted output, respectively.
We augment the maximum likelihood loss in [1]} as the weighted maximum likelihood loss, independently for each model in the ensemble.
Specifically, the dynamic model is initially trained using un-weighted maximum likelihood loss.
We then employ the warm-start strategy to start each subsequent weighted re-training from the current dynamic model, with the weights \(\omega (s,a)\)  from the latest marginal importance weight network.
To unify the learning rate across each run of model (re-)training, the weights \(\omega (s,a)\) 's are normalized so that their mean is 1 across the offline dataset.
