Most of the prior offline model-based RL (MBRL) methods [1]}, [2]}, [3]}, [4]}, [5]},
however, first pretrain a one-step forward dynamic model via maximum likelihood estimation (MLE) on the offline dataset,
and then use the learned model to train the policy, without further improving the dynamic model during the policy learning process.
As a result, the objective function used for model training (e.g., MLE) and the objective of model utilization are unrelated with each other.
Specifically, the model is trained to be “simply a mimic of the world,” but is used to improve the performance of the learned policy [6]}, [7]}, [8]}.
Though such a training paradigm is historically rooted [9]}, [10]},
this issue of objective mismatch in the model training and model utilization has been identified as problematic in recent works [7]}, [8]}.
In offline MBRL, this issue is exacerbated, since the learned model can hardly be globally accurate, due to the limited amount of offline data and the complexity of the control tasks.
