Objective mismatch in MBRL.    
Our AMPL is related to prior works on online and offline MBRL that mitigate the mismatched model objectives.
In online (off-policy) MBRL, [1]} identify the mismatched objectives between the MLE model-training and the model's usage of improving the control performance.
This paper, however, only proposes an impractical weighted learning loss that requires the optimal trajectory. [2]} design an objective to jointly optimize the policy and the model in online MBRL.
[3]} propose a per-sample diagonal scaling matrix for training a deterministic model in online RL, which improves on [4]} and [5]}.
In offline MBRL, [6]} propose a weighting scheme to learn the transition model.
This method requires the behavior policy and estimates the trajectory-wise importance-sampling weight, which is known to suffer from the “curse of horizon” [7]}.
Thus, this method may not scale to complex high-dimensional offline RL tasks.
[8]} propose a minimax objective for model learning, where the model is optimized over function classes of the MIW and the state-value function.
Most similar to our work, [9]} also use a MIW-weighted MLE objective for model training.
They use a different version of the MIW estimated via multiple full-trajectory rollouts in the learned model.
This strategy may suffer from the inaccuracy of the learned model [10]}, [11]}, [12]} and can be time-consuming.
In this paper, we develop a fixed-point style method to train the MIW that only requires samples from the offline dataset and the initial state-distribution, which can be more stable and efficient.
