Motivated by the objective-mismatch issue,
we develop an iterative offline MBRL method,
alternating between training the dynamic model and the policy to maximize a lower bound of the true expected return.
This lower bound, leading to a weighted MLE objective for the dynamic-model training, is relaxed to a tractable regularized objective for the policy learning.
To train the dynamic model by the proposed objective,
we need to estimate the marginal importance weights (MIW) between the offline-data distribution and the stationary state-action distribution of the current policy [1]}, [2]}.
This estimation tends to be unstable by standard approaches [3]}, [4]}, which require saddle-point optimization.
Instead, we propose a simple yet stable fixed-point-style method for MIW estimation, which can be directly incorporated into our alternating training framework.
With these considerations, our method, offline Alternating Model-Policy Learning (AMPL), performs competitively on a wide range of continuous-control offline RL datasets in the D4RL benchmark [5]}.
These empirical results and ablation study show the efficacy of our proposed algorithmic designs.
