Offline reinforcement learning (offline RL) [1]}, [2]}, where the agents are trained from static and pre-collected datasets, avoids direct interactions with the underlying real-world environment during the learning process.
Unlike traditional online RL,
whose success largely depends on simulator-based trial-and-error [3]}, [4]},
offline RL enables training policies for real-world applications, where it is infeasible or even risky to collect online experimental data, such as robotics, advertisement, or dialog systems [5]}, [6]}, [7]}, [8]}.
Though promising,
it remains challenging to train agents under the offline setting, due to the discrepancy between the distribution of the offline data and the state-action distribution induced by the current learning policy.
With such a discrepancy, directly transferring standard online off-policy RL methods [9]}, [10]}, [11]} to the offline setting tends to be problematic [12]}, [13]}, especially when the offline data cannot sufficiently cover the state-action space [14]}.
To tackle this issue in offline RL,
recent works [15]}, [16]}, [17]} propose to approximate the policy-induced state-action distribution by leveraging a learned dynamic model to draw imaginary rollouts.
These additional synthetic rollouts help mitigate the distributional discrepancy and
stabilize the policy-learning algorithms under the offline setting.
