Recent work
has identified extrapolation error as a major challenge for offline
(deep) reinforcement learning
[1]}, [2]}, [3]}, [4]}, [5]}, [6]}, with bootstrapping often highlighted as either the cause or an amplifier of the effect:
The value of missing or under-represented state-action pairs
in the dataset can be over-estimated, either transiently (due to insufficient training or data)
or even asymptotically (due to modelling or dataset bias), resulting in a potentially severely
under-performing acquired policy.
The corrective feedback-loop [7]},
whereby value over-estimation is self-correcting via exploitation during interaction with the environment
(while under-estimation is corrected by exploration),
is critically missing in the offline setting.
