One of the fundamental challenges of imitation learning (or any sequential prediction task where the learner consumes some function of its own prior predictions) is the likelihood of significant covariate shift between training-time data and test-time observations [1]}. In short, this happens because the learner might end up in states not seen in the demonstrations and is thus unsure how to act.
Early work in this area includes that of [2]} in the natural language processing and that of [3]} in imitation learning and robotics, both of which come to the preceding conclusion.
Recent work by [4]} shows that there are actually more than one regime of covariate shift. In the “easy” regime where the expert is realizable, off-policy methods like behavior cloning match expert performance when data and model capacity are large enough. However, in harder regimes where the expert is non-realizable due to model misspecification, off-policy methods compound in error and one must rely on on-policy methods, either those that require an interactive expert [3]} or an interactive simulator [6]}, [7]}.
