We formulate the problem of learning from observations as a distribution matching problem: we want to find the policy parameters that result in observation sequences that are similar to those in a dataset of expert demonstrations. This is similar to recent work [1]}, [2]}, [3]} that uses adversarial optimization. Our approach differs in that we fit a density model on expert observation sequences, which we then use to produce rewards for policy search with RL optimizers, decoupling the policy optimization from the reward learning processes. To the best of our knowledge the closest approaches to ours are [4]} where they also use neural density estimators although for occupancy measure estimation and address state-action imitation instead of state-only. Their formulation requires a discounted infinite horizon agent however, as opposed to our undiscounted finite horizon RL optimization. And [5]}, which also addresses the lack of smoothness of the KL-divergence objectives, by opting to minimize Wasserstein distance however instead of noise-expanded distributions, although unlike ours their reward signal is non-stationary.
Our imitation signal is non-adversarial, stationary and reusable for downstream tasks.
