State-of-the-art results in imitation learning are currently held by adversarial methods that iteratively estimate the divergence between student and expert policies and then minimize this divergence to bring the imitation policy closer to expert behavior. Analogous techniques for imitation learning from observations alone (without expert action labels), however, have not enjoyed the same ubiquitous successes. 
 Recent work in adversarial methods for generative models has shown that the measure used to judge the discrepancy between real and synthetic samples is an algorithmic design choice, and that different choices can result in significant differences in model performance. Choices including Wasserstein distance and various $f$-divergences have already been explored in the adversarial networks literature, while more recently the latter class has been investigated for imitation learning. Unfortunately, we find that in practice this existing imitation-learning framework for using $f$-divergences suffers from numerical instabilities stemming from the combination of function approximation and policy-gradient reinforcement learning. In this work, we alleviate these challenges and offer a reparameterization of adversarial imitation learning as $f$-divergence minimization before further extending the framework to handle the problem of imitation from observations only. Empirically, we demonstrate that our design choices for coupling imitation learning and $f$-divergences are critical to recovering successful imitation policies. Moreover, we find that with the appropriate choice of $f$-divergence, we can obtain imitation-from-observation algorithms that outperform baseline approaches and more closely match expert performance in continous-control tasks with low-dimensional observation spaces. With high-dimensional observations, we still observe a significant gap with and without action labels, offering an interesting avenue for future work. Imitation Learning (IL) (Osa et al., 2018) refers to a paradigm of reinforcement learning in which the learning agent has access to an optimal, reward-maximizing expert for the underlying environment. In most work, this access is provided through a dataset of trajectories where each observed state is annotated with the action prescribed by the expert policy. This is often an extremely powerful learning paradigm in contrast to standard reinforcement learning, since not all tasks of interest admit easily-specified reward functions. Additionally, not all environments are amenable to the prolonged and potentially unsafe exploration needed for reward-maximizing agents to arrive at satisfactory policies (Achiam et al., 2017; Chow et al., 2019) . While the traditional formulation of the IL problem assumes access to optimal expert action labels, the provision of such information can often be laborious (in the case of a real, human expert) or incur significant financial cost (such as using elaborate instrumentation to record expert actions). Additionally, this restrictive assumption removes a vast number of rich, observation-only data sources from consideration (Zhou et al., 2018) . To bypass these challenges, recent work (Liu et al., 2018; Torabi et al., 2018a; b; Edwards et al., 2019; has explored what is perhaps a more natural problem formulation in which an agent must recover an imitation policy from a dataset containing only expert observation sequences. While this Imitation Learning from Observations (ILfO) setting carries tremendous potential, such as enabling an agent to learn complex tasks from watching freely available videos on the Internet, it also is fraught with significant additional challenges. In this paper, we show how to incorporate recent advances in generative-adversarial training of deep neural networks to tackle imitation-learning problems and advance the state-of-the-art in ILfO. With these considerations in mind, the overarching goal of this work is to enable sample-efficient imitation from expert demonstrations, both with and without the provision of expert action labels. Figure 1: Evaluating the original f -VIM framework (and its ILfO counterpart, f -VIMO) in the Ant (S ∈ R 111 ) and Hopper (S ∈ R 11 ) domains with the Total Variation distance. f -VIM/VIMO-sigmoid denotes our instantiation of the frameworks, detailed in Sections 4.2 and 4.3. Note that, in both plots, the lines for TV-VIM and TV-VIMO overlap. The rich literature on Generative Adversarial Networks (Goodfellow et al., 2014) has expanded in recent years to include alternative formulations of the underlying objective that yield qualitatively different solutions to the saddle-point optimization problem Dziugaite et al., 2015; Nowozin et al., 2016; . Of notable interest are the findings of Nowozin et al. (2016) who present Variational Divergence Minimization (VDM), a generalization of the generative-adversarial approach to arbitrary choices of distance measures between probability distributions drawn from the class of f -divergences (Ali & Silvey, 1966; . Applying VDM with varying choices of fdivergence, Nowozin et al. (2016) encounter learned synthetic distributions that can exhibit differences from one another while producing equally realistic samples. Translating this idea for imitation is complicated by the fact that the optimization of the generator occurs via policy-gradient reinforcement learning (Sutton et al., 2000) . Existing work in combining adversarial IL and f -divergences , despite being well-motivated, fails to account for this difference; the end results (shown partially in Figure 1 , where TV-VIM is the method of , and discussed further in later sections) are imitation-learning algorithms that scale poorly to environments with higher-dimensional observations. In this work, we assess the effect of the VDM principle and consideration of alternative fdivergences in the contexts of IL and ILfO. We begin by reparameterizing the framework of for the standard IL problem. Our version transparently exposes the choices practitioners must make when designing adversarial imitation algorithms for arbitrary choices of f -divergence. We then offer a single instantiation of our framework that, in practice, allows stable training of good policies across multiple choices of f -divergence. An example is illustrated in Figure 1 where our methods (TV-VIM-sigmoid and TV-VIMO-sigmoid) result in significantly superior policies. We go on to extend our framework to encapsulate the ILfO setting and examine the efficacy of the resulting new algorithms across a range of continuous-control tasks in the MuJoCo (Todorov et al., 2012) domain. Our empirical results validate our framework as a viable unification of adversarial imitation methods under the VDM principle. With the assistance of recent advances in stabilizing regularization for adversarial training (Mescheder et al., 2018) , improvements in performance can be attained under an appropriate choice of f -divergence. However, there is still a significant performance gap between the recovered imitation policies and expert behavior for tasks with high dimensional observations, leaving open directions for future work in developing improved ILfO algorithms. To highlight the importance of carefully selecting the variational function activation g f and validate our modifications to the f -VIM framework, we present results in Figure 2 comparing to the original f -VIM framework of and its natural ILfO counterpart. Activation functions for the original methods are chosen according to the choices outlined in ; Nowozin et al. (2016) . In our experiments using the KL and reverse KL divergences, we found that none of the trials reached completion due to numerical instabilities caused by exploding policy gradients. Consequently, we only present results for the Total Variation distance. We observe that under the original f -GAN activation selection, we fail to produce meaningful imitation policies with learning stagnating after 100 iterations or less. As previously mentioned, we suspect that this stems from the use of tanh with TV leading to a dissipating reward signal. We present results in Figure 3 to assess the utility of varying the choice of divergence in f -VIM and f -VIMO across each domain. In considering the impact of f -divergence choice, we find that most of the domains must be examined in isolation to observe a particular subset of f -divergences that stand out. In the IL setting, we find that varying the choice of f -divergence can yield different learning curves but, ultimately, produce near-optimal (if not optimal) imitation policies across all domains. In contrast, we find meaningful choices of f -divergence in the ILfO setting including {KL, TV} for Hopper, RKL for HalfCheetah, and {GAN, TV} for Walker. We note that the use of discriminator regularization per Mescheder et al. (2018) is crucial to achieving these performance gains, whereas the regularization generally fails to help performance in the IL setting. This finding is supportive of the logical intuition that ILfO poses a fundamentally more-challenging problem than standard IL. As a negative result, we find that the Ant domain (the most difficult environment with S ⊂ R 111 and A ⊂ R 8 ) still poses a challenge for ILfO algorithms across the board. More specifically, we observe that discriminator regularization hurts learning in both the IL and ILfO settings. While the choice of RKL does manage to produce a marginal improvement over GAIFO, the gap between existing stateof-the-art and expert performance remains unchanged. It is an open challenge for future work to either identify the techniques needed to achieve optimal imitation policies from observations only or characterize a fundamental performance gap when faced with sufficiently large observation spaces. In Figure 4 , we vary the total number of expert demonstrations available during learning and observe that certain choices of f -divergences can be more robust in the face of less expert data, both in the IL and ILfO settings. We find that KL-VIM and TV-VIM are slightly more performant than GAIL when only provided with a single expert demonstration. Notably, in each domain we see that certain choices of divergence for f -VIMO do a better job of residing close to their f -VIM counterparts suggesting that future improvements may come from examining f -divergences in the small-data regime. This idea is further exemplified when accounting for results collected while using discriminator regularization (Mescheder et al., 2018) . We refer readers to the Appendix for the associated learning curves. Our work leaves many open directions for future work to close the performance gap between student and expert policies in the ILfO setting. While we found the sigmoid function to be a suitable instantiation of our framework, exploring alternative choices of variational function activations could prove useful in synthesizing performant ILfO algorithms. Alternative choices of f -divergences could lead to more substantial improvements than the choices we examine in this paper. Moreover, while this work has a direct focus on f -divergences, Integral Probability Metrics (IPMs) (Müller, 1997; ) represent a distinct but well-established family of divergences between probability distributions. The success of Total Variation distance in our experiments, which doubles as both a fdivergence and IPM (Sriperumbudur et al., 2009) , is suggestive of future work building IPM-based ILfO algorithms . In this work, we present a general framework for imitation learning and imitation learning from observations under arbitrary choices of f -divergence. We empirically validate a single instantiation of our framework across multiple f -divergences, demonstrating that we overcome the shortcomings of prior work and offer a wide class of IL and ILfO algorithms capable of scaling to larger problems. (Schaal, 1997; Atkeson & Schaal, 1997; Argall et al., 2009) , where an agent must leverage demonstration data (typically provided as trajectories, each consisting of expert state-action pairs) to produce an imitation policy that correctly captures the demonstrated behavior. Within the context of LfD, a finer distinction can be made between behavioral cloning (BC) (Bain & Sommut, 1999; Pomerleau, 1989) and inverse reinforcement learning (IRL) (Ng et al.; Abbeel & Ng, 2004; Syed & Schapire, 2007; Ziebart et al., 2008; Finn et al., 2016; Ho & Ermon, 2016) approaches; BC approaches view the demonstration data as a standard dataset of input-output pairs and apply traditional supervisedlearning techniques to recover an imitation policy. Alternatively, IRL-based methods synthesize an estimate of the reward function used to train the expert policy before subsequently applying a reinforcement-learning algorithm (Sutton & Barto, 1998; Abbeel & Ng, 2004) to recover the corresponding imitation policy. Although not a focus of this work, we also acknowledge the myriad of approaches that operate at the intersection of IL and reinforcement learning or augment reinforcement learning with IL (Rajeswaran et al., 2017; Hester et al., 2018; Salimans & Chen, 2018; Sun et al., 2018; Borsa et al., 2019; Tirumala et al., 2019) . While BC approaches have been successful in some settings (Niekum et al., 2015; Giusti et al., 2016; Bojarski et al., 2016) , they are also susceptible to failures stemming from covariate shift where minute errors in the actions of the imitation policy compound and force the agent into regions of the state space not captured in the original demonstration data. While some preventative measures for covariate shift do exist (Laskey et al., 2017b) , a more principled solution can be found in methods like DAgger (Ross et al., 2011) and its descendants (Ross & Bagnell, 2014; Sun et al., 2017; Le et al., 2018 ) that remedy covariate shift by querying an expert to provide on-policy action labels. It is worth noting, however, that these approaches are only feasible in settings that admit such online interaction with an expert (Laskey et al., 2016) and, even then, failure modes leading to poor imitation policies do exist (Laskey et al., 2017a) . The algorithms presented in this work fall in with IRL-based approaches to IL. Early successes in this regime tend to rely on hand-engineered feature representations for success (Abbeel & Ng, 2004; Ziebart et al., 2008; Levine et al., 2011) . Only in recent years, with the aid of deep neural networks, has there been a surge in the number of approaches that are capable of scaling to the raw, highdimensional observations found in real-world control problems (Finn et al., 2016; Ho & Ermon, 2016; Duan et al., 2017; Li et al., 2017; Fu et al., 2017; Kim & Park, 2018) . Our work focuses attention exclusively on adversarial methods for their widespread effectiveness across a range of imitation tasks without requiring interactive experts (Ho & Ermon, 2016; Li et al., 2017; Fu et al., 2017; Kostrikov et al., 2018) ; at the heart of these methods is the Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016) approach which produces high-fidelity imitation policies and achieves state-of-the-art results across numerous continuous-control benchmarks by leveraging the expressive power of Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) for modeling complex distributions over a high-dimensional support. From an IRL perspective, GAIL can be viewed as iteratively optimizing a parameterized reward function (discriminator) that, when used to optimize an imitation policy (generator) via policy-gradient reinforcement learning (Sutton et al., 2000) , allows the agent to shift its own behavior closer to that of the expert. From the perspective of GANs, this is achieved by discriminating between the respective distributions over state-action pairs visited by the imitation and expert policies before training a generator to fool the discriminator and induce a state-action visitation distribution similar to that of the expert. While a large body of prior work exists for IL, numerous recent works have drawn attention to the more challenging problem of imitation learning from observation (Sermanet et al., 2017; Liu et al., 2018; Goo & Niekum, 2018; Kimura et al., 2018; Torabi et al., 2018a; b; Edwards et al., 2019; . In an effort to more closely resemble observational learning in humans and leverage the wealth of publicly-available, observation-only data sources, the ILfO problem considers learning from expert demonstration data where no expert action labels are provided. Many early approaches to ILfO use expert observation sequences to learn a semantic embedding space so that distances between observation sequences of the imitation and expert policies can serve as a cost signal to be minimized via reinforcement learning (Gupta et al., 2017; Sermanet et al., 2017; Liu et al., 2018 choice and the multimodality of the expert trajectory distribution, we provide an empirical evaluation of their f -VIM framework across a range of continous control tasks in the Mujoco domain (Todorov et al., 2012) . Empirically, we find that some of the design choices f -VIM inherits from the original f -GAN work are problematic when coupled with adversarial IL and training of the generator by policy-gradient reinforcement learning, instead of via direct backpropagation as in traditional GANs. Consequently, we refactor their framework to expose this point and provide one practical instantiation that works well empirically. We then go on to extend the f -VIM framework to the IFO problem (f -VIMO) and evaluate the resulting algorithms empirically against the state-of-the-art, GAIFO.
The overall goal of this work is to enable sample-efficient imitation from expert demonstrations, both with and without the provision of expert action labels, through the use of f-divergences.
{"src": "scitldr-AIC_train", "paper_id": "SyxDXJStPS"}