---
abstract: 'Classically, imitation learning algorithms have been developed for idealized situations, e.g., the demonstrations are often required to be collected in the exact same environment and usually include the demonstrator’s actions. Recently, however, the research community has begun to address some of these shortcomings by offering algorithmic solutions that enable imitation learning from observation (), e.g., learning to perform a task from *visual* demonstrations that may be in a different environment and do not include actions. Motivated by the fact that agents often also have access to their own internal states (i.e., *proprioception*), we propose and study an  algorithm that leverages this information in the policy learning process. The proposed architecture learns policies over proprioceptive state representations and compares the resulting trajectories visually to the demonstration data. We experimentally test the proposed technique on several MuJoCo domains and show that it outperforms other imitation from observation algorithms by a large margin.'
author:
- Faraz Torabi$^1$
- |
    Garrett Warnell$^2$Peter Stone$^1$ $^1$The University of Texas at Austin\
    $^2$Army Research Laboratory\
    {faraztrb, pstone}@cs.utexas.edu, garrett.a.warnell.civ@mail.mil
bibliography:
- 'ijcai19.bib'
title: Imitation Learning from Video by Leveraging Proprioception
---

Introduction
============

Imitation learning [@schaal1997learning; @argall2009survey; @osa2018algorithmic] is a popular method by which artificial agents learn to perform tasks. In the imitation learning framework, an expert agent provides demonstrations of a task to a learning agent, and the learning agent attempts to mimic the expert. Unfortunately, many existing imitation learning algorithms have been designed for idealized situations, e.g., they require that the demonstrations be collected in the exact same environment as the one that the imitator is in and/or that the demonstrations include the demonstrator’s actions, i.e., the internal control signals that were used to drive the behavior. These limitations result in the exclusion of a large amount of existing resources, including a large number of videos uploaded to the internet. For example, 300 hours of video are uploaded to YouTube every minute[^1], many of which include different types of tasks being performed. Without new imitation learning techniques, none of this video can be used to instruct artificial agents.

Fortunately, the research community has recently begun to focus on addressing the above limitations by considering the specific problem of imitation from observation () [@liu2017imitation; @torabi2019recent].  considers situations in which agents attempt to learn tasks by observing demonstrations that contain only state information (e.g., videos). Among  algorithms that learn tasks by watching videos, most attempt to learn imitation policies that rely solely on self-observation through video, i.e., they use a convolutional neural network (CNN) that maps images of themselves to actions. However, in many cases, the imitating agent also has access to its own *proprioceptive* state information, i.e., direct knowledge of itself such as the joint angles and torques associate with limbs. In this paper, we argue that  algorithms that ignore this information are missing an opportunity that could potentially improve the performance and the efficiency of the learning process. Therefore, we are interested here in  algorithms that can make use of both visual and proprioceptive state information.

In this paper, we build upon our previous work [@torabi2018generative] proposing an algorithm that uses a -like [@goodfellow2014generative] architecture to learn tasks perform  directly from videos. Unlike our prior work, however, our method *also* uses proprioceptive information from the imitating agent during the learning process. We hypothesize that the addition of such information will improve both learning speed and the final performance of the imitator, and we test this hypothesis experimentally in several standard simulation domains. We compare our method with other, state-of-the-art approaches that do not leverage proprioception, and our results validate our hypothesis, i.e., the proposed technique outperforms the others by a large margin.

The rest of this paper is organized as follows. In Section \[sec:related-work\], we review related work in imitation from observation. In Section \[sec:background\], we review technical details surrounding Markov decision processes, imitation learning, and . The proposed algorithm is presented in Section \[sec:algorithm\], and we describe the experiments that we have performed in Section \[sec:experiments\].

Related Work {#sec:related-work}
============

In this section, we review research in imitation learning, plan/goal recognition by mirroring, and recent advances in imitation from observation ().

Conventionally, imitation learning is used in autonomous agents to learn tasks from demonstrated [*state-action*]{} trajectories. The algorithms developed for this task can be divided into two general categories, (1) behavioral cloning [@bain1999framework; @ross2011reduction; @daftry2016learning] in which the agents learn a direct mapping from the demonstrated states to the actions, and (2) inverse reinforcement learning (IRL) [@abbeel2004apprenticeship; @bagnell2007boosting; @baker2009action] in which the agents first learn a reward function based on the demonstrations and then learn to perform the task using a reinforcement learning ()  [@sutton1998reinforcement] algorithm.

In contrast, imitation from observation () is a framework for learning a task from [*state-only*]{} demonstrations. This framework has recently received a great deal of attention from the research community. The  algorithms that have been developed can be categorized as either (1) model-based, or (2) model-free. Model-based algorithms require the agent to learn an explicit model of its environment as part of the imitation learning process. One algorithm of this type is behavioral cloning from observation () [@torabi2018behavioral], in which the imitator learns a dynamics model of its environment using experience collected by a known policy, and then uses this model to infer the missing demonstrator actions. Using the inferred actions, the imitator then computes an imitation policy using behavioral cloning [@bain1995a]. Another approach of this type is reinforced inverse dynamics modeling (RIDM) [@torabi2019RIDM] which also learns a model of its environment using an exploration policy and then it further optimizes the model using a sparse reward function. It is shown that in some experiments the algorithm can even outperform the expert. Another model-based approach to  is imitating latent policies from observation () [@edwards2018imitating]. Given the current state of the expert, this approach predicts the next state using a latent policy and a forward dynamics model. It then uses the difference between the predicted state and the actual demonstrator next state to update both the model and the imitation policy. Afterwards, the imitator interacts with its environment to correct the action labels.

Model-free algorithms, on the other hand, do not require any sort of model to learn imitation policies. One set of approaches of this type learns a time-dependent representation of tasks and then relies on hand-designed, time-aligned reward functions to learn the task via . For example, @sermanet2017time propose an algorithm that learns an embedding function using a triplet loss function that seeks to push states that are close together in time closer together in the embedded space, while pushing other states further away. @liu2017imitation also propose a new architecture to learn a state representation—specifically, one that is capable of handling viewpoint differences. @gupta2017learning also propose a neural network architecture to try to learn a state representation that can overcome possible embodiment mismatch between the demonstrator and the imitator. Each of these approaches requires multiple demonstrations of the same task to be time-aligned, which is typically not a realistic assumption. @aytar2018playing propose an  algorithm that first learns an embedding using a self-supervised objective, and then constructs a reward function based on the embedding representation difference between the current state of the imitator and a specific checkpoint generated by the visual demonstration. @goo2018learning propose an algorithm that uses a shuffle-and-learn style [@misra2016shuffle] loss in order to train a neural network that can predict progress in the task which can then be used as the reward function.

Another set of model-free algorithms follow a more end-to-end approach to learning policies directly from observations. An algorithm of this type is generative adversarial imitation from observation () [@torabi2018generative], which uses a -like architecture to bring the state transition distribution of the imitator closer to that of the demonstrator. Another approach of this type is the work of @merel2017learning , which is concerned instead with single state distributions. @stadie2017third also propose an algorithm in this space that combines adversarial domain confusion methods [@ganin2016domain] with adversarial imitation learning algorithms in an attempt to overcome changes in viewpoint. The method we propose in this paper also belongs to the category of end-to-end model-free imitation from observation algorithms. However, it is different from the algorithms discussed above in that we explicitly incorporate the imitator’s proprioceptive information in the learning process in order to study the improvement such information can make with respect to the performance and speed of the learning process.

A method that is closely related to imitation from observation is plan/goal recognition through mirroring [@vered2016online; @vered2018towards] in that in it attempts to infer higher-level variables such as the goal or the future plan by observing other agents. However, in plan and goal recognition the observer already has fixed controllers, and then uses these controllers to match/explain the observed agent in order to infer their goal/plan. In imitation from observation, on the other hand, the agent seeks to learn a controller that the observer can use to imitate the observed agent.

Background {#sec:background}
==========

In this section, we establish notation and provide background information about Markov decision processes (s) and adversarial imitation learning.

Notation
--------

We consider artificial learning agents operating in the framework of Markov decision processes (s). An  can be described as a 6-tuple ${\mathcal{M}}= \{{\mathcal{S}}, {\mathcal{A}}, P, r, \gamma\}$, where ${\mathcal{S}}$ and ${\mathcal{A}}$ are state and action spaces, $P(s_{t+1}|s_t, a_t)$ is a function which represents the probability of an agent transitioning from state $s_t$ at time $t$ to $s_{t+1}$ at time $t+1$ by taking action $a_t$, $r:{\mathcal{S}}\times {\mathcal{A}}\rightarrow {\mathcal{R}}$ is a function that represents the reward feedback that the agent receives after taking a specific action at a given state, and $\gamma$ is a discount factor. In the context of the notation established above, we are interested here in learning a policy $\pi:{\mathcal{S}}\rightarrow {\mathcal{A}}$ that can be used to select an action at each state.

In this paper, we refer to $s$ as the *proprioceptive* state, i.e., $s$ is the most basic, internal state information available to the agent (e.g., the joint angles of a robotic arm). Since we are also concerned with visual observations of agent behavior, we denote these observations as $o \in {\mathcal{O}}$, i.e., an image of the agent at time $t$ is denoted as $o_t$. The visual observations of the agent are determined both by the agent’s current proprioceptive state $s$, and also other factors relating to image formation such as camera position. Importantly, due to phenomena such as occlusion, it is not always possible to infer $s$ from $o$ alone.

In imitation learning (), agents do not receive reward feedback $r$. Instead, they have access to expert demonstrations of the task. These demonstrations $\tau_e = \{(s_t,a_t)\}$ are composed of the state and action sequences experienced by the demonstrator. Here, however, we specifically consider the problem of imitation from observation (), in which the agent only has access to sequences of visual observations of the demonstrator performing the task, i.e., $\tau_e = \{o_t\}$.

Adversarial Imitation Learning
------------------------------

Generative adversarial imitation learning () is a recent imitation learning algorithm developed by @ho2016generative that formulates the problem of finding an imitating policy as that of solving the following optimization problem: $$\label{gail}
\begin{split}
\min_{\pi \in \prod} \displaystyle{\max_{D \in (0,1)^{\mathcal{S} \times \mathcal{A}}}} & -\lambda_H H(\pi) + \mathbb{E}_\pi[\log(D(s,a)] +\\ &\mathbb{E}_{\pi_E}[\log(1-D(s,a))]\;,
\end{split}$$ where $H$ is the entropy function, and the discriminator function $D:\mathcal{S} \times \mathcal{A} \rightarrow (0,1)$ can be thought of as a classifier trained to differentiate between the state-action pairs provided by the demonstrator and those experienced by the imitator. The objective in (\[gail\]) is similar the one used in generative adversarial networks (s) [@goodfellow2014generative], and the associated algorithm can be thought of as trying to induce an imitator state-action occupancy measure that is similar to that of the demonstrator. Even more recently, there has been research on methods that seek to improve on  by, e.g., increasing sample efficiency [@kostrikov2018discriminatoractorcritic; @sasaki2018sample] and improving reward representation [@fu2018learning; @qureshi2018adversarial].

The method we propose in this paper is most related to generative adversarial imitation from observation [@torabi2018generative], which models the imitating policy using a randomly-initialized convolutional neural network, executes the policy to generate recorded video of the imitator’s behavior, and then trains a discriminator to differentiate between video of the demonstrator and video of the imitator. Next, it uses the discriminator as a reward function for the imitating agent (higher rewards corresponding to behavior the discriminator classifies as coming from the demonstrator), and uses a policy gradient technique (e.g.,  [@schulman2015trust]) to update the policy. The process repeats until convergence. This algorithm differs from what we propose in that  uses visual data [*both*]{} in the process of discriminator [*and*]{} policy learning. That is, the learned behavior policy maps images $o$ to actions using a convolutional neural network. The technique we propose, on the other hand, leverages proprioceptive information in the policy learning step, instead learning policies that map proprioceptive states $s$ to actions using a multilayer perceptron architecture.

(-3,0.3) rectangle (14, 3.3); at (-1.8, 2) [Proprioceptive\
features ($s_t$)\
(e.g. joint angles)]{}; (input image) at (-0.2,2.4) [![image](walker-trans2.png){height="30mm"}]{}; (.3,2) – (1.1,2); (1.1,1) – (1.1,3) – (2.1,3) – (2.1,1) – (1.1,1); at (1.6, 2) [MLP\
($\pi_\theta$)]{}; (2.1,2) – (2.7,2); (2.9,1.3) – (2.9,2.7) – (3.1,2.7) – (3.1,1.3) – (2.9,1.3); (2.9,2.5) – (3.1,2.5); (2.9,1.5) – (3.1,1.5); (2.9,1.7) – (3.1,1.7); at (3, 1.9) [.]{}; at (3, 2.1) [.]{}; at (3, 2.3) [.]{}; at (3, 1) [$a_t$]{}; (3.3,2) – (3.9,2); (3.9,1.5) – (3.9,2.5) – (4.7,2.5) – (4.7,1.5) – (3.9,1.5); at (4.3, 2) [Env]{}; (4.7,2) – (5.5,2); (input image) at (6.35,2.2) [![image](walker2.png){height="15mm" width="15mm"}]{}; (5.6,2.95) – (5.8,2.75); (7.1,2.95) – (7.3,2.75); (5.6,1.45) – (5.8,1.25); (input image) at (6.55,2) [![image](walker2.png){height="15mm" width="15mm"}]{}; at (6.45, .8) [$\tau_i=\{o_{t-2}:o_{t+1}\}$]{}; (7.5,2) – (7.9,2) – (7.9,2.7) – (8.9,2.7) – (8.9,2.5);

(8.5,2.5) – (10.5,2.5) – (10,1.5) – (9,1.5) – (8.5,2.5); at (9.5, 2) [CNN\
($D_\phi$)]{}; (9.5,1.5) – (9.5,1); at (9.5, 0.75) [v]{};

(input image) at (12.4,2.2) [![image](walker.png){height="15mm" width="15mm"}]{}; (11.65,2.95) – (11.85,2.75); (13.15,2.95) – (13.35,2.75); (11.65,1.45) – (11.85,1.25); (input image) at (12.6,2) [![image](walker.png){height="15mm" width="15mm"}]{}; at (12.5, .8) [$\tau_e=\{o_{t-2}:o_{t+1}\}$]{}; (11.5,2) – (11.1,2) – (11.1,2.7) – (10.1,2.7) – (10.1,2.5);

Proposed Method {#sec:algorithm}
===============

As presented in Section \[sec:background\], we are interested in the problem of imitation from observation (), where an imitating agent has access to visual demonstrations, $\tau_e = \{o_t\}$, of an expert performing a task, and seeks to learn a behavior that is approximately the same as the expert’s. In many previous approaches to this problem, the imitator selects actions on the basis of visual self-observation alone (i.e., using images of itself). We hypothesize that also leveraging available proprioceptive state information, $s$, during the learning process will result in better and faster learning.

Inspired by , our algorithm is comprised of two pieces: (1) a generator, which corresponds to the imitation policy, and (2) a discriminator, which serves as the reward function for the imitator. We model the imitation policy as a multilayer perceptron (MLP), $\pi_\theta$. The imitating agent, being aware of its own proprioceptive features $s$, feeds them into the policy network and receives as output a distribution over actions from which the selected action $a$ can be sampled. The imitator then executes this action and we record a video of the resulting behavior. After several actions have been executed, we have accumulated a collection of visual observations of the imitator’s behavior, $\tau_i = \{o\}$.

Meanwhile, we use a convolutional neural network as a discriminator $D_\phi$. Given visual observations of the demonstrator, $\tau_e$, and observations of the imitator, $\tau_i$, we train the discriminator to differentiate between the data coming from these different sources. Since single video frames lack observability in most cases, we instead stack four frames, $\{o_{t-2}, o_{t-1}, o_t, o_{t+1}\}$, and feed this stack as input to the discriminator.

Initialize policy $\pi_\theta$ randomly Initialize discriminator $D_\phi$ randomly Obtain visual demonstrations $\tau_e=\{o\}$ Execute $\pi_\theta$ and record video observation $\tau_i=\{o\}$ Update the discriminator $D_\phi$ using loss $$\label{eq:disc}
            \begin{split}
                - & \Big(\mathbb{E}_{\tau_i} [\log(D_\phi(o_{t-2}:o_{t+1}))]+ \\& \mathbb{E}_{\tau_e} [\log(1-D_\phi(o_{t-2}:o_{t+1}))]\Big)
            \end{split}$$ Update $\pi_\theta$ by performing updates with gradient steps of

$$\label{eq:step}
            \begin{split}
                \mathbb{E}_{\tau_i} [\nabla_\theta \log \pi_\theta(a|s) Q(s,a)] - \lambda \nabla_\theta H(\pi_\theta),
            \end{split}$$

where $$\label{eq:step}
            \begin{split}
                & Q(\hat{s}_t,\hat{a}_t) = \\& -\mathbb{E}_{\tau_i} [\log(D_\phi(o_{t-2}:o_{t+1}))| s_0=\hat{s}_t, a_0=\hat{a}_t]
            \end{split}$$

We train the discriminator to output values closer to zero for the transitions coming from the expert, and values closer to one for those coming from the imitator. Therefore, the discriminator aims to solve the following optimization problem: $$\label{eq:disc}
\begin{split}
\displaystyle{\max_\phi}& \Big(\mathbb{E}_{\tau_i} [\log(D_\phi(o_{t-2}:o_{t+1}))]+\\& \mathbb{E}_{\tau_e} [\log(1-D_\phi(o_{t-2}:o_{t+1}))]\Big) \; .
\end{split}$$ The lower the value outputted by the discriminator, the higher the chance of the input being from the expert. Recall that the objective for the imitator is to mimic the demonstrator, which can be thought of as fooling the discriminator. Therefore, we use $$\label{eq:rew}
\begin{split}
-\Big(\mathbb{E}_{\tau_i} [\log(D_\phi(o_{t-2}:o_{t+1}))]\Big)\\[5pt]
\end{split}$$ as the reward to update the imitation policy using . In particular, we use proximal policy optimization () [@schulman2017proximal] with gradient steps of $$\label{eq:step}
\begin{split}
\mathbb{E}_{\tau_i} [\nabla_\theta \log \pi_\theta(a|s) Q(s,a)] - \lambda \nabla_\theta H(\pi_\theta),\\[5pt]
\end{split}$$ where $Q(s,a)$ is the state-action value, i.e. the potential reward that the agent receives starting from $s$ and taking action $a$: $$\label{eq:step}
\begin{split}
& Q(\hat{s}_t,\hat{a}_t) = \\&-\mathbb{E}_{\tau_i} [\log(D_\phi(o_{t-2}:o_{t+1}))| s_0=\hat{s}_t, a_0=\hat{a}_t].\\[5pt]
\end{split}$$

As presented, our algorithm uses the visual information in order to learn the reward function by comparing visual data generated by the imitator and the demonstrator. It also takes advantage of proprioceptive state features in the process of policy learning by learning a mapping from those features to actions using a reinforcement learning algorithm. Pseudocode and a diagrammatic representation of our proposed algorithm are presented in Algorithm \[alg\] and Figure \[fig:alg\], respectively.

Experiments {#sec:experiments}
===========

The algorithm introduced above combines proprioceptive state information with video observations in an adversarial imitation learning paradigm. We hypothesize that using the extra state information in the proposed way will lead to both faster imitation learning and better performance on the imitated task when compared to similar techniques that ignore proprioception. In this section, we describe the experimental procedure by which we evaluated this hypothesis, and discuss the results.

Setup
-----

We evaluated our method on a subset of the continuous control tasks available via OpenAI Gym [@1606.01540] and the MuJoCo simulator [@todorov2012mujoco]: MountainCarContinuous, InvertedPendulum, InvertedDoublePendulum, Hopper, Walker2d, HalfCheetah.

To generate the demonstration data, we first trained an expert agents using pure reinforcement learning (i.e., not from imitation). More specifically, we used proximal policy optimization () [@schulman2017proximal] and the ground truth reward function provided by OpenAI Gym. After the expert agents were trained, we recorded $64 \times 64$, $30$-fps video demonstrations of their behavior.

We compared the proposed method with three other imitation from observation algorithms that do *not* exploit the imitator’s proprioceptive state information: Time Contrastive Networks (TCN) [@sermanet2017time], Behavioral Cloning from Observation (BCO) [@torabi2018behavioral], Generative Adversarial Imitation Learning (GAIfO) [@torabi2018generative; @torabi2019adversarial][^2].

Results
-------

We hypothesized that our method would outperform the baselines with respect to two criteria: (1) the final performance of the trained imitator, i.e., how the imitator performs the task compared to the demonstrator (as measured by the ground truth reward functions), and (2) the speed of the imitation learning process as measured by number of learning iterations. The results shown here were generated using ten independent trials, where each trial used a different random seed to initialize the environments, model parameters, etc.

Figure \[fig:performance\] depicts our experimental results pertaining to the first criterion, i.e., the final task performance of trained imitating agents in each domain. The rectangular bars and error bars represent the mean return and the standard error, respectively, as measured over $1000$ trajectories. We report performance using a normalized task score, i.e., scores are scaled in such a way that the demonstrating agent’s performance corresponds to $1.0$ and the performance of an agent with random behavior corresponds to $0.0$. The x-axis represents the number of demonstration trajectories, i.e., videos, available to the imitator. In general, it can be seen that the proposed method indeed outperforms the baselines in almost all cases, which shows that using the available proprioceptive state information can make a remarkable difference in the final task performance achieved by imitation learning. In the particular case of InvertedPendulum, both  and the proposed method achieve a final task performance equal to that of the demonstrator, likely due to the simplicity of the task. However, for the rest of the tasks, it can be clearly seen that the proposed approach performs better than [^3]. Further, we can see that increasing the number of demonstrated trajectories results in increased task performance.

To validate our hypothesis with respect to learning speed, we also studied the transient performance of the various learning algorithms. Because only one other method, , performed as well as the expert in only one domain, InvertedPendulum, Figure \[fig:process\] only depicts the results for these algorithms in that domain. The x-axis shows the number of iterations, i.e., the number of update cycles for both the policy and the discriminator. Since updating the policy requires interaction with the environment, a smaller number of iterations also corresponds to less overhead during the learning process. As shown in the figure, our method converges to expert-level performance much faster than , which supports our hypothesis that leveraging proprioception speeds the imitation learning process.

In Figure \[fig:performance\], we can see that two of the baseline methods— and —do not achieve task performance anywhere near that of the expert.

![The rectangular bars and error bars represent the mean normalized return and the standard error, respectively, as measured over 1000 trials. The normalized values have been scaled in such a way that expert and random performance are $1.0$ and $0.0$, respectively. The x-axis represents the number of available video demonstration trajectories.[]{data-label="fig:performance"}](performance-bar.pdf){width="\linewidth"}

For InvertedPendulum and InvertedDoublePendulum, we suspect that  performs poorly due to possible overfitting of the learned state embedding to the specific demonstrations and, therefore, does not generalize well toward supporting the overall goal of keeping the pendulum balanced above the rod. For Hopper, Walker2d, and HalfCheetah, the poor performance of  may be due to the fact that the tasks are cyclical in nature and therefore not well-suited to the time-dependent learned state embedding.  performs relatively better in MountainCarContinuous, compared to other domains because this domain does have the properties required by . As for , we posit that the low performance is due to the well-known compounding-error issue present in behavioral cloning.

One interesting thing to note is that Walker2d results in larger error bars for our technique than those seen for any of the other domains. We hypothesize that the reason for this is that the video frames provide very poor information regarding the state of the demonstrator—here, the agent has two legs, which sometimes results in occlusion and, therefore, uncertainty regarding which action the agent should take.

Finally, we can see that the proposed technique performs the most poorly in the HalfCheetah domain. We hypothesize that this is due to the speed at which the demonstrator acts: frame-to-frame differences are large, e.g., three to four consecutive frames cover a complete cycle of the agent jumping forwards. This rate of change may make it difficult for our discriminator to extract a pattern of behavior, which, consequently, would make it much more difficult for the agent to move its behavior closer to that of the demonstrator. Therefore, one way that performance might be improved is to increase the frame rate at which the demonstrations are sampled. Another way, as suggested by Figure \[fig:performance\], would be to increase the number of demonstration trajectories beyond what is shown here.

![Performance of imitation agents with respect to the number of iterations (N). Solid colored lines represent the mean return and shaded areas represent standard errors. The returns are scaled so that the performance of the expert and random policies be zero and one, respectively.[]{data-label="fig:process"}](performance-process.pdf)

Conclusion and Future Work
==========================

In this paper, we hypothesized that including proprioception would be beneficial to the learning process in the  paradigm. To test this hypothesis, we presented a new imitation from observation algorithm that leverages both available visual and proprioceptive information. It uses visual information to compare the imitator’s behavior to that of the demonstrator, and uses this comparison as a reward function for training a policy over proprioceptive states. We showed that leveraging this state information can significantly improve both the performance and the efficiency of the learning process.

However, to achieve the end-goal of true imitation from observation, several challenges remain. For example,  algorithms should be able to overcome embodiment mismatch (the imitator and the demonstrator have different embodiments), and viewpoint mismatch (the visual demonstrations are recorded from different viewpoints.). Resolving these limitations is a natural next step for extending this research. Another way to improve upon the proposed method is to attempt to make the training more reliable by incorporating techniques developed to improve the stability of s, such as the work of @arjovsky2017wasserstein . Further, to the best of our knowledge, nobody has been able to deploy -like methods on real robots due to high sample complexity. Therefore, techniques that seek to improve the learning process with respect to this metric should also be investigated further.

Acknowledgments {#acknowledgments .unnumbered}
===============

This work has taken place in the Learning Agents Research Group (LARG) at the Artificial Intelligence Laboratory, The University of Texas at Austin. LARG research is supported in part by grants from the National Science Foundation (IIS-1637736, IIS-1651089, IIS-1724157), the Office of Naval Research (N00014-18-2243), Future of Life Institute (RFP2-000), Army Research Lab, DARPA, Intel, Raytheon, and Lockheed Martin. Peter Stone serves on the Board of Directors of Cogitai, Inc. The terms of this arrangement have been reviewed and approved by the University of Texas at Austin in accordance with its policy on objectivity in research.

[^1]: <https://bit.ly/2quPG6O>

[^2]: The considered domains, methods, and implementations are presented in more detail in the longer version of the paper on arXiv [@torabi2019imitation]

[^3]: Note that the performance of  on Hopper is different from what was presented in the  paper [@torabi2018generative]. We hypothesize that the reason is twofold: (1) different physics engines—MuJoCo is used in this paper, but in the previous work [@torabi2018generative] Pybullet [@coumans2016pybullet] was used , and (2) differences in video appearance—in this work we do not alter the default simulator parameters, whereas in the previous work [@torabi2018generative] some of the parameters were modified such as the colors used in the video frames in order to increase the contrast between the agent and the background.
