Intrinsically motivated goal exploration algorithms enable machines to discover repertoires of policies that produce a diversity of effects in complex environments. These exploration algorithms have been shown to allow real world robots to acquire skills such as tool use in high-dimensional continuous state and action spaces. However, they have so far assumed that self-generated goals are sampled in a specifically engineered feature space, limiting their autonomy. In this work, we propose an approach using deep representation learning algorithms to learn an adequate goal space. This is a developmental 2-stage approach: first, in a perceptual learning stage, deep learning algorithms use passive raw sensor observations of world changes to learn a corresponding latent space; then goal exploration happens in a second stage by sampling goals in this latent space. We present experiments with a simulated robot arm interacting with an object, and we show that exploration algorithms using such learned representations can closely match, and even sometimes improve, the performance obtained using engineered representations. Spontaneous exploration plays a key role in the development of knowledge and skills in human children. For example, young children spend a large amount of time exploring what they can do with their body and external objects, independently of external objectives such as finding food or following instructions from adults. Such intrinsically motivated exploration BID6 BID17 BID31 ) leads them to make ratcheting discoveries, such as learning to locomote or climb in various styles and on various surfaces, or learning to stack and use objects as tools. Equipping machines with similar intrinsically motivated exploration capabilities should also be an essential dimension for lifelong open-ended learning and artificial intelligence.In the last two decades, several families of computational models have both contributed to a better understanding of such exploration processes in infants, and how to apply them efficiently for autonomous lifelong machine learning . One general approach taken by several research groups BID1 BID4 BID16 has been to model the child as intrinsically motivated to make sense of the world, exploring like a scientist that imagines, selects and runs experiments to gain knowledge and control over the world. These models have focused in particular on three kinds of mechanisms argued to be essential and complementary to enable machines and animals to efficiently explore and discover skill repertoires in the real world BID10 : embodiment 1 , intrinsic motivation and social guidance 3 . This article focuses on challenges related to learning goal representations for intrinsically motivated exploration, but also leverages models of embodiment, through the use of parameterized Dynamic Movement Primitives controllers BID20 and social guidance, through the use of observations of another agent.Given an embodiment, intrinsically motivated exploration 4 consists in automatically and spontaneously conducting experiments with the body to discover both the world dynamics and how it can be controlled through actions. Computational models have framed intrinsic motivation as a family of mechanisms that self-organize agents exploration curriculum, in particular through generating and selecting experiments that maximize measures such as novelty BID0 BID48 , predictive information gain BID25 , learning progress BID43 BID21 , compression progress BID44 , competence progress , predictive information BID26 or empowerment BID41 . When used in the Reinforcement Learning (RL) framework (e.g. BID48 BID43 BID21 BID4 ), these measures have been called intrinsic rewards, and they are often applied to reward the "interestingness" of actions or states that are explored. They have been consistently shown to enable artificial agents or robots to make discoveries and solve problems that would have been difficult to learn using a classical optimization or RL approach based only on the target reward (which is often rare or deceptive) BID11 BID47 . Recently, they have been similarly used to guide exploration in difficult deep RL problems with sparse rewards, e.g. BID5 BID19 BID49 BID35 .However , many of these computational approaches have considered intrinsically motivated exploration at the level of micro-actions and states (e.g. considering low-level actions and pixel level perception). Yet, children 's intrinsically motivated exploration leverages abstractions of the environments, such as objects and qualitative properties of the way they may move or sound, and explore by setting self-generated goals BID53 , ranging from objects to be reached, toy towers to be built, or paper planes to be flown. A computational framework proposed to address this higher-level form of exploration has been Intrinsically Motivated Goal Exploration Processes (IMGEPs) BID2 BID15 , which is closely related to the idea of goal babbling BID39 . Within this approach , agents are equipped with a mechanism enabling them to sample a goal in a space of parameterized goals 5 , before they try to reach it by executing an experiment. Each time they sample a goal, they dedicate a certain budget of experiments time to improve the solution to reach this goal, using lower-level optimization or RL methods for example. Most importantly, in the same time, they take advantage of information gathered during this exploration to discover other outcomes and improve solutions to other goals 6 .This property of cross-goal learning often enables efficient exploration even if goals are sampled randomly in goal spaces containing many unachievable goals. Indeed, generating random goals (including unachievable ones) will very often produce goals that are outside the convex hull of already discovered outcomes, which in turn leads to exploration of variants of known corresponding policies, pushing the convex hull further. Thus, this fosters exploration of policies that have a high probability to produce novel outcomes without the need to explicitly measure novelty. This explains why forms of random goal exploration are a form of intrinsically motivated exploration. However, more powerful goal sampling strategies exist. A particular one consists in using meta-learning algorithms to monitor the evolution of competences over the space of goals and to select the next goal to try, according to the expected competence progress resulting from practicing it . This enables to automate curriculum sequences of goals of progressively increasing complexity, which has been shown to allow high-dimensional real world robots to acquire efficiently repertoires of locomotion skills or soft object manipulation , or advanced forms of nested tool use BID15 Here a goal is not necessarily an end state to be reached, but can characterize certain parameterized properties of changes of the world, such as following a parameterized trajectory. 6 E.g. while learning how to move an object to the right, they may discover how to move it to the left.sampling them randomly BID9 BID27 or adaptively BID13 ).Yet, a current limit of existing algorithms within the family of Intrinsically Motivated Goal Exploration Processes is that they have assumed that the designer 7 provides a representation allowing the autonomous agent to generate goals, together with formal tools used to measure the achievement of these goals (e.g. cost functions). For example, the designer could provide a representation that enables the agent to imagine goals as potential continuous target trajectories of objects BID15 , or reach an end-state starting from various initial states defined in Euclidean space BID13 , or realize one of several discrete relative configurations of objects BID9 ), which are high-level abstractions from the pixels. While this has allowed to show the power of intrinsically motivated goal exploration architectures, designing IMGEPs that sample goals from a learned goal representation remains an open question. There are several difficulties. One concerns the question of how an agent can learn in an unsupervised manner a representation for hypothetical goals that are relevant to their world before knowing whether and how it is possible to achieve them with the agent's own action system. Another challenge is how to sample "interesting" goals using a learned goal representation, in order to remain in regions of the learned goal parameters that are not too exotic from the underlying physical possibilities of the world. Finally, a third challenge consists in understanding which properties of unsupervised representation learning methods enable an efficient use within an IMGEP architecture so as to lead to efficient discovery of controllable effects in the environment.In this paper, we present one possible approach named IMGEP-UGL where aspects of these difficulties are addressed within a 2-stage developmental approach, combining deep representation learning and goal exploration processes:Unsupervised Goal space Learning stage (UGL): In the first phase, we assume the learner can passively observe a distribution of world changes (e.g. different ways in which objects can move), perceived through raw sensors (e.g. camera pixels or other forms of low-level sensors in other modalities). Then, an unsupervised representation learning algorithm is used to learn a lower-dimensional latent space representation (also called embedding) of these world configurations. After training, a Kernel Density Estimator (KDE) is used to estimate the distribution of these observations in the latent space. In this paper, we proposed a new Intrinsically Motivated Goal Exploration architecture with Unsupervised Learning of Goal spaces (IMGEP-UGL). Here, the Outcome Space (also used as Goal Space) representation is learned using passive observations of world changes through low-level raw sensors (e.g. movements of objects caused by another agent and perceived at the pixel level).Within the perspective of research on Intrinsically Motivated Goal Exploration started a decade ago , and considering the fundamental problem of how AI agents can autonomously explore environments and skills by setting their own goals, this new architecture constitutes a milestone as it is to our knowledge the first goal exploration architecture where the goal space representation is learned, as opposed to hand-crafted.Furthermore, we have shown in two simulated environments (involving a high-dimensional continuous action arm) that this new architecture can be successfully implemented using multiple kinds of unsupervised learning algorithms, including recent advanced deep neural network algorithms like Variational Auto-Encoders. This flexibility opens the possibility to benefit from future advances in unsupervised representation learning research. Yet, our experiments have shown that all algorithms we tried (except RGE-RFVAE) can compete with an IMGEP implementation using engineered feature representations. We also showed, in the context of our test environments, that providing to IMGEP-UGL algorithms a target embedding dimension larger than the true dimensionality of the phenomenon can be beneficial through leveraging exploration dynamics properties of IMGEPs. Though we must investigate more systematically the extent of this effect, this is encouraging from an autonomous learning perspective, as one should not assume that the learner initially knows the target dimensionality.Limits and future work. The experiments presented here were limited to a fairly restricted set of environments. Experimenting over a larger set of environments would improve our understanding of IMGEP-UGL algorithms in general. In particular, a potential challenge is to consider environments where multiple objects/entities can be independently controlled, or where some objects/entities are not controllable (e.g. animate entities). In these cases, previous work on IMGEPs has shown that random Goal Policies should be either replaced by modular Goal Policies (considering a modular goal space representation, see BID15 ), or by active Goal Policies which adaptively focus the sampling of goals in subregions of the Goal Space where the competence progress is maximal . For learning modular representations of Goal Spaces, an interesting avenue of investigations could be the use of the Independently Controllable Factors approach proposed in BID51 .Finally, in this paper, we only studied a learning scenario where representation learning happens first in a passive perceptual learning stage, and is then fixed during a second stage of autonomous goal exploration. While this was here motivated both by analogies to infant development and to facilitate evaluation, the ability to incrementally and jointly learn an outcome space representation and explore the world is a stimulating topic for future work.Bernouilli distribution of ξ parameters 13 , and the log likelihood of the dataset D is expressed as: DISPLAYFORM0 with DISPLAYFORM1 . For a binary valued input vector x (i) , the unitary Cost Function to minimize is: DISPLAYFORM2 provided that f θ is the encoder part of the architecture and g φ is the decoding part of the architecture. This Cost Function can be minimized using Stochastic Gradient Descent BID7 , or more advanced optimizers such as Adagrad BID12 ) or Adam (Kingma & Ba, 2015 .Depending on the depth of the network 14 , those architectures can prove difficult to train using vanilla Stochastic Gradient Descent. A particularly successful procedure to overcome this difficulty is to greedily train each pairs of encoding-decoding layers and stacking those to sequentially form the complete network. This procedure, known as stacked AEs, accelerates convergence. But it has shown bad results with our problem, and thus was discarded for the sake of clarity.Variational Auto-Encoders (VAEs) If we assume that the observed data are realizations of a random variable x ∼ p(x|ψ), we can hypothesize that they are conditioned by a random vector of independent factors z ∼ p(z|ψ). In this setting, learning the model would amount to searching the parameters ψ of both distributions. We might use the same principle of maximum likelihood as before to find the best parameters by computing the likelihood log L(D) = N i=1 log p(x (i) |ψ) by using the fact that p(x|ψ) = p(x, z|ψ)dz = p(x|z, ψ)p(z|ψ)dz. Unfortunately, in most cases, this integral is intractable and cannot be approximated by Monte-Carlo sampling in reasonable time.To overcome this problem, we can introduce an arbitrary distribution q(z|x, χ) and remark that the following holds: DISPLAYFORM3 with the Evidence Lower Bound being: DISPLAYFORM4 Looking at Equation FORMULA10 , we can see that since the KL divergence is non-negative, L(q, ψ) ≤ log p(x|ψ) − D KL ([q(z|x, χ) p(z|x, ψ)] whatever the q distribution, hence the name of Evidence Lower Bound (ELBO). Consequently, maximizing the ELBO have the effect to maximize the log likelihood, while minimizing the KL-Divergence between the approximate q(z|x) distribution, and the true unknown posterior p(z|x, ψ). The approach taken by VAEs is to learn the parameters of both conditional distributions p(x|z, ψ) and q(z|x, χ) as non-linear functions. Under some restricted conditions, Equation (6) can be turned into a valid cost function to train a neural network. First, we hypothesize that q(z|x, χ) and p(z|ψ) follow Multivariate Gaussian distributions with diagonal covariances, which allows us to compute the b term in closed form. Second, using the Gaussian assumption on q, we can reparameterize the inner sampling operation by z = µ + σ 2 with ∼ N (0, I). Using this trick, the Path-wise Derivative estimator can be used for the a member of the ELBO. Under those conditions, and assuming that p(x|ψ) follows a Multivariate Bernouilli distribution , we can write the cost function used to train the neural network as: DISPLAYFORM5 where f χ represents the encoding and sampling part of the architecture and g ψ represents the decoding part of the architecture. In essence, this derivation simplifies to the initial cost function used in AEs augmented by a term penalizing the divergence between q(z|x, χ) and the assumed prior that p(x|ψ) = N (0, I).Normalizing Flow overcomes the problem stated earlier, by permitting more expressive prior distributions (Rezende & Mohamed, 2015) . It is based on the classic rule of change of variables for random variables. Considering a random variable z 0 ∼ q(z 0 ), and an invertible transformation t : DISPLAYFORM6 We can then directly chain different invertible transformations t 1 , t 2 , . . . , t K to produce a new ran- DISPLAYFORM7 . In this case, we have: DISPLAYFORM8 This formulation is interesting because the Law Of The Unconscious Statistician allows us to compute expectations over q(z k ) without having a precise knowledge of it: DISPLAYFORM9 provided that h does not depends on q(z k ). Using this principle on the ELBO allows us to derive the following: DISPLAYFORM10 This is nothing more than the regular ELBO with an additional term concerning the log-determinant of the transformations. In practice, as before, we use p(z 0 ) = N (z 0 ; 0, I), and q(z 0 |x) = N (z 0 ; µ(x), diag(σ(x)2 )). We only have to find out parameterized transformations t, whose parameters can be learned and have a defined log-determinant . Using radial flow, which is expressed as: DISPLAYFORM11 where r = |z − c|, h(α, r) = 1 α+r and α, β, c are learnable parameters of the transformation, our cost function can be written as: DISPLAYFORM12 (1 + log(σ( DISPLAYFORM13 provided that f χ represents the encoding, sampling ad transforming part of the architecture, g ψ represents the decoding part of the architecture, and β k , α k , c k are the parameters of the different transformations. Other types of transformations have been proposed lately. The Householder flow BID52 ) is a volume preserving transformation, meaning that its log determinant equals 1, with the consequence that it can be used with no modifications of the loss function. A more convoluted type of transformations based on a masked autoregressive auto-encoder, the Inverse Autoregressive Flow, was proposed in BID23 . We did not explore those two last approaches.
We propose a novel Intrinsically Motivated Goal Exploration architecture with unsupervised learning of goal space representations, and evaluate how various implementations enable the discovery of a diversity of policies.
{"src": "scitldr-AIC_train", "paper_id": "S1DWPP1A-"}