Dynamics-Aware Unsupervised Discovery of Skills (Paper Explained)
This RL framework can discover low-level skills all by itself without any reward. Even better, at test time it can compose its learned skills and reach a specified goal without any additional learning! Warning: Math-heavy!

OUTLINE:
0:00 - Motivation
2:15 - High-Level Overview
3:20 - Model-Based vs Model-Free Reinforcement Learning
9:00 - Skills
12:10 - Mutual Information Objective
18:40 - Decomposition of the Objective
27:10 - Unsupervised Skill Discovery Algorithm
42:20 - Planning in Skill Space
48:10 - Conclusion

Paper: https://arxiv.org/abs/1907.01657
Website: https://sites.google.com/view/dads-skill
Code: https://github.com/google-research/dads

Abstract:
Conventionally, model-based reinforcement learning (MBRL) aims to learn a global model for the dynamics of the environment. A good model can potentially enable planning algorithms to generate a large variety of behaviors and solve diverse tasks. However, learning an accurate model for complex dynamical systems is difficult, and even then, the model might not generalize well outside the distribution of states on which it was trained. In this work, we combine model-based learning with model-free learning of primitives that make model-based planning easy. To that end, we aim to answer the question: how can we discover skills whose outcomes are easy to predict? We propose an unsupervised learning algorithm, Dynamics-Aware Discovery of Skills (DADS), which simultaneously discovers predictable behaviors and learns their dynamics. Our method can leverage continuous skill spaces, theoretically, allowing us to learn infinitely many behaviors even for high-dimensional state-spaces. We demonstrate that zero-shot planning in the learned latent space significantly outperforms standard MBRL and model-free goal-conditioned RL, can handle sparse-reward tasks, and substantially improves over prior hierarchical RL methods for unsupervised skill discovery.

Authors: Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, Karol Hausman

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there, take a look at this humanoid right here. It walks from one checkpoint to another checkpoint, and then to the next checkpoint, and so on. And that is its task, it gets reward from walking from checkpoint to checkpoint. Take a look at this. And this is called the end. It also walks from checkpoint to checkpoint. Now we've seen a lot of reinforcement learning algorithms in this environment is called Muko Joe, where you basically teach these little things to walk around. So the what's the impressive part here, the impressive part is that at training time, this and has never ever seen what a checkpoint is, and has never gotten any reward from walking from checkpoint to another checkpoint. Actually, it has never gotten any reward for anything that that is given from the environment, it has discovered the skill of walking by itself. And then at test time, there is no additional learning when it goes from checkpoint to checkpoint, it simply composes the skills that it knows from its unsupervised discovery phase in order to go from checkpoint to checkpoint. So here you can see, this paper basically proposes to learn these skills in a completely unsupervised way at the beginning sort of so in the training phase, it learns these skills, you can see these skills that the humanoid has learned. And then all you have to do at test time is to compose these skills to reach a given goal. And these are the things that the end has learned. Watch out, this is trippy. You can see it has learned various walks various ways walking here. And if you know anything about this environment, it's actually not that easy to make the ant walk by itself. So the the discovery here that these skills that are discovered are various ways of walking is actually already pretty impressive. And the last thing here, this cheetah, of course, also has to has learned to walk back forward kind of jump around and so on. So we're going to dive into this paper. It's called dynamics aware unsupervised discovery of skills by Archie Sharma and other people of Google brain. So this was published at iClear 2020. And on a high level, I already said it's basically proposing to learn unsupervised skills, and then to compose these skills in a model based planning method at test time to reach a given goal without additional training without additional training on the reward that you give at test time. As always, if you like videos like this, you're very welcome to subscribe and share it to everyone you know. Yeah. Okay, let's dive in. So they say, conventionally model based reinforcement learning aims to learn a global model for the dynamics of the environment, which is not exactly true, right? So we have we dive into model based and model free reinforcement learning. Model based reinforcement learning basically means that you have a model of the environment. A example for this is, let's say, tic tac toe. So in tic tac toe, I know, I have nine actions at my disposal. And if I take action, let's say I take action zero, which is to make a let's say I'm the X player. So I take action zero. And if I you know, number my things correctly, then that will result in this state of the world. Okay, so I know exactly how the world will look when I take a given action. And what that allows me to do is that allows me to actually plan. So I cannot plan ahead, I can say what would happen if I took action zero, so I can do this in my mind. And then what would happen if I took action one, I can be like, okay, that's gonna happen. And I can do this with many things. And I can in my mind, continue this and basically roll out the entire games, or and then only do the given action that has led to the best result at the end, right. So this is model model based reinforcement learning means you have a model of the environment, you know what's going to happen when you perform given actions. And you can also combine this with machine learning, like, you know, alpha, alpha go alpha zero or so they have models of the games they're playing, they know what's going to happen. But it's very intractable to basically go down this entire tree and plan out everything. So they combine it with machine learning. It doesn't change that it's a model based. Now in, in opposition to that in model free in in reinforcement learning, what you do, you are these agent, the heirs of the environment, and you simply have to do an action, I do action zero, and the environment just gives you back a reward and the next observation. And it you have basically no clue how the environment will change. If you do something, you all these and all these, these agents do or the classic model free agents do is basically they're trying to have a neural network somewhere in them. And you put the observation in here and outcomes and action. And you can do this in various ways into Q learning or policy gradient or actor critic, and so on. But ultimately, it's simply mapping the optic current observation, and maybe the last few to the best action to take without explicitly modeling what happens in the environment. Now, when they say model based reinforcement learning, what they mean is technically, what you can do, if you're in if you are in the model free, if you're in this situation, what you could do is you could say, Well, since these model based RL techniques tend to work better, I could hear inside the agent, I could try to learn a model of the environment, E prime, and I could try to basically learn what happens in my environment when I do a certain action. And then I could use that model right here, in order to do this planning that I know from up here, okay. So in this case, they go for exactly this they go for, let's learn a model of the environment, this is not an exact model, it's a learned model. And then let's use that to plan. Now, this usually has a bunch of, you know, very, a bunch of things that go against it, namely, if this model right here is bad, then the planning in the model will often accumulate and even exaggerate the errors that are in this model. So it's sometimes very hard to learn a model of the world and then use that for planning. And I've recently done a paper where where curious AI takes denoising auto encoders to regularize exactly such a planning procedure to counter this. And this paper right here is a different approach of combining this learned model. This learned model. So, okay, that was about the first sentence. They say it aims to learn a global model for the dynamics of the environment. A good model can potentially enable planning algorithms to generate a large variety of behaviors and solve diverse tasks, which is true, right? If I have a model of the environment, then I could just use it to plan, I wouldn't even have to do anything fancy anymore, right? If I have a model of how my tic-tac-toe works, I can just plan, plan my way to success. And I can do this alpha zero style. Or if the, if this state tree is small enough, I can actually just use a planner. And I don't even have to, I don't have to do anything anymore. If I have a good model, they say however, learning an accurate model for complex dynamical systems is difficult. And even then the model might not generalize well outside the distribution of states on which it was trained. So this is another problem. If you learn a model, it's only going to be valid in a certain range. Okay. And say in this work, we combine model based learning with model free learning of and the model free learning is of primitives that make model based planning easy. So what they attempt to do is they attempt in an unsupervised fashion to learn so called set of skills. And the set of skills could be something like walk forward, walk backward, stay put, jump. So they attempt to learn things like this in a model free way, that the model is simply asked to come up with these things or the agent is simply asked task to come up with these things. And then in in stage two, a planner can use these skills and decompose a plan. Now this plan here, the special thing about this plan, this planner, it doesn't operate in the space of actions of like small scale actions, it actually operates in the space of these skills right here. So here would be walk forward, walk back. So and with if we have a good enough model of the environment, then will tell us if I walk forward in this situation, what will happen? Okay, so I can walk forward and then after that I could walk backward, what's going to happen? And if I have a good model of the environment over the and the actions now are these macro actions of these skills, then I can use planning to reach my goal. Okay, so the question is, how do we come up with useful skills that the planner can then use? So they need to be somewhat diverse, right? But also, and here is the crucial part and the the sort of contribution of this paper, they say, how can we discover skills, whose outcomes are easy to predict? And this is how they counteract this notion here, that if your environment model is crap, then you're you're, it can't basically can't be used for planning, you'll just make it worse. So what they say is that these things right here, these skills that we learn, we will learn them in a way that make them easy to predict. So it makes it easy to predict what will happen after I do them. So they must be at the same time diverse. So only, you know, if you stay put, it's pretty easy to predict what's going to happen like nothing. Okay, but we're going to see in the exact objective that they have to be sort of diverse. But also, so only one of them can be stay put, the other ones have to do something else. But also, they should be easily predictable by the environment model. And if you learn the skills such that they're easily predictable, your environment model will make less errors, and then you can use it for planning. Okay, let's dive in. They do actually open source code, and they have more of these videos, if you want to check it out, I'll link everything in the description. Okay, let's actually dive into the the meat right here. They say they do maximize the mutual information. And we're going to see between between what and what they want to maximize the mutual information. If you don't know what the mutual information is, the mutual information is a quantity. The mutual information between x and y is a quantity, that's the entropy of x minus the entropy of x conditioned on y, or you can also decompose it in the other way around entropy of y minus entropy of y conditioned on x. And we're going to they apply this, where is it equation two right here. Okay, so what they want is the following, they want to maximize the mutual information. And the mutual information basically means how much does one variable tell me about the other variable, okay. The mutual information between the skill and we're going to see what that is, and the next state. So what, what is a skill? A skill is one entry in our table right here. So these here are skills, skills, and they're in their index by z. Now z here, it seems like they're discrete, right? But in this case, they would be z would be a continuous vector. But it's easier if you imagine that. So each one of this is like z one, z two, and so on. They're going to be continuous, but in our case, we'll just think of a discrete set of skills to be learned. Okay, so they say we want to maximize the mutual information between the skill that's currently in action. So in every, every time, the agent has to like choose a skill and saying like, Okay, now I'm going to walk forward. And it's in a given state. And what you want to say is you have to maximize the mutual information between the skill and the next state, which means that it means two things, which you can see right here, you can decompose it in two different ways. One way is the following is to say, if I know which state I'm in, what's the entropy over my? Okay, that's a wrong bad way of formulating it. It's if I knew if I know these two things. So if I know the state I'm in, and the next state that I'm going to write, I can in hindsight, I look back, if I know both things, what can I say about this skill z right here, that I couldn't say just from the starting state. So the starting state, let's say is the person right here, and the end state is the person a little bit more over there. So let's let's call this forward, it's the person is looking to the right. Okay. Now, if I only show you if I only show you the state on the left, the starting state, what can you tell me which action is going to follow right here? Basically, you can't tell me much, it could be any action like walk forward, walk back, stay put. But if I also show you the next state, then you can pretty confidently say, Ah, I know what you did, you did walk forward. Okay, so this, in this situation, we would have a high mutual information between z and s prime in the formulation here, if you decompose it in the other way, it this is equivalent, but it's a different way of thinking about it, it means, when I show you these two things, how much more can you tell me about the next state than if I only show you this. So in this formulation, what we would do is, we would say, I tell you, the human is here looking to the right, what can you tell me about the next state? And you like what I, I couldn't tell you very much, right? It can be anything. But if I then tell you the action is walk forward, then you could say, Ah, now I now I get it, it's probably going to be something like this. Okay, this also would be a high mutual information. So you see that the the task of maximizing this mutual information is good, because what happens if I don't, if I have a low mutual information, if I have a low mutual information, it would mean that I could predict the next state, just as well from the current state, it doesn't make a difference whether you give me the skill or not, it would not make a difference. And that is only the case if my skills are basically either all the same or all pretty, pretty random and pretty useless. Right. So if, if all my skills are basically walking backwards, then I don't, you don't have to tell me which skill you do, I'm gonna know that the next state is like this. So you can see that the objective of maximizing the mutual information between the skill and the next state is going to result in a situation where these skills are going to be first of all, diverse, and second of all, easy to easy to predict. Okay. And to see this, yeah, we only have to imagine what would happen, what would happen in a in a situation where the skills weren't diverse or weren't easy to predict. And you'll get exactly the situation where the information of the skill doesn't help you in predicting the next state, because yeah, either it's obvious or it's random. Okay, so we agree that it makes sense to maximize the mutual information. And they decompose this into two objectives. So they say the mutual information is basically what you'll have to do is you'll have to it decomposes into two terms where you can into two terms in a lower bound in the mutual information. And this is kind of the this is sort of the standard variational approximation literature. If you're into that read up on variational auto encoders and things like this. Basically, the two steps here are you tighten the variational lower bound and you maximize the approximate lower bound. Okay, so you have the mutual information and you can lower bound it. Okay, you can you can lower bound it by this quantity. Now the if since this is a lower bound, you can prove that this is a lower bound. It means that the higher you make this term, then the the more basically, okay, if it's a lower, I don't know how to formulate, but it should be fairly obvious if this if this thing on the right is a lower bound to the mutual information, then maximizing the thing on the right will maximize the thing on the left. And it will do so. Very so imagine I is up here. And this E on the right side is down here. It's a lower bound, right? It's lower than I. So if I maximize E, well, I haven't really done anything to I but if I maximize it even more up to here, and since it's a lower bound, I know now my I must be at least higher than this. Okay, so maximizing a lower bound to a quantity will ultimately increase the quantity. But you can also the efficiency by which it does this depends on how tight the bound is. So if the bound is very tight, like this, you see I is not much above E. If the bound is very tight, then maximizing E will result in a faster maximization of I. Okay, so you can do two things. You can maximize the quantity that is the lower bound, or you can tighten the bound. And here we can see that the difference that the tightness of the bound depends on this quantity right here, which is the KL divergence between this and this. So yeah, let's watch this in the context of we can actually go go through it on the high level. If you've never done this variational approximation sort of math, then this might be a bit informative. Okay, so the thing right here is just pops out of the definition of the mutual information. It's the it's basically the differences of the entropies, which the entropies are log quantities, right. So if you have a log a minus log b, that you can also write this as the log of the fraction of a over b, that's just a property of the log. And so it's expectations over logs, these entropies. So you can write it as this thing right here. Okay. And this basically says, this is very high, if or very low, depending so you need to whether something is low or high always will depend on what you exactly you have to consider. But ultimately, what you'll want is the ratio between this quantity, which is the probability of the next state given the current state and the skill you're taking divided by just the probability over the next state, given the the current state in expectation over all the skills, current states and next states. Now what they're saying is this here, this is basically the environment, right? This is if you are in a state and you perform a skill, what's the next state? That's that's the true environment appear is the true environment, which we don't know, right? We don't know what the environment's going to do. But we would like to learn a model for the environment. And this model for the environment is now q, q theta, phi, theta, phi, Greek letter. So q phi here is going to be a neural network that will approximate the environment. And in this probabilistic framework, it is going to be a learned distribution that will approximate the distribution of p. So we approximate it by this, but now this is here it says equal equal, right? This is not equal, because this is just an approximation. So the equality must become must be basically compensated for by this term, right here, you can see, this here is expanded into these two, you can go through the exact definitions and see why this is an equality. But basically, you can say that the mutual information is this expectation, or it is this expectation. But now you have to correct for the fact that here, you only have an approximation. And you have to correct for the fact by exactly the amount by which the approximation is different than the quantity that you're approximating. And this is this key KL divergence right here. So the KL divergence basically measures how different two distributions are. It's, it's sort of a distance, not exactly distance, but sort of a distance between these two distributions right here. It says, here's the real world. And here is your estimate of the real world, how much do they disagree. And that quantity plus, then you can replace your world, the exact world distribution by your approximate distribution, and you still are equal to the mutual information. And now the basically the trick is you say, oh, the KL divergence is always positive. It's a quantity that it can only be a positive number. So if I leave it away, certainly, this is only this is going to be a lower bound to the quantity. Okay. All right. So two tasks right here. First of all, tighten the variational bound, which means make this quantity small, make your approximate world model as close as possible to the real world. How do we do this neural network, okay, you input trajectories. I was in this state, I performed this skill, and I ended up in this state. Sorry, that's this. And then you simply match your neural network simply matches what happens in the real world, it learns the transition function, basically. So that's, that's the tightening of the variational bound. And the second step is this right here to to maximize the approximate lower bound, right, the first step was tighten variational low bound, that basically means make your world model more accurate. And the second is tighten that maximize the approximate lower bound. Now, this is going to part the this is going to be the part that says, now, given that I already have a better world model right here, can I improve my, can I sort of improve my skills, such that they become easier to predict and more diverse? Can I improve my skills such that this mutual information right here gets to be high as high as possible? Okay. So this is sort of an alternating thing. And you can see this in this very, very, very, very confusing diagram, honestly. So what are you going to do in this algorithm? First of all, in each episode, you're going to select a skill at random. And as I said, these skills, they're not predefined. So no one tells the agent to walk for it simply says, Okay, you have like, in a discrete case, you would have like, you have five skill slots, right? And the only thing I require is that they're sort of, you know, consistent over time. So skill one is always going to be sort of the same thing and skill two, but agent, you can basically decide what skill one is, right? But make the skills such that it's predictable, and that the different skills are diverse. Okay, so you're going to sample one of the skills, like skill zero or whatnot. And then you're going to do two things. First of all, you're going to learn these skill dynamics, which is you're going to learn your approximate model of the world. Okay. And how do you do that? Basically, here, you're the agent, and the agent will. So what what does the agent have to do, the agent will take in the skill Z. And it will take in the current state of the world, and it will output an action. Now, this is the model free part, right? So the agent that somehow has to come up with saying, ah, skill zero, that's, that's, that's walking forward. And in this situation, walking forward means I have to lift my leg or something like this. So you're going to take your skill, you're going to with your agent perform an action based on that skill and the current state of the world, then the environment is going to give you the next state right here. And from those things, you can now learn your world model, you know, I was in state s, I, I performed action a, but I performed action a based on skill Z. And then I ended up in state s prime. And I can learn a model of the world, right? This is a triple, I can do supervised learning of a world model. Now here, they do probabilistic learning, but and we're going to see in a second how that works. But ultimately, they approximate the world with their model. Cool. So that's the this outer loop. And then what are they going to do next, they're going to use that world model to determine a reward for the agent, and the reward for the agent for taking the action. So the reward is going to be, oh, agent, you took action a. Now what's your reward for doing this? This is the model free reinforcement learning part, your reward is going to be very high, if if this was very predictable. And if it is also diverse, right, so now, the agent has to sort of map sort of the agent has to go and make this quantity very high this, we want the outcome of these actions to be predictable, and dive and the actions themselves to be diverse. It is, I'm sorry, it's very hard to keep all of this very straight, okay. But ultimately, two steps, learn world model from the experience that you've generated. And second thing, learn the agent such that it maximizes this this quantity that we've seen before. And you do this via giving the agent a reward that is proportional to the mutual information. And we've already seen that we can approximate the mutual information by by this quantity here. Okay. So learn world model, and make the agent go higher mutual information, two steps. Okay, learn world model is very, very classic, you can say, okay, I need to improve, I need to minimize this KL divergence. So I need the gradient with respect to the parameters of my world model, I can write down the KL divergence like this. And then since I can do this reverse, so log a over b is log a minus log b. And since the world doesn't depend on the parameters of my model, this will simply give me this thing right here, which is the gradient of the log probability basically of my neural network. And this can be just optimized straightforward. This is a neural network optimized with gradient descent. These are the inputs, this is the output. Now, okay, you this is all probability distributions, but ultimately, you can you can do it pretty straightforward. Okay, so corresponds to maximizing the likelihood of the samples from P under q. Now, the second step, maximize the approximate lower bound, okay. So after they say after fitting q after improving our world model, we can optimize pi pi is the agent that actually takes the actions based on the skill. So it's given a skill, and it needs to perform an action. And it needs to maximize this quantity, as we've seen, needs to maximize the mutual information between if I know the action and if I don't, or the mutual information between the skill and the next state. I say note, this is a reinforcement learning style optimization, with a reward function of this quantity. However, so you look at the quantity that they need right here, the quantity is going to be this thing. And this thing is just I feed the skill and the state into my world model. And I look what what comes out of the world model. So this I can compute, right. But this thing right here, I can't compute because this is this is what happens in the world when I'm in state s, and I just run my agent over in expectation over all the skills. So this I don't know, they have a log of this is intractable. And they so we approximate the reward function for pi as this thing right here. Now, first, let's look at what this thing is. So the reward of taking action a and action a is based on skill z, right? So skill z was fed into the agent, the agent comes up with action a. Oh, you want me to walk forward in this situation? Okay, I'm going to lift my leg. That's the action. Okay, so the reward for this action, given this skill, and given the current state is going to be what it's going to be very high, if this here is very high. So it's going to be very high, if the probability so this s prime is the state you ended up in, right? So after taking the action, you ended up in s prime. So if what does it mean, when this quantity is very high, it means that my world model q, that is approximating the world thinks that this state is very probable, if you were in this state, and are given the skill z. So this basically means that the neural network can predict with very high accuracy, what's going to happen if you are in this state and are given this skill to perform, right? This is one of the things that we want. Now, what is it divided by? It's divided by this. And you can see here, the zi are other skills. So it is, what does this mean? This is almost the same quantity, it means how well can the same neural network predict the next state, if you were given a different skill. So it means if I'm here, and I ended up here, how well can you predict it if I tell you that I walked forward? And here you ask, well, how well can you predict it if I also if I told you you walked backward, if I told you you jumped if I told you, and so on. So you basically aggregate over all the other over all the other skills you could perform. And each time you ask the neural network, well, how likely is it that you end up in the state that I ended up when? So what does it mean if this quantity is high, or sorry, if the entire sum here is high, that means that the skill doesn't really give you much information, the neural network is very good, no matter which skill you selected, right? It's very accurate in predicting the next state doesn't really matter. The skill doesn't really matter. And this is what we don't want, right? We want that the the skills are very diverse, right? So the top part is they're easy, it's easy to predict what will happen if you perform a given skill. And if we divide this by the bottom part, and this makes it such that these skills are very diverse, because if they're not diverse, then it doesn't really matter which one you perform. And then this quantity on the bottom will be very high, but we divide by it. So we want, we want it to be low. Okay, that the reward is going to be the log of this fraction here. And this makes sense, right intuitively, but they're going to try to motivate this mathematically. And for motivating this mathematically, of course, they need to approximate this quantity right here. This quantity is the denominator. So they this denominator is a proc is an approximation to this, it's an approximation. As you can see here, this is sort of sort of a sample based approximation to the transition from s to s prime under the distribution of z. But what you want is just is the transition from s to s prime, not in your approximation, but in the real world. And they formulate this, they say, okay, we can decompose it as such as a as an integral over this conditional right here. So they bring in the z variable. And then they say, well, this, this is approximately approximately, we can replace this here by this. And we can replace this here by this, they say, well, since the this is a an approximation, the two, this is the world model is an approximation to the real world, we can sort of replace that. And then this is the this is the part that doesn't convince me they say, well, this p z of s, we can just replace it by p z. Now this is it's very tricky to see what these quantities are. Ultimately, it ends up being that right here. But it's it's it's so tricky. So they say, we replace p z given s by p of z. And, okay, let's think about this for a second. What does the top the bottom quantity is simply the distribution over your skills. And depending on your sample them, this could be like a uniform distribution over your skills, like that's fine. But what's the top thing? The top thing, basically, we can use base formula to reformulate it. It's p of s given z times p of right times p of z divided by p of s. So this quantity depends on multiple things. Here's that prior again. And this means what's the general distribution of states? What's the general distribution of states if your agent acts in the world, right? This and this we don't know. We don't we don't know. And also this right here. What's the distribution in the true world? What's the probability of a state given a given given that you were acting under a skill z. And this is also something we don't know because we don't know the world. We don't have the world model. So you run into the same problem again and again, that you're trying to approximate this and they want to make this so mathematically rigorous, but ultimately, and they go in the appendix, they go through various ways that they could solve this. But ultimately, they're just to say, well, this is approximately the same. So this right here, it basically means what skills if you're if you're in a certain state, what skills brought you here, basically, what skills brought you here? What's the distribution of skills that brought you to this state? And they say, well, we're just going to approximate that by the prior distribution over our skills, we disregard the state here. And this seems overly shaky. Like, as I said, the entire paper makes sense. But I just feel it's trying to be overly mathematical, and then run into a point where you can't be and then they're just okay, we'll, we'll just replace it. And then sort of things break down, like, you can only be overly mathematical to some degree. It doesn't really fit. But okay, so this is how you discover the skills, you maximize these quantities, alternately, you learn the world model, and you improve your your skills by making them diverse and easily predictable. So how do you then plan using these skills? This is the second part of the paper. And this is just as complicated as the first part. So they say, given the learned skills, so the learned skills are policies over action given the DZ, right? So now you know how to like walk forward and walk back and so on. And now you're, you're placed in a world, and you're given this checkpoint, it says, well, walk there. And you want this to do this using planning, you don't want to learn anymore, you simply want to plan. Okay, what do you do? And as I said, this is even more. So what you want to do is you want to do something like model predictive control, but not over actions, but over your learned skills. So you have this planner in the NPC and the planner will in its head roll out a number of different a number of different plans, it will kind of explore a bunch of different different plans, Z, it will roll them out, I'll say, okay, if I do this, and this, and this, and this, what will happen using its world model that it has learned, it will observe what's going to be the reward in each of these cases. Now, they say here access the environment reward, but can also be estimated. And this is another sort of, I feel weak point of this in that they now assume they have the true reward function. But they don't have a world model, right? They don't have the world model, but they assume that they can sort of always ask for the true reward, which is probably not the case if you if you like if you had a true world, but it could be the case, the reward could be something like, well, if you're over there, you get higher reward, but you don't exactly know how to get over there. In any case, so they roll out a bunch of trajectories in their head, they kind of plan forward, see what's going to happen if they do this or that or this or that. And then they choose the best one of these forward thoughts. And they execute it in the real world, right? So they say, Well, I'm going to you choose the skill walk forward. So the agent is now going to be tasked with walking forward. And it's going to do that in the real world for a certain amount of steps, like 10 steps of walking forward. After 10 steps of walking forward, you go back, you say, I'm in this new situation right here, what should I do? And again, the planner is going to be like, if you first walk forward and then walk back, where are you going to be, and so on. So the planner will always plan, basically to go from where you are, to the checkpoint, using a composition of the skills that you have learned. So the planner may be fine, okay, if I first walk forward, walk back a bit, and so on, I'm going to get to the goal, I'm going to reach the goal. Now please, agent execute this first thing, walk forward, the agent executes it, and maybe it won't, you know, it won't do as well, it will maybe end up here. And then it says, Well, I'm here now, please plan again. So it plans again, it says, Okay, I can still kind of walk back, I'll be here here, but then I have to do something else. So now walk back. And, okay, so this is what's going to happen. But it is going to happen in a weird way. Namely, what we keep our normal, since everything is continuous, we'll keep normal distributions of all our future steps. So we don't say, Okay, I go here, and then I go here. What you'll say is I approximately go here. And after that, I'll approximately go here. And you will do it in such a way that the peak of this normal distribution is going to be the highest, where you think you will get the most reward. If you follow this trajectory, like if you follow this trajectory, you get a very high reward. And if I follow a trajectory that maybe goes here, I won't get a high reward. If it actually turns out in your imagination that you do get a high reward for this trajectory, you'll change this distribution, such that the peak is here. And, of course, the tighter the peak is, the more sure you are. So you sort of are looking, if you look out into the world, you want the closest steps to be very peaky. And then as you look out, they can be more more sort of broad. And that's how you plan ahead, you keep refining, you keep doing a step. So if you go from here to finally you choose, I want to go here where the tip is the highest over here, then you imagine forward again, you refine these distributions over the future. And then you take the next step that gets you to the where the highest peak is right here, basically, and so on. This is simply planning in a continuous domain, it is pretty analogous to how you would plan in like, alpha go, if you or tic tac toe if you had a planner, but since everything's continuous, it makes it just so much harder. So they Yeah, they always update these distributions, as you can see here, to the skill that gave you a high reward in your imagination compared to the rewards of the other plans that you had. Okay, well, this was a long, long way until we got here. But if you recap, so first, they in an unsupervised fashion, learn these low level skills such that they're easily predictable by their own world model, and diverse. And then in the second step, they can use that to to do basically planning. So they first learn these skills, and then the planner composes them to make the agent do something. And again, the agent will never have to learn how to do this go from checkpoint to checkpoint, because the planner can just compose these low level skills. So they have these experiments right here. And we won't go through the experiment because this video is already very, very long, but they basically show that they their learned things, actually, their learned skills do end up being very diverse do end up predictable, have a high variance, and so on, they have to give certain priors to it to make it actually work in a real setting. But the results you can actually see in these videos and in the graphs, I invite you to check out the paper if you're still here. Thanks for being here. I hope this this was one of the most more complicated and mathy papers we looked at. But I think I still think it's fun. And I still think the outcome is pretty impressive right here, how you can use math to derive basically these intuitive, very intuitive objectives to learn. It's also pretty cool. Alright, that was it for me. And bye bye.
[{"start": 0.0, "end": 8.52, "text": " Hi there, take a look at this humanoid right here. It walks from one checkpoint to another"}, {"start": 8.52, "end": 13.84, "text": " checkpoint, and then to the next checkpoint, and so on. And that is its task, it gets reward"}, {"start": 13.84, "end": 20.080000000000002, "text": " from walking from checkpoint to checkpoint. Take a look at this. And this is called the"}, {"start": 20.080000000000002, "end": 26.5, "text": " end. It also walks from checkpoint to checkpoint. Now we've seen a lot of reinforcement learning"}, {"start": 26.5, "end": 33.24, "text": " algorithms in this environment is called Muko Joe, where you basically teach these little"}, {"start": 33.24, "end": 39.56, "text": " things to walk around. So the what's the impressive part here, the impressive part is that at"}, {"start": 39.56, "end": 47.28, "text": " training time, this and has never ever seen what a checkpoint is, and has never gotten"}, {"start": 47.28, "end": 52.400000000000006, "text": " any reward from walking from checkpoint to another checkpoint. Actually, it has never"}, {"start": 52.4, "end": 58.48, "text": " gotten any reward for anything that that is given from the environment, it has discovered"}, {"start": 58.48, "end": 65.92, "text": " the skill of walking by itself. And then at test time, there is no additional learning"}, {"start": 65.92, "end": 70.92, "text": " when it goes from checkpoint to checkpoint, it simply composes the skills that it knows"}, {"start": 70.92, "end": 80.28, "text": " from its unsupervised discovery phase in order to go from checkpoint to checkpoint. So here"}, {"start": 80.28, "end": 86.52, "text": " you can see, this paper basically proposes to learn these skills in a completely unsupervised"}, {"start": 86.52, "end": 92.36, "text": " way at the beginning sort of so in the training phase, it learns these skills, you can see"}, {"start": 92.36, "end": 97.72, "text": " these skills that the humanoid has learned. And then all you have to do at test time is"}, {"start": 97.72, "end": 103.54, "text": " to compose these skills to reach a given goal. And these are the things that the end has"}, {"start": 103.54, "end": 110.80000000000001, "text": " learned. Watch out, this is trippy. You can see it has learned various walks various ways"}, {"start": 110.80000000000001, "end": 115.24000000000001, "text": " walking here. And if you know anything about this environment, it's actually not that easy"}, {"start": 115.24000000000001, "end": 122.96000000000001, "text": " to make the ant walk by itself. So the the discovery here that these skills that are"}, {"start": 122.96000000000001, "end": 128.8, "text": " discovered are various ways of walking is actually already pretty impressive. And the"}, {"start": 128.8, "end": 133.92000000000002, "text": " last thing here, this cheetah, of course, also has to has learned to walk back forward"}, {"start": 133.92000000000002, "end": 139.60000000000002, "text": " kind of jump around and so on. So we're going to dive into this paper. It's called dynamics"}, {"start": 139.60000000000002, "end": 147.84, "text": " aware unsupervised discovery of skills by Archie Sharma and other people of Google brain."}, {"start": 147.84, "end": 153.16000000000003, "text": " So this was published at iClear 2020. And on a high level, I already said it's basically"}, {"start": 153.16, "end": 161.32, "text": " proposing to learn unsupervised skills, and then to compose these skills in a model based"}, {"start": 161.32, "end": 168.96, "text": " planning method at test time to reach a given goal without additional training without additional"}, {"start": 168.96, "end": 176.92, "text": " training on the reward that you give at test time. As always, if you like videos like this,"}, {"start": 176.92, "end": 184.2, "text": " you're very welcome to subscribe and share it to everyone you know. Yeah. Okay, let's"}, {"start": 184.2, "end": 191.48, "text": " dive in. So they say, conventionally model based reinforcement learning aims to learn"}, {"start": 191.48, "end": 198.72, "text": " a global model for the dynamics of the environment, which is not exactly true, right? So we have"}, {"start": 198.72, "end": 205.39999999999998, "text": " we dive into model based and model free reinforcement learning. Model based reinforcement learning"}, {"start": 205.4, "end": 212.32, "text": " basically means that you have a model of the environment. A example for this is, let's"}, {"start": 212.32, "end": 220.84, "text": " say, tic tac toe. So in tic tac toe, I know, I have nine actions at my disposal. And if"}, {"start": 220.84, "end": 228.4, "text": " I take action, let's say I take action zero, which is to make a let's say I'm the X player."}, {"start": 228.4, "end": 233.84, "text": " So I take action zero. And if I you know, number my things correctly, then that will"}, {"start": 233.84, "end": 240.24, "text": " result in this state of the world. Okay, so I know exactly how the world will look when"}, {"start": 240.24, "end": 245.72, "text": " I take a given action. And what that allows me to do is that allows me to actually plan."}, {"start": 245.72, "end": 250.72, "text": " So I cannot plan ahead, I can say what would happen if I took action zero, so I can do"}, {"start": 250.72, "end": 257.4, "text": " this in my mind. And then what would happen if I took action one, I can be like, okay,"}, {"start": 257.4, "end": 262.26, "text": " that's gonna happen. And I can do this with many things. And I can in my mind, continue"}, {"start": 262.26, "end": 269.96, "text": " this and basically roll out the entire games, or and then only do the given action that"}, {"start": 269.96, "end": 274.8, "text": " has led to the best result at the end, right. So this is model model based reinforcement"}, {"start": 274.8, "end": 280.4, "text": " learning means you have a model of the environment, you know what's going to happen when you perform"}, {"start": 280.4, "end": 287.59999999999997, "text": " given actions. And you can also combine this with machine learning, like, you know, alpha,"}, {"start": 287.6, "end": 292.84000000000003, "text": " alpha go alpha zero or so they have models of the games they're playing, they know what's"}, {"start": 292.84000000000003, "end": 298.12, "text": " going to happen. But it's very intractable to basically go down this entire tree and"}, {"start": 298.12, "end": 304.6, "text": " plan out everything. So they combine it with machine learning. It doesn't change that it's"}, {"start": 304.6, "end": 312.06, "text": " a model based. Now in, in opposition to that in model free in in reinforcement learning,"}, {"start": 312.06, "end": 316.6, "text": " what you do, you are these agent, the heirs of the environment, and you simply have to"}, {"start": 316.6, "end": 322.0, "text": " do an action, I do action zero, and the environment just gives you back a reward and the next"}, {"start": 322.0, "end": 329.32000000000005, "text": " observation. And it you have basically no clue how the environment will change. If you"}, {"start": 329.32000000000005, "end": 336.08000000000004, "text": " do something, you all these and all these, these agents do or the classic model free"}, {"start": 336.08000000000004, "end": 343.92, "text": " agents do is basically they're trying to have a neural network somewhere in them. And you"}, {"start": 343.92, "end": 349.08000000000004, "text": " put the observation in here and outcomes and action. And you can do this in various ways"}, {"start": 349.08000000000004, "end": 354.08000000000004, "text": " into Q learning or policy gradient or actor critic, and so on. But ultimately, it's simply"}, {"start": 354.08000000000004, "end": 360.56, "text": " mapping the optic current observation, and maybe the last few to the best action to take"}, {"start": 360.56, "end": 366.82, "text": " without explicitly modeling what happens in the environment. Now, when they say model"}, {"start": 366.82, "end": 373.68, "text": " based reinforcement learning, what they mean is technically, what you can do, if you're"}, {"start": 373.68, "end": 379.2, "text": " in if you are in the model free, if you're in this situation, what you could do is you"}, {"start": 379.2, "end": 387.0, "text": " could say, Well, since these model based RL techniques tend to work better, I could hear"}, {"start": 387.0, "end": 393.52, "text": " inside the agent, I could try to learn a model of the environment, E prime, and I could try"}, {"start": 393.52, "end": 399.26, "text": " to basically learn what happens in my environment when I do a certain action. And then I could"}, {"start": 399.26, "end": 406.24, "text": " use that model right here, in order to do this planning that I know from up here, okay."}, {"start": 406.24, "end": 414.92, "text": " So in this case, they go for exactly this they go for, let's learn a model of the environment,"}, {"start": 414.92, "end": 420.56, "text": " this is not an exact model, it's a learned model. And then let's use that to plan. Now,"}, {"start": 420.56, "end": 427.88, "text": " this usually has a bunch of, you know, very, a bunch of things that go against it, namely,"}, {"start": 427.88, "end": 436.52, "text": " if this model right here is bad, then the planning in the model will often accumulate"}, {"start": 436.52, "end": 441.88, "text": " and even exaggerate the errors that are in this model. So it's sometimes very hard to"}, {"start": 441.88, "end": 447.9, "text": " learn a model of the world and then use that for planning. And I've recently done a paper"}, {"start": 447.9, "end": 457.0, "text": " where where curious AI takes denoising auto encoders to regularize exactly such a planning"}, {"start": 457.0, "end": 463.8, "text": " procedure to counter this. And this paper right here is a different approach of combining"}, {"start": 463.8, "end": 475.08, "text": " this learned model. This learned model. So, okay, that was about the first sentence. They"}, {"start": 475.08, "end": 479.88, "text": " say it aims to learn a global model for the dynamics of the environment. A good model"}, {"start": 479.88, "end": 485.76, "text": " can potentially enable planning algorithms to generate a large variety of behaviors and"}, {"start": 485.76, "end": 490.71999999999997, "text": " solve diverse tasks, which is true, right? If I have a model of the environment, then"}, {"start": 490.71999999999997, "end": 496.08, "text": " I could just use it to plan, I wouldn't even have to do anything fancy anymore, right?"}, {"start": 496.08, "end": 502.59999999999997, "text": " If I have a model of how my tic-tac-toe works, I can just plan, plan my way to success. And"}, {"start": 502.59999999999997, "end": 507.56, "text": " I can do this alpha zero style. Or if the, if this state tree is small enough, I can"}, {"start": 507.56, "end": 513.1, "text": " actually just use a planner. And I don't even have to, I don't have to do anything anymore."}, {"start": 513.1, "end": 517.96, "text": " If I have a good model, they say however, learning an accurate model for complex dynamical"}, {"start": 517.96, "end": 524.08, "text": " systems is difficult. And even then the model might not generalize well outside the distribution"}, {"start": 524.08, "end": 528.84, "text": " of states on which it was trained. So this is another problem. If you learn a model,"}, {"start": 528.84, "end": 535.84, "text": " it's only going to be valid in a certain range. Okay. And say in this work, we combine model"}, {"start": 535.84, "end": 543.1600000000001, "text": " based learning with model free learning of and the model free learning is of primitives"}, {"start": 543.1600000000001, "end": 552.36, "text": " that make model based planning easy. So what they attempt to do is they attempt in an unsupervised"}, {"start": 552.36, "end": 559.36, "text": " fashion to learn so called set of skills. And the set of skills could be something like"}, {"start": 559.36, "end": 573.8000000000001, "text": " walk forward, walk backward, stay put, jump. So they attempt to learn things like this"}, {"start": 573.8000000000001, "end": 581.2, "text": " in a model free way, that the model is simply asked to come up with these things or the"}, {"start": 581.2, "end": 587.08, "text": " agent is simply asked task to come up with these things. And then in in stage two, a"}, {"start": 587.08, "end": 594.5600000000001, "text": " planner can use these skills and decompose a plan. Now this plan here, the special thing"}, {"start": 594.5600000000001, "end": 600.7800000000001, "text": " about this plan, this planner, it doesn't operate in the space of actions of like small"}, {"start": 600.7800000000001, "end": 606.08, "text": " scale actions, it actually operates in the space of these skills right here. So here"}, {"start": 606.08, "end": 612.76, "text": " would be walk forward, walk back. So and with if we have a good enough model of the environment,"}, {"start": 612.76, "end": 618.4399999999999, "text": " then will tell us if I walk forward in this situation, what will happen? Okay, so I can"}, {"start": 618.4399999999999, "end": 623.76, "text": " walk forward and then after that I could walk backward, what's going to happen? And if I"}, {"start": 623.76, "end": 630.84, "text": " have a good model of the environment over the and the actions now are these macro actions"}, {"start": 630.84, "end": 637.98, "text": " of these skills, then I can use planning to reach my goal. Okay, so the question is, how"}, {"start": 637.98, "end": 644.4, "text": " do we come up with useful skills that the planner can then use? So they need to be somewhat"}, {"start": 644.4, "end": 650.64, "text": " diverse, right? But also, and here is the crucial part and the the sort of contribution"}, {"start": 650.64, "end": 660.88, "text": " of this paper, they say, how can we discover skills, whose outcomes are easy to predict?"}, {"start": 660.88, "end": 666.6, "text": " And this is how they counteract this notion here, that if your environment model is crap,"}, {"start": 666.6, "end": 671.16, "text": " then you're you're, it can't basically can't be used for planning, you'll just make it"}, {"start": 671.16, "end": 678.2, "text": " worse. So what they say is that these things right here, these skills that we learn, we"}, {"start": 678.2, "end": 685.32, "text": " will learn them in a way that make them easy to predict. So it makes it easy to predict"}, {"start": 685.32, "end": 689.98, "text": " what will happen after I do them. So they must be at the same time diverse. So only,"}, {"start": 689.98, "end": 694.36, "text": " you know, if you stay put, it's pretty easy to predict what's going to happen like nothing."}, {"start": 694.36, "end": 701.24, "text": " Okay, but we're going to see in the exact objective that they have to be sort of diverse."}, {"start": 701.24, "end": 706.92, "text": " But also, so only one of them can be stay put, the other ones have to do something else."}, {"start": 706.92, "end": 714.08, "text": " But also, they should be easily predictable by the environment model. And if you learn"}, {"start": 714.08, "end": 719.12, "text": " the skills such that they're easily predictable, your environment model will make less errors,"}, {"start": 719.12, "end": 728.32, "text": " and then you can use it for planning. Okay, let's dive in. They do actually open source"}, {"start": 728.32, "end": 733.48, "text": " code, and they have more of these videos, if you want to check it out, I'll link everything"}, {"start": 733.48, "end": 745.28, "text": " in the description. Okay, let's actually dive into the the meat right here. They say they"}, {"start": 745.28, "end": 754.8399999999999, "text": " do maximize the mutual information. And we're going to see between between what and what"}, {"start": 754.8399999999999, "end": 758.36, "text": " they want to maximize the mutual information. If you don't know what the mutual information"}, {"start": 758.36, "end": 763.6, "text": " is, the mutual information is a quantity. The mutual information between x and y is"}, {"start": 763.6, "end": 768.3199999999999, "text": " a quantity, that's the entropy of x minus the entropy of x conditioned on y, or you"}, {"start": 768.3199999999999, "end": 775.16, "text": " can also decompose it in the other way around entropy of y minus entropy of y conditioned"}, {"start": 775.16, "end": 785.3, "text": " on x. And we're going to they apply this, where is it equation two right here. Okay,"}, {"start": 785.3, "end": 792.04, "text": " so what they want is the following, they want to maximize the mutual information. And the"}, {"start": 792.04, "end": 797.7199999999999, "text": " mutual information basically means how much does one variable tell me about the other"}, {"start": 797.72, "end": 808.96, "text": " variable, okay. The mutual information between the skill and we're going to see what that"}, {"start": 808.96, "end": 818.76, "text": " is, and the next state. So what, what is a skill? A skill is one entry in our table right"}, {"start": 818.76, "end": 827.6, "text": " here. So these here are skills, skills, and they're in their index by z. Now z here, it"}, {"start": 827.6, "end": 832.8, "text": " seems like they're discrete, right? But in this case, they would be z would be a continuous"}, {"start": 832.8, "end": 839.4, "text": " vector. But it's easier if you imagine that. So each one of this is like z one, z two,"}, {"start": 839.4, "end": 846.12, "text": " and so on. They're going to be continuous, but in our case, we'll just think of a discrete"}, {"start": 846.12, "end": 853.12, "text": " set of skills to be learned. Okay, so they say we want to maximize the mutual information"}, {"start": 853.12, "end": 860.92, "text": " between the skill that's currently in action. So in every, every time, the agent has to"}, {"start": 860.92, "end": 867.24, "text": " like choose a skill and saying like, Okay, now I'm going to walk forward. And it's in"}, {"start": 867.24, "end": 872.04, "text": " a given state. And what you want to say is you have to maximize the mutual information"}, {"start": 872.04, "end": 879.8399999999999, "text": " between the skill and the next state, which means that it means two things, which you"}, {"start": 879.8399999999999, "end": 885.76, "text": " can see right here, you can decompose it in two different ways. One way is the following"}, {"start": 885.76, "end": 896.86, "text": " is to say, if I know which state I'm in, what's the entropy over my? Okay, that's a wrong"}, {"start": 896.86, "end": 907.7, "text": " bad way of formulating it. It's if I knew if I know these two things. So if I know the"}, {"start": 907.7, "end": 913.36, "text": " state I'm in, and the next state that I'm going to write, I can in hindsight, I look"}, {"start": 913.36, "end": 922.5, "text": " back, if I know both things, what can I say about this skill z right here, that I couldn't"}, {"start": 922.5, "end": 932.52, "text": " say just from the starting state. So the starting state, let's say is the person right here,"}, {"start": 932.52, "end": 942.68, "text": " and the end state is the person a little bit more over there. So let's let's call this"}, {"start": 942.68, "end": 949.16, "text": " forward, it's the person is looking to the right. Okay. Now, if I only show you if I"}, {"start": 949.16, "end": 954.28, "text": " only show you the state on the left, the starting state, what can you tell me which action is"}, {"start": 954.28, "end": 959.3, "text": " going to follow right here? Basically, you can't tell me much, it could be any action"}, {"start": 959.3, "end": 965.8399999999999, "text": " like walk forward, walk back, stay put. But if I also show you the next state, then you"}, {"start": 965.8399999999999, "end": 973.4599999999999, "text": " can pretty confidently say, Ah, I know what you did, you did walk forward. Okay, so this,"}, {"start": 973.46, "end": 980.96, "text": " in this situation, we would have a high mutual information between z and s prime in the formulation"}, {"start": 980.96, "end": 985.1600000000001, "text": " here, if you decompose it in the other way, it this is equivalent, but it's a different"}, {"start": 985.1600000000001, "end": 991.8000000000001, "text": " way of thinking about it, it means, when I show you these two things, how much more can"}, {"start": 991.8000000000001, "end": 998.0, "text": " you tell me about the next state than if I only show you this. So in this formulation,"}, {"start": 998.0, "end": 1004.64, "text": " what we would do is, we would say, I tell you, the human is here looking to the right,"}, {"start": 1004.64, "end": 1010.84, "text": " what can you tell me about the next state? And you like what I, I couldn't tell you very"}, {"start": 1010.84, "end": 1018.24, "text": " much, right? It can be anything. But if I then tell you the action is walk forward,"}, {"start": 1018.24, "end": 1023.68, "text": " then you could say, Ah, now I now I get it, it's probably going to be something like this."}, {"start": 1023.68, "end": 1031.36, "text": " Okay, this also would be a high mutual information. So you see that the the task of maximizing"}, {"start": 1031.36, "end": 1036.8, "text": " this mutual information is good, because what happens if I don't, if I have a low mutual"}, {"start": 1036.8, "end": 1044.12, "text": " information, if I have a low mutual information, it would mean that I could predict the next"}, {"start": 1044.12, "end": 1050.56, "text": " state, just as well from the current state, it doesn't make a difference whether you give"}, {"start": 1050.56, "end": 1057.76, "text": " me the skill or not, it would not make a difference. And that is only the case if my skills are"}, {"start": 1057.76, "end": 1064.0, "text": " basically either all the same or all pretty, pretty random and pretty useless. Right. So"}, {"start": 1064.0, "end": 1069.56, "text": " if, if all my skills are basically walking backwards, then I don't, you don't have to"}, {"start": 1069.56, "end": 1074.6399999999999, "text": " tell me which skill you do, I'm gonna know that the next state is like this. So you can"}, {"start": 1074.64, "end": 1081.76, "text": " see that the objective of maximizing the mutual information between the skill and the next"}, {"start": 1081.76, "end": 1089.92, "text": " state is going to result in a situation where these skills are going to be first of all,"}, {"start": 1089.92, "end": 1099.4, "text": " diverse, and second of all, easy to easy to predict. Okay. And to see this, yeah, we only"}, {"start": 1099.4, "end": 1105.48, "text": " have to imagine what would happen, what would happen in a in a situation where the skills"}, {"start": 1105.48, "end": 1111.0400000000002, "text": " weren't diverse or weren't easy to predict. And you'll get exactly the situation where"}, {"start": 1111.0400000000002, "end": 1115.52, "text": " the information of the skill doesn't help you in predicting the next state, because"}, {"start": 1115.52, "end": 1122.8000000000002, "text": " yeah, either it's obvious or it's random. Okay, so we agree that it makes sense to maximize"}, {"start": 1122.8, "end": 1130.32, "text": " the mutual information. And they decompose this into two objectives. So they say the"}, {"start": 1130.32, "end": 1139.12, "text": " mutual information is basically what you'll have to do is you'll have to it decomposes"}, {"start": 1139.12, "end": 1148.84, "text": " into two terms where you can into two terms in a lower bound in the mutual information."}, {"start": 1148.84, "end": 1154.1599999999999, "text": " And this is kind of the this is sort of the standard variational approximation literature."}, {"start": 1154.1599999999999, "end": 1160.1999999999998, "text": " If you're into that read up on variational auto encoders and things like this. Basically,"}, {"start": 1160.1999999999998, "end": 1167.9399999999998, "text": " the two steps here are you tighten the variational lower bound and you maximize the approximate"}, {"start": 1167.9399999999998, "end": 1176.62, "text": " lower bound. Okay, so you have the mutual information and you can lower bound it. Okay,"}, {"start": 1176.62, "end": 1184.6, "text": " you can you can lower bound it by this quantity. Now the if since this is a lower bound, you"}, {"start": 1184.6, "end": 1191.8, "text": " can prove that this is a lower bound. It means that the higher you make this term, then the"}, {"start": 1191.8, "end": 1197.56, "text": " the more basically, okay, if it's a lower, I don't know how to formulate, but it should"}, {"start": 1197.56, "end": 1202.6599999999999, "text": " be fairly obvious if this if this thing on the right is a lower bound to the mutual information,"}, {"start": 1202.66, "end": 1208.44, "text": " then maximizing the thing on the right will maximize the thing on the left. And it will"}, {"start": 1208.44, "end": 1216.92, "text": " do so. Very so imagine I is up here. And this E on the right side is down here. It's a lower"}, {"start": 1216.92, "end": 1223.8400000000001, "text": " bound, right? It's lower than I. So if I maximize E, well, I haven't really done anything to"}, {"start": 1223.8400000000001, "end": 1229.48, "text": " I but if I maximize it even more up to here, and since it's a lower bound, I know now my"}, {"start": 1229.48, "end": 1235.08, "text": " I must be at least higher than this. Okay, so maximizing a lower bound to a quantity"}, {"start": 1235.08, "end": 1241.52, "text": " will ultimately increase the quantity. But you can also the efficiency by which it does"}, {"start": 1241.52, "end": 1249.0, "text": " this depends on how tight the bound is. So if the bound is very tight, like this, you"}, {"start": 1249.0, "end": 1257.68, "text": " see I is not much above E. If the bound is very tight, then maximizing E will result"}, {"start": 1257.68, "end": 1264.68, "text": " in a faster maximization of I. Okay, so you can do two things. You can maximize the quantity"}, {"start": 1264.68, "end": 1270.3, "text": " that is the lower bound, or you can tighten the bound. And here we can see that the difference"}, {"start": 1270.3, "end": 1274.92, "text": " that the tightness of the bound depends on this quantity right here, which is the KL"}, {"start": 1274.92, "end": 1286.04, "text": " divergence between this and this. So yeah, let's watch this in the context of we can"}, {"start": 1286.04, "end": 1293.12, "text": " actually go go through it on the high level. If you've never done this variational approximation"}, {"start": 1293.12, "end": 1300.84, "text": " sort of math, then this might be a bit informative. Okay, so the thing right here is just pops"}, {"start": 1300.84, "end": 1307.36, "text": " out of the definition of the mutual information. It's the it's basically the differences of"}, {"start": 1307.36, "end": 1313.36, "text": " the entropies, which the entropies are log quantities, right. So if you have a log a"}, {"start": 1313.36, "end": 1319.6399999999999, "text": " minus log b, that you can also write this as the log of the fraction of a over b, that's"}, {"start": 1319.6399999999999, "end": 1327.4799999999998, "text": " just a property of the log. And so it's expectations over logs, these entropies. So you can write"}, {"start": 1327.4799999999998, "end": 1337.3799999999999, "text": " it as this thing right here. Okay. And this basically says, this is very high, if or very"}, {"start": 1337.38, "end": 1345.2, "text": " low, depending so you need to whether something is low or high always will depend on what"}, {"start": 1345.2, "end": 1353.96, "text": " you exactly you have to consider. But ultimately, what you'll want is the ratio between this"}, {"start": 1353.96, "end": 1360.6000000000001, "text": " quantity, which is the probability of the next state given the current state and the"}, {"start": 1360.6, "end": 1368.1999999999998, "text": " skill you're taking divided by just the probability over the next state, given the the current"}, {"start": 1368.1999999999998, "end": 1375.6799999999998, "text": " state in expectation over all the skills, current states and next states. Now what they're"}, {"start": 1375.6799999999998, "end": 1383.12, "text": " saying is this here, this is basically the environment, right? This is if you are in"}, {"start": 1383.12, "end": 1388.48, "text": " a state and you perform a skill, what's the next state? That's that's the true environment"}, {"start": 1388.48, "end": 1393.24, "text": " appear is the true environment, which we don't know, right? We don't know what the environment's"}, {"start": 1393.24, "end": 1399.44, "text": " going to do. But we would like to learn a model for the environment. And this model"}, {"start": 1399.44, "end": 1413.4, "text": " for the environment is now q, q theta, phi, theta, phi, Greek letter. So q phi here is"}, {"start": 1413.4, "end": 1420.1200000000001, "text": " going to be a neural network that will approximate the environment. And in this probabilistic"}, {"start": 1420.1200000000001, "end": 1425.98, "text": " framework, it is going to be a learned distribution that will approximate the distribution of"}, {"start": 1425.98, "end": 1435.2, "text": " p. So we approximate it by this, but now this is here it says equal equal, right? This is"}, {"start": 1435.2, "end": 1443.0, "text": " not equal, because this is just an approximation. So the equality must become must be basically"}, {"start": 1443.0, "end": 1451.28, "text": " compensated for by this term, right here, you can see, this here is expanded into these"}, {"start": 1451.28, "end": 1456.14, "text": " two, you can go through the exact definitions and see why this is an equality. But basically,"}, {"start": 1456.14, "end": 1463.32, "text": " you can say that the mutual information is this expectation, or it is this expectation."}, {"start": 1463.32, "end": 1467.28, "text": " But now you have to correct for the fact that here, you only have an approximation. And"}, {"start": 1467.28, "end": 1474.28, "text": " you have to correct for the fact by exactly the amount by which the approximation is different"}, {"start": 1474.28, "end": 1480.76, "text": " than the quantity that you're approximating. And this is this key KL divergence right here."}, {"start": 1480.76, "end": 1488.68, "text": " So the KL divergence basically measures how different two distributions are. It's, it's"}, {"start": 1488.68, "end": 1493.76, "text": " sort of a distance, not exactly distance, but sort of a distance between these two distributions"}, {"start": 1493.76, "end": 1497.64, "text": " right here. It says, here's the real world. And here is your estimate of the real world,"}, {"start": 1497.64, "end": 1506.28, "text": " how much do they disagree. And that quantity plus, then you can replace your world, the"}, {"start": 1506.28, "end": 1512.8, "text": " exact world distribution by your approximate distribution, and you still are equal to the"}, {"start": 1512.8, "end": 1518.84, "text": " mutual information. And now the basically the trick is you say, oh, the KL divergence"}, {"start": 1518.84, "end": 1525.52, "text": " is always positive. It's a quantity that it can only be a positive number. So if I leave"}, {"start": 1525.52, "end": 1532.04, "text": " it away, certainly, this is only this is going to be a lower bound to the quantity. Okay."}, {"start": 1532.04, "end": 1537.32, "text": " All right. So two tasks right here. First of all, tighten the variational bound, which"}, {"start": 1537.32, "end": 1542.8, "text": " means make this quantity small, make your approximate world model as close as possible"}, {"start": 1542.8, "end": 1550.52, "text": " to the real world. How do we do this neural network, okay, you input trajectories. I was"}, {"start": 1550.52, "end": 1557.0, "text": " in this state, I performed this skill, and I ended up in this state. Sorry, that's this."}, {"start": 1557.0, "end": 1561.62, "text": " And then you simply match your neural network simply matches what happens in the real world,"}, {"start": 1561.62, "end": 1568.56, "text": " it learns the transition function, basically. So that's, that's the tightening of the variational"}, {"start": 1568.56, "end": 1579.1599999999999, "text": " bound. And the second step is this right here to"}, {"start": 1579.1599999999999, "end": 1583.32, "text": " to maximize the approximate lower bound, right, the first step was tighten variational low"}, {"start": 1583.32, "end": 1588.56, "text": " bound, that basically means make your world model more accurate. And the second is tighten"}, {"start": 1588.56, "end": 1595.12, "text": " that maximize the approximate lower bound. Now, this is going to part the this is going"}, {"start": 1595.12, "end": 1600.6, "text": " to be the part that says, now, given that I already have a better world model right"}, {"start": 1600.6, "end": 1610.12, "text": " here, can I improve my, can I sort of improve my skills, such that they become easier to"}, {"start": 1610.12, "end": 1616.6, "text": " predict and more diverse? Can I improve my skills such that this mutual information right"}, {"start": 1616.6, "end": 1626.28, "text": " here gets to be high as high as possible? Okay. So this is sort of an alternating thing."}, {"start": 1626.28, "end": 1633.1999999999998, "text": " And you can see this in this very, very, very, very confusing diagram, honestly. So what"}, {"start": 1633.1999999999998, "end": 1637.8799999999999, "text": " are you going to do in this algorithm? First of all, in each episode, you're going to select"}, {"start": 1637.8799999999999, "end": 1644.08, "text": " a skill at random. And as I said, these skills, they're not predefined. So no one tells the"}, {"start": 1644.08, "end": 1649.52, "text": " agent to walk for it simply says, Okay, you have like, in a discrete case, you would have"}, {"start": 1649.52, "end": 1655.5, "text": " like, you have five skill slots, right? And the only thing I require is that they're sort"}, {"start": 1655.5, "end": 1659.52, "text": " of, you know, consistent over time. So skill one is always going to be sort of the same"}, {"start": 1659.52, "end": 1665.6, "text": " thing and skill two, but agent, you can basically decide what skill one is, right? But make"}, {"start": 1665.6, "end": 1672.1799999999998, "text": " the skills such that it's predictable, and that the different skills are diverse. Okay,"}, {"start": 1672.18, "end": 1677.76, "text": " so you're going to sample one of the skills, like skill zero or whatnot. And then you're"}, {"start": 1677.76, "end": 1687.8200000000002, "text": " going to do two things. First of all, you're going to learn these skill dynamics, which"}, {"start": 1687.8200000000002, "end": 1694.3200000000002, "text": " is you're going to learn your approximate model of the world. Okay. And how do you do"}, {"start": 1694.32, "end": 1702.6, "text": " that? Basically, here, you're the agent, and the agent will. So what what does the agent"}, {"start": 1702.6, "end": 1710.0, "text": " have to do, the agent will take in the skill Z. And it will take in the current state of"}, {"start": 1710.0, "end": 1715.3999999999999, "text": " the world, and it will output an action. Now, this is the model free part, right? So the"}, {"start": 1715.3999999999999, "end": 1721.6399999999999, "text": " agent that somehow has to come up with saying, ah, skill zero, that's, that's, that's walking"}, {"start": 1721.64, "end": 1728.8000000000002, "text": " forward. And in this situation, walking forward means I have to lift my leg or something like"}, {"start": 1728.8000000000002, "end": 1733.76, "text": " this. So you're going to take your skill, you're going to with your agent perform an"}, {"start": 1733.76, "end": 1738.2, "text": " action based on that skill and the current state of the world, then the environment is"}, {"start": 1738.2, "end": 1744.5400000000002, "text": " going to give you the next state right here. And from those things, you can now learn your"}, {"start": 1744.5400000000002, "end": 1751.5400000000002, "text": " world model, you know, I was in state s, I, I performed action a, but I performed action"}, {"start": 1751.54, "end": 1761.28, "text": " a based on skill Z. And then I ended up in state s prime. And I can learn a model of"}, {"start": 1761.28, "end": 1765.44, "text": " the world, right? This is a triple, I can do supervised learning of a world model. Now"}, {"start": 1765.44, "end": 1770.6, "text": " here, they do probabilistic learning, but and we're going to see in a second how that"}, {"start": 1770.6, "end": 1777.8799999999999, "text": " works. But ultimately, they approximate the world with their model. Cool. So that's the"}, {"start": 1777.88, "end": 1783.74, "text": " this outer loop. And then what are they going to do next, they're going to use that world"}, {"start": 1783.74, "end": 1791.16, "text": " model to determine a reward for the agent, and the reward for the agent for taking the"}, {"start": 1791.16, "end": 1797.7600000000002, "text": " action. So the reward is going to be, oh, agent, you took action a. Now what's your"}, {"start": 1797.7600000000002, "end": 1803.24, "text": " reward for doing this? This is the model free reinforcement learning part, your reward is"}, {"start": 1803.24, "end": 1815.36, "text": " going to be very high, if if this was very predictable. And if it is also diverse, right,"}, {"start": 1815.36, "end": 1824.1200000000001, "text": " so now, the agent has to sort of map sort of the agent has to go and make this quantity"}, {"start": 1824.1200000000001, "end": 1831.74, "text": " very high this, we want the outcome of these actions to be predictable, and dive and the"}, {"start": 1831.74, "end": 1838.66, "text": " actions themselves to be diverse. It is, I'm sorry, it's very hard to keep all of this"}, {"start": 1838.66, "end": 1845.44, "text": " very straight, okay. But ultimately, two steps, learn world model from the experience that"}, {"start": 1845.44, "end": 1852.64, "text": " you've generated. And second thing, learn the agent such that it maximizes this this"}, {"start": 1852.64, "end": 1859.4, "text": " quantity that we've seen before. And you do this via giving the agent a reward that is"}, {"start": 1859.4, "end": 1866.0, "text": " proportional to the mutual information. And we've already seen that we can approximate"}, {"start": 1866.0, "end": 1879.92, "text": " the mutual information by by this quantity here. Okay. So learn world model, and make"}, {"start": 1879.92, "end": 1887.64, "text": " the agent go higher mutual information, two steps. Okay, learn world model is very, very"}, {"start": 1887.64, "end": 1893.76, "text": " classic, you can say, okay, I need to improve, I need to minimize this KL divergence. So"}, {"start": 1893.76, "end": 1901.6000000000001, "text": " I need the gradient with respect to the parameters of my world model, I can write down the KL"}, {"start": 1901.6000000000001, "end": 1908.1200000000001, "text": " divergence like this. And then since I can do this reverse, so log a over b is log a"}, {"start": 1908.1200000000001, "end": 1915.2800000000002, "text": " minus log b. And since the world doesn't depend on the parameters of my model, this will simply"}, {"start": 1915.28, "end": 1922.12, "text": " give me this thing right here, which is the gradient of the log probability basically"}, {"start": 1922.12, "end": 1927.28, "text": " of my neural network. And this can be just optimized straightforward. This is a neural"}, {"start": 1927.28, "end": 1933.92, "text": " network optimized with gradient descent. These are the inputs, this is the output. Now, okay,"}, {"start": 1933.92, "end": 1939.24, "text": " you this is all probability distributions, but ultimately, you can you can do it pretty"}, {"start": 1939.24, "end": 1944.72, "text": " straightforward. Okay, so corresponds to maximizing the likelihood of the samples from P under"}, {"start": 1944.72, "end": 1956.64, "text": " q. Now, the second step, maximize the approximate lower bound, okay. So after they say after"}, {"start": 1956.64, "end": 1964.08, "text": " fitting q after improving our world model, we can optimize pi pi is the agent that actually"}, {"start": 1964.08, "end": 1969.1200000000001, "text": " takes the actions based on the skill. So it's given a skill, and it needs to perform an"}, {"start": 1969.12, "end": 1977.0, "text": " action. And it needs to maximize this quantity, as we've seen, needs to maximize the mutual"}, {"start": 1977.0, "end": 1984.52, "text": " information between if I know the action and if I don't, or the mutual information between"}, {"start": 1984.52, "end": 1993.2399999999998, "text": " the skill and the next state. I say note, this is a reinforcement learning style optimization,"}, {"start": 1993.24, "end": 2000.2, "text": " with a reward function of this quantity. However, so you look at the quantity that they need"}, {"start": 2000.2, "end": 2007.36, "text": " right here, the quantity is going to be this thing. And this thing is just I feed the skill"}, {"start": 2007.36, "end": 2012.92, "text": " and the state into my world model. And I look what what comes out of the world model. So"}, {"start": 2012.92, "end": 2019.72, "text": " this I can compute, right. But this thing right here, I can't compute because this is"}, {"start": 2019.72, "end": 2028.3600000000001, "text": " this is what happens in the world when I'm in state s, and I just run my agent over in"}, {"start": 2028.3600000000001, "end": 2036.72, "text": " expectation over all the skills. So this I don't know, they have a log of this is intractable."}, {"start": 2036.72, "end": 2043.6200000000001, "text": " And they so we approximate the reward function for pi as this thing right here. Now, first,"}, {"start": 2043.62, "end": 2054.04, "text": " let's look at what this thing is. So the reward of taking action a and action a is based on"}, {"start": 2054.04, "end": 2060.72, "text": " skill z, right? So skill z was fed into the agent, the agent comes up with action a. Oh,"}, {"start": 2060.72, "end": 2065.16, "text": " you want me to walk forward in this situation? Okay, I'm going to lift my leg. That's the"}, {"start": 2065.16, "end": 2070.88, "text": " action. Okay, so the reward for this action, given this skill, and given the current state"}, {"start": 2070.88, "end": 2078.46, "text": " is going to be what it's going to be very high, if this here is very high. So it's going"}, {"start": 2078.46, "end": 2087.8, "text": " to be very high, if the probability so this s prime is the state you ended up in, right?"}, {"start": 2087.8, "end": 2094.1800000000003, "text": " So after taking the action, you ended up in s prime. So if what does it mean, when this"}, {"start": 2094.18, "end": 2103.62, "text": " quantity is very high, it means that my world model q, that is approximating the world thinks"}, {"start": 2103.62, "end": 2111.0, "text": " that this state is very probable, if you were in this state, and are given the skill z."}, {"start": 2111.0, "end": 2117.3799999999997, "text": " So this basically means that the neural network can predict with very high accuracy, what's"}, {"start": 2117.38, "end": 2125.12, "text": " going to happen if you are in this state and are given this skill to perform, right? This"}, {"start": 2125.12, "end": 2132.76, "text": " is one of the things that we want. Now, what is it divided by? It's divided by this. And"}, {"start": 2132.76, "end": 2141.6, "text": " you can see here, the zi are other skills. So it is, what does this mean? This is almost"}, {"start": 2141.6, "end": 2149.94, "text": " the same quantity, it means how well can the same neural network predict the next state,"}, {"start": 2149.94, "end": 2158.66, "text": " if you were given a different skill. So it means if I'm here, and I ended up here, how"}, {"start": 2158.66, "end": 2165.04, "text": " well can you predict it if I tell you that I walked forward? And here you ask, well,"}, {"start": 2165.04, "end": 2168.98, "text": " how well can you predict it if I also if I told you you walked backward, if I told you"}, {"start": 2168.98, "end": 2176.08, "text": " you jumped if I told you, and so on. So you basically aggregate over all the other over"}, {"start": 2176.08, "end": 2182.48, "text": " all the other skills you could perform. And each time you ask the neural network, well,"}, {"start": 2182.48, "end": 2188.18, "text": " how likely is it that you end up in the state that I ended up when? So what does it mean"}, {"start": 2188.18, "end": 2196.16, "text": " if this quantity is high, or sorry, if the entire sum here is high, that means that the"}, {"start": 2196.16, "end": 2201.58, "text": " skill doesn't really give you much information, the neural network is very good, no matter"}, {"start": 2201.58, "end": 2205.58, "text": " which skill you selected, right? It's very accurate in predicting the next state doesn't"}, {"start": 2205.58, "end": 2211.2599999999998, "text": " really matter. The skill doesn't really matter. And this is what we don't want, right? We"}, {"start": 2211.2599999999998, "end": 2219.02, "text": " want that the the skills are very diverse, right? So the top part is they're easy, it's"}, {"start": 2219.02, "end": 2226.12, "text": " easy to predict what will happen if you perform a given skill. And if we divide this by the"}, {"start": 2226.12, "end": 2231.9, "text": " bottom part, and this makes it such that these skills are very diverse, because if they're"}, {"start": 2231.9, "end": 2236.8599999999997, "text": " not diverse, then it doesn't really matter which one you perform. And then this quantity"}, {"start": 2236.8599999999997, "end": 2245.1, "text": " on the bottom will be very high, but we divide by it. So we want, we want it to be low. Okay,"}, {"start": 2245.1, "end": 2251.7799999999997, "text": " that the reward is going to be the log of this fraction here. And this makes sense,"}, {"start": 2251.78, "end": 2257.5, "text": " right intuitively, but they're going to try to motivate this mathematically. And for motivating"}, {"start": 2257.5, "end": 2262.34, "text": " this mathematically, of course, they need to approximate this quantity right here. This"}, {"start": 2262.34, "end": 2269.26, "text": " quantity is the denominator. So they this denominator is a proc is an approximation"}, {"start": 2269.26, "end": 2277.42, "text": " to this, it's an approximation. As you can see here, this is sort of sort of a sample"}, {"start": 2277.42, "end": 2284.5, "text": " based approximation to the transition from s to s prime under the distribution of z."}, {"start": 2284.5, "end": 2293.5, "text": " But what you want is just is the transition from s to s prime, not in your approximation,"}, {"start": 2293.5, "end": 2301.96, "text": " but in the real world. And they formulate this, they say, okay, we can decompose it"}, {"start": 2301.96, "end": 2313.42, "text": " as such as a as an integral over this conditional right here. So they bring in the z variable."}, {"start": 2313.42, "end": 2325.2200000000003, "text": " And then they say, well, this, this is approximately approximately, we can replace this here by"}, {"start": 2325.22, "end": 2333.7, "text": " this. And we can replace this here by this, they say, well, since the this is a an approximation,"}, {"start": 2333.7, "end": 2338.8199999999997, "text": " the two, this is the world model is an approximation to the real world, we can sort of replace"}, {"start": 2338.8199999999997, "end": 2346.74, "text": " that. And then this is the this is the part that doesn't convince me they say, well, this"}, {"start": 2346.74, "end": 2355.58, "text": " p z of s, we can just replace it by p z. Now this is it's very tricky to see what these"}, {"start": 2355.58, "end": 2362.54, "text": " quantities are. Ultimately, it ends up being that right here. But it's it's it's so tricky."}, {"start": 2362.54, "end": 2376.1, "text": " So they say, we replace p z given s by p of z. And, okay, let's think about this for a"}, {"start": 2376.1, "end": 2381.7, "text": " second. What does the top the bottom quantity is simply the distribution over your skills."}, {"start": 2381.7, "end": 2386.5, "text": " And depending on your sample them, this could be like a uniform distribution over your skills,"}, {"start": 2386.5, "end": 2392.52, "text": " like that's fine. But what's the top thing? The top thing, basically, we can use base"}, {"start": 2392.52, "end": 2408.06, "text": " formula to reformulate it. It's p of s given z times p of right times p of z divided by"}, {"start": 2408.06, "end": 2418.06, "text": " p of s. So this quantity depends on multiple things. Here's that prior again. And this"}, {"start": 2418.06, "end": 2424.7, "text": " means what's the general distribution of states? What's the general distribution of states"}, {"start": 2424.7, "end": 2431.58, "text": " if your agent acts in the world, right? This and this we don't know. We don't we don't"}, {"start": 2431.58, "end": 2439.24, "text": " know. And also this right here. What's the distribution in the true world? What's the"}, {"start": 2439.24, "end": 2449.3799999999997, "text": " probability of a state given a given given that you were acting under a skill z. And"}, {"start": 2449.3799999999997, "end": 2454.54, "text": " this is also something we don't know because we don't know the world. We don't have the"}, {"start": 2454.54, "end": 2459.66, "text": " world model. So you run into the same problem again and again, that you're trying to approximate"}, {"start": 2459.66, "end": 2464.9799999999996, "text": " this and they want to make this so mathematically rigorous, but ultimately, and they go in the"}, {"start": 2464.9799999999996, "end": 2468.9399999999996, "text": " appendix, they go through various ways that they could solve this. But ultimately, they're"}, {"start": 2468.94, "end": 2477.02, "text": " just to say, well, this is approximately the same. So this right here, it basically means"}, {"start": 2477.02, "end": 2485.86, "text": " what skills if you're if you're in a certain state, what skills brought you here, basically,"}, {"start": 2485.86, "end": 2490.9, "text": " what skills brought you here? What's the distribution of skills that brought you to this state?"}, {"start": 2490.9, "end": 2495.78, "text": " And they say, well, we're just going to approximate that by the prior distribution over our skills,"}, {"start": 2495.78, "end": 2503.2200000000003, "text": " we disregard the state here. And this seems overly shaky. Like, as I said, the entire"}, {"start": 2503.2200000000003, "end": 2512.1200000000003, "text": " paper makes sense. But I just feel it's trying to be overly mathematical, and then run into"}, {"start": 2512.1200000000003, "end": 2517.6200000000003, "text": " a point where you can't be and then they're just okay, we'll, we'll just replace it. And"}, {"start": 2517.6200000000003, "end": 2525.38, "text": " then sort of things break down, like, you can only be overly mathematical to some degree."}, {"start": 2525.38, "end": 2532.3, "text": " It doesn't really fit. But okay, so this is how you discover the skills, you maximize"}, {"start": 2532.3, "end": 2538.46, "text": " these quantities, alternately, you learn the world model, and you improve your your skills"}, {"start": 2538.46, "end": 2545.02, "text": " by making them diverse and easily predictable. So how do you then plan using these skills?"}, {"start": 2545.02, "end": 2551.36, "text": " This is the second part of the paper. And this is just as complicated as the first part."}, {"start": 2551.36, "end": 2555.5, "text": " So they say, given the learned skills, so the learned skills are policies over action"}, {"start": 2555.5, "end": 2562.5, "text": " given the DZ, right? So now you know how to like walk forward and walk back and so on."}, {"start": 2562.5, "end": 2568.26, "text": " And now you're, you're placed in a world, and you're given this checkpoint, it says,"}, {"start": 2568.26, "end": 2574.78, "text": " well, walk there. And you want this to do this using planning, you don't want to learn"}, {"start": 2574.78, "end": 2583.6200000000003, "text": " anymore, you simply want to plan. Okay, what do you do? And as I said, this is even more."}, {"start": 2583.6200000000003, "end": 2589.1400000000003, "text": " So what you want to do is you want to do something like model predictive control, but not over"}, {"start": 2589.1400000000003, "end": 2598.0600000000004, "text": " actions, but over your learned skills. So you have this planner in the NPC and the planner"}, {"start": 2598.06, "end": 2607.1, "text": " will in its head roll out a number of different a number of different plans, it will kind"}, {"start": 2607.1, "end": 2613.7799999999997, "text": " of explore a bunch of different different plans, Z, it will roll them out, I'll say,"}, {"start": 2613.7799999999997, "end": 2618.98, "text": " okay, if I do this, and this, and this, and this, what will happen using its world model"}, {"start": 2618.98, "end": 2625.86, "text": " that it has learned, it will observe what's going to be the reward in each of these cases."}, {"start": 2625.86, "end": 2631.38, "text": " Now, they say here access the environment reward, but can also be estimated. And this"}, {"start": 2631.38, "end": 2639.1, "text": " is another sort of, I feel weak point of this in that they now assume they have the true"}, {"start": 2639.1, "end": 2645.28, "text": " reward function. But they don't have a world model, right? They don't have the world model,"}, {"start": 2645.28, "end": 2652.4, "text": " but they assume that they can sort of always ask for the true reward, which is probably"}, {"start": 2652.4, "end": 2658.34, "text": " not the case if you if you like if you had a true world, but it could be the case, the"}, {"start": 2658.34, "end": 2662.94, "text": " reward could be something like, well, if you're over there, you get higher reward, but you"}, {"start": 2662.94, "end": 2670.54, "text": " don't exactly know how to get over there. In any case, so they roll out a bunch of trajectories"}, {"start": 2670.54, "end": 2675.42, "text": " in their head, they kind of plan forward, see what's going to happen if they do this"}, {"start": 2675.42, "end": 2684.34, "text": " or that or this or that. And then they choose the best one of these forward thoughts. And"}, {"start": 2684.34, "end": 2689.14, "text": " they execute it in the real world, right? So they say, Well, I'm going to you choose"}, {"start": 2689.14, "end": 2694.5, "text": " the skill walk forward. So the agent is now going to be tasked with walking forward. And"}, {"start": 2694.5, "end": 2699.36, "text": " it's going to do that in the real world for a certain amount of steps, like 10 steps of"}, {"start": 2699.36, "end": 2703.86, "text": " walking forward. After 10 steps of walking forward, you go back, you say, I'm in this"}, {"start": 2703.86, "end": 2708.86, "text": " new situation right here, what should I do? And again, the planner is going to be like,"}, {"start": 2708.86, "end": 2714.6600000000003, "text": " if you first walk forward and then walk back, where are you going to be, and so on. So the"}, {"start": 2714.6600000000003, "end": 2721.3, "text": " planner will always plan, basically to go from where you are, to the checkpoint, using"}, {"start": 2721.3, "end": 2727.34, "text": " a composition of the skills that you have learned. So the planner may be fine, okay,"}, {"start": 2727.34, "end": 2734.2200000000003, "text": " if I first walk forward, walk back a bit, and so on, I'm going to get to the goal, I'm going to reach"}, {"start": 2734.2200000000003, "end": 2740.7000000000003, "text": " the goal. Now please, agent execute this first thing, walk forward, the agent executes it,"}, {"start": 2740.7000000000003, "end": 2745.38, "text": " and maybe it won't, you know, it won't do as well, it will maybe end up here. And then"}, {"start": 2745.38, "end": 2750.6600000000003, "text": " it says, Well, I'm here now, please plan again. So it plans again, it says, Okay, I can still"}, {"start": 2750.6600000000003, "end": 2754.98, "text": " kind of walk back, I'll be here here, but then I have to do something else. So now walk"}, {"start": 2754.98, "end": 2763.3, "text": " back. And, okay, so this is what's going to happen. But it is going to happen in a weird"}, {"start": 2763.3, "end": 2772.22, "text": " way. Namely, what we keep our normal, since everything is continuous, we'll keep normal"}, {"start": 2772.22, "end": 2778.6, "text": " distributions of all our future steps. So we don't say, Okay, I go here, and then I"}, {"start": 2778.6, "end": 2785.06, "text": " go here. What you'll say is I approximately go here. And after that, I'll approximately"}, {"start": 2785.06, "end": 2790.7799999999997, "text": " go here. And you will do it in such a way that the peak of this normal distribution"}, {"start": 2790.7799999999997, "end": 2796.18, "text": " is going to be the highest, where you think you will get the most reward. If you follow"}, {"start": 2796.18, "end": 2800.86, "text": " this trajectory, like if you follow this trajectory, you get a very high reward. And if I follow"}, {"start": 2800.86, "end": 2806.9, "text": " a trajectory that maybe goes here, I won't get a high reward. If it actually turns out"}, {"start": 2806.9, "end": 2810.98, "text": " in your imagination that you do get a high reward for this trajectory, you'll change"}, {"start": 2810.98, "end": 2817.38, "text": " this distribution, such that the peak is here. And, of course, the tighter the peak is, the"}, {"start": 2817.38, "end": 2823.3, "text": " more sure you are. So you sort of are looking, if you look out into the world, you want the"}, {"start": 2823.3, "end": 2829.9, "text": " closest steps to be very peaky. And then as you look out, they can be more more sort of"}, {"start": 2829.9, "end": 2835.86, "text": " broad. And that's how you plan ahead, you keep refining, you keep doing a step. So if"}, {"start": 2835.86, "end": 2840.82, "text": " you go from here to finally you choose, I want to go here where the tip is the highest"}, {"start": 2840.82, "end": 2848.58, "text": " over here, then you imagine forward again, you refine these distributions over the future."}, {"start": 2848.58, "end": 2853.92, "text": " And then you take the next step that gets you to the where the highest peak is right"}, {"start": 2853.92, "end": 2862.38, "text": " here, basically, and so on. This is simply planning in a continuous domain, it is pretty"}, {"start": 2862.38, "end": 2869.9, "text": " analogous to how you would plan in like, alpha go, if you or tic tac toe if you had a planner,"}, {"start": 2869.9, "end": 2876.34, "text": " but since everything's continuous, it makes it just so much harder. So they Yeah, they"}, {"start": 2876.34, "end": 2881.44, "text": " always update these distributions, as you can see here, to the skill that gave you a"}, {"start": 2881.44, "end": 2890.06, "text": " high reward in your imagination compared to the rewards of the other plans that you had."}, {"start": 2890.06, "end": 2899.2599999999998, "text": " Okay, well, this was a long, long way until we got here. But if you recap, so first, they"}, {"start": 2899.2599999999998, "end": 2905.18, "text": " in an unsupervised fashion, learn these low level skills such that they're easily predictable"}, {"start": 2905.18, "end": 2911.02, "text": " by their own world model, and diverse. And then in the second step, they can use that"}, {"start": 2911.02, "end": 2920.44, "text": " to to do basically planning. So they first learn these skills, and then the planner composes"}, {"start": 2920.44, "end": 2927.7, "text": " them to make the agent do something. And again, the agent will never have to learn how to"}, {"start": 2927.7, "end": 2932.56, "text": " do this go from checkpoint to checkpoint, because the planner can just compose these"}, {"start": 2932.56, "end": 2940.22, "text": " low level skills. So they have these experiments right here. And we won't go through the experiment"}, {"start": 2940.22, "end": 2947.8599999999997, "text": " because this video is already very, very long, but they basically show that they their learned"}, {"start": 2947.8599999999997, "end": 2955.66, "text": " things, actually, their learned skills do end up being very diverse do end up predictable,"}, {"start": 2955.66, "end": 2961.9399999999996, "text": " have a high variance, and so on, they have to give certain priors to it to make it actually"}, {"start": 2961.9399999999996, "end": 2969.4199999999996, "text": " work in a real setting. But the results you can actually see in these videos and in the"}, {"start": 2969.42, "end": 2974.26, "text": " graphs, I invite you to check out the paper if you're still here. Thanks for being here."}, {"start": 2974.26, "end": 2980.26, "text": " I hope this this was one of the most more complicated and mathy papers we looked at."}, {"start": 2980.26, "end": 2986.62, "text": " But I think I still think it's fun. And I still think the outcome is pretty impressive"}, {"start": 2986.62, "end": 2993.34, "text": " right here, how you can use math to derive basically these intuitive, very intuitive"}, {"start": 2993.34, "end": 3000.78, "text": " objectives to learn. It's also pretty cool. Alright, that was it for me. And bye bye."}]