Author Interview - ACCEL: Evolving Curricula with Regret-Based Environment Design
#ai #accel #evolution

This is an interview with the authors Jack Parker-Holder and Minqi Jiang.
Original Paper Review Video: https://www.youtube.com/watch?v=povBDxUn1VQ

Automatic curriculum generation is one of the most promising avenues for Reinforcement Learning today. Multiple approaches have been proposed, each with their own set of advantages and drawbacks. This paper presents ACCEL, which takes the next step into the direction of constructing curricula for multi-capable agents. ACCEL combines the adversarial adaptiveness of regret-based sampling methods with the capabilities of level-editing, usually found in Evolutionary Methods.

OUTLINE:
0:00 - Intro
1:00 - Start of interview
4:45 - How did you get into this field?
8:10 - What is minimax regret?
11:45 - What levels does the regret objective select?
14:20 - Positive value loss (correcting my mistakes)
21:05 - Why is the teacher not learned?
24:45 - How much domain-specific knowledge is needed?
29:30 - What problems is this applicable to?
33:15 - Single agent vs population of agents
37:25 - Measuring and balancing level difficulty
40:35 - How does generalization emerge?
42:50 - Diving deeper into the experimental results
47:00 - What are the unsolved challenges in the field?
50:00 - Where do we go from here?

Website: https://accelagent.github.io
Paper: https://arxiv.org/abs/2203.01302
ICLR Workshop: https://sites.google.com/view/aloe2022
Book on topic: https://www.oreilly.com/radar/open-endedness-the-last-grand-challenge-youve-never-heard-of/

Abstract:
It remains a significant challenge to train generally capable agents with reinforcement learning (RL). A promising avenue for improving the robustness of RL agents is through the use of curricula. One such class of methods frames environment design as a game between a student and a teacher, using regret-based objectives to produce environment instantiations (or levels) at the frontier of the student agent's capabilities. These methods benefit from their generality, with theoretical guarantees at equilibrium, yet they often struggle to find effective levels in challenging design spaces. By contrast, evolutionary approaches seek to incrementally alter environment complexity, resulting in potentially open-ended learning, but often rely on domain-specific heuristics and vast amounts of computational resources. In this paper we propose to harness the power of evolution in a principled, regret-based curriculum. Our approach, which we call Adversarially Compounding Complexity by Editing Levels (ACCEL), seeks to constantly produce levels at the frontier of an agent's capabilities, resulting in curricula that start simple but become increasingly complex. ACCEL maintains the theoretical benefits of prior regret-based methods, while providing significant empirical gains in a diverse set of environments. An interactive version of the paper is available at this http URL.

Authors: Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, Tim Rocktäschel

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi, this is an interview with the authors of the paper evolving curricula with regret based environment design. If you haven't seen it, I've made a review of this paper yesterday, the day before this video is released, and I went over the paper in detail and explained what's inside of it. So if you haven't seen that it would be a good place to start. Today, I'm interviewing the authors of this paper, Jack and Minchi, who are real experts in this domain. Now during the interview, we go a lot deeper than I could do myself in the paper review. And you learn a lot more about how things work in this paper. But also in the entire field is a very exciting field. And it's a real privilege to be able to interview all of these people. I hope you're having fun. Please let me know in the comments how I can make these videos better for you. And thank you to everyone who does watch who does comment who does share. Thank you to all the supporters on Patreon to all the discord members and to everyone else who is excited by machine learning. I hope you're doing well. Stay hydrated. Now let's get into the interview. Jack Parker Holder and Minchi Qiang. Did I get this right? Yeah. Thank you. Welcome very much to the show. Thanks for having me. It's nice to be here. You I think your paper here, it was of one one sort an example of a very cool paper, because it's not on a set, it's a bit out of the mainstream, usually reinforcement learning tackles improving the agent as much as possible, where you you go much into this road of poet and work before it improving the environment. But also, I think it's a good lesson in how to kind of put a bit of publicity behind the paper because you made this this very cool website right here when this with the interactive demo where I can play around with the terrain, right? Okay, if it only works. And you have these these these kind of nice animations of how things develop during training and so on. And I think like, how much do you think something like this helps a paper after it's released? Like, what was your impression of just just kind of or maybe you can tell me a little bit how did you how did you even decide paper aside to make a website like this and presented in a form that's interactive? I think with RL research, especially when you look at curriculum design, you're modifying the environments, there's always really interesting visualizations that you can share. But I think having just like the standard PDF format that everyone publishes on archive in is really, really limiting. And there's just so much, there's so much amazing like assets you can actually share in terms of your agent behavior in terms of the emergent complexity that these algorithms generate. So we really wanted to share that with readers. And we thought that would definitely capture more of people's imaginations when they engage with our work. And there's like also just a huge sort of lineage of work that tries to do a similar thing like our template for this website is actually taken from distill. So distill pub has so many great works. And they put so much effort into making such beautiful interactive publications. And we definitely took a lot of inspiration from that. David Ha, Google Brain has a bunch of publications like with world models and potential agent that did similar things. Yeah. And then also we use the teach my agent work from the flowers lab as well, which had some of the like building blocks for this. And that was really cool. But I think the other thing is like, there's always this question with these type of methods, if you picked the test environments where your method works, and as reviewers ourselves, we're always very cynical of this. And so we kind of thought, what if we just let people try and break into happens? And of course, you can break it pretty easily. And that actually leads to kind of exciting questions of how you can make it better in future work. But at the same time, it's kind of nice to see how it does and doesn't work. Because then the day I think we should be more honest about the robustness of our agents. And this is quite a nice tool to not only make it fun, but also kind of demonstrate it. I think more also for not just for readers, but I think just for ourselves as researchers, like in the process of making this tool, and starting to actually run the agent and tons of visualized environments, we actually started to discover certain shortcomings of the agent, like, you can look at all these plots all day long, and you see all the metrics go up into the right, but then you don't actually see sort of the blind spots that come up during training, until you actually visualize it. And we discovered a few interesting motifs that that consistently challenged the agent, even though it's overall quite robust. Yeah, because we're actually going to talk we're talking about maybe like making it so that it defaulted to levels that we know it can do well on but then we just thought, I kind of removed the fun. And at the end of the day, if it breaks and someone's inspired to improve it, that's ultimately a good thing. So yeah, I mean, you do you do have the metrics to prove that it does something well, right? And anything after that is a bonus, essentially. How did you how did you get even into this? How did you get even into this into this field? Do you maybe want to, like give a 30 second bio of yourself? Like, how did you arrive at this point? Sure. So I mean, from my perspective, I'm hoping I'll find PhD and I thought it was really inspirational, really cool work. But I didn't really know if I'd ever get to work on something like that. And then obviously, interning last summer at a meta with Tim and Ed and Minchi, who are on paper and weaker as well. The group was working on generalization and starting to improve on idea and build on ideas such as like pairs and these algorithms. And so then, so when I came in, we were talking a little bit about, like shortcomings of our methods. And then poet obviously comes up as another example. And we were kind of thinking, how do we take some of the ideas with poet and really incorporate it into our existing, like regret based curriculum methods. And so then it became the kind of obvious that we want to try this environment and this type of work, I guess it was kind of a fusion of a fusion of different things. So it was like top down initially, and then also ended up being bottom up. Yeah. And I guess curriculum learning was something I kind of stumbled on and the first year of my PhD. And basically, I was originally trying a bunch of sort of random ideas of, I always had this notion that maybe RL could be made more efficient if you train agents on levels that were just within reach. And then you basically progressively increased the level complexity in terms of a curriculum. And so we worked on a prior method as well called prioritize level replay, which is this pink PLR baseline here. And that one ended up doing quite well, especially when combined with data augmentation on the open AI proc gen benchmark. And so right after that, I got in touch with another researcher at UC Berkeley, a fellow named Michael Dennis. And he was one of the first authors on the emerging complexity for zero shot robustness paper that introduced the paired algorithm. And so this is a this is the paper that kind of introduced a lot of the formal theory, decision theory around mini max regret policies in their application within deep RL. And it kind of was the first paper that showed that if you optimize for mini max regret in using deep RL, it makes sense and you get nice experimental results that show robustness and zero shot transfer. And so we started discussing and we realized that actually a lot of theory could be applied to PLR. And that PLR was actually another instantiation of this mini max regret game, which is at the heart of this theory. And an Excel is sort of like, you know, the latest version, it's sort of the culmination of the ideas we've explored so far in this direction. Yeah, I guess it's worth noting that we published robust PLR paper in Europe last year. So that was really that what was finishing just around June July time when I joined at Meta. And so really, we were looking, we kind of knew that method was very empirically strong and there has to be nice. But it still maybe lacked something in that it couldn't really have some creative process to design its own levels, because it could only sample I think, as you, as you pointed out in your in your review. So ultimately, if the space is very high dimensional, and you only sample one high regret level, once you've mastered it, you have to then go back to the drawing board. Whereas the nice thing about Excel is that inspired by poet, it can really kind of build its own complexity over time. And so it really is kind of like a progression through a really sequence of papers, I guess. And, and Michael's been on now three of them in a row, because he was on Paired and then robust PLR and now Excel. Can you give like a layman's, a layman's explanation for optimizing for mini max regret? Because there are a bunch of like, it's regret and then max and then min. What's, what does it ultimately boil down to? So, so, so this largely comes from this emerging complexity paper from Michael Daneson and Natasha Jax. Essentially, the theory there is essentially framing, framing a concept called unsupervised environment design as essentially this problem where you want to design environments that maximize for some metric. And that metric is usually some behavioral metric that's associated with the student agent. And so in this game, in this mini max regret game, we care about maximizing the regret of the agent. And so if you frame the game as a game where it's a two player game, it's zero sum, the payoff for the student is the negative regret. And the payoff for the teacher is the positive regret. Essentially, you have a game where the teacher tries to increase the regret of the student and the students trying to minimize its regret. So if you think about two players, zero sum games, they always have a Nash equilibrium. And at the Nash equilibrium of this game, it's got to be the policy that the student plays that essentially is a mini max regret policy. It's minimizing its worst case regret, because if it's not doing this, the teacher must be able to change its policy and play more of a certain level that further increases the regret. And so by definition at a Nash equilibrium, neither player has an improving response. So it must be that the student has a mini max regret policy. So what does that mean in layman's terms, it basically means that the student behaves in a way that essentially it's able to do well in any level that's solvable inside of the parameterized space of tasks that the teacher can use to propose the next level. So the teacher would have... The teacher's moves would essentially be the levels, like the actions of the teacher would be, I play this level. Yeah. So it's within this abstraction called a UPOMDP, which is just like a partially observable Markov decision process. But you add an additional set of variables called the free parameters. In the papers, we usually use the term theta to denote them. And so those are the positions of where the obstacles are in the maze, in the maze domain. It might be starting position of the agent, goal position. Inside of a car racing environment, it might be the position of where the tracks are. And so these are the design parameters. And so a strategy of the teacher is essentially choose some distribution over choices of the possible free parameters that it can sample as the next level. Sorry, Jack, you go. All right. I was going to say like the nice intuitive property of this is that it makes the agent has to learn to solve all of the simplest solvable environments as well. So in some other methods, like Poet, they're trying to achieve the maximum complexity, which is like, it's very cool and it's well motivated. But this is quite different in that we're actually happy if even later in training, our agent's training on simple levels, if it means that it can solve all of the simple levels, because we don't really care as much about solving like crazy complex things if it can break some simple thing, which I think is seems to make sense, at least to me. Yeah, that was one of my, let's say, worries right here is that if you if you and I framed this a little bit as you are at this zone of proximal development with your agent in that somehow made a drum like you try to reach levels that are just outside of where the agent can handle it. And then you try to edit those a little bit or maybe just where the agent can handle them. And then you try to edit them a little bit. And you try to filter by the ones that pass some threshold in this estimated regret. So my first question would be coming back to this regret, you formulated as the so it's it's a formulated as the difference to the optimal policy, right? The difference to to the optimal policy, I'm going to guess on this particular level that you're at. Why doesn't this like disregard the approximation that you do? If I could calculate this very accurately, wouldn't this select for super duper difficult levels that that could be solved with the optimal policy, right? Not impossible, but just super difficult ones? That's a great question. I think part of the part of the nuanced detail here is that so one reason that makes this all work is the discount factor. So basically, the so in the original paper that introduced paired and this idea of the mini match regret game, the reward function for that environment actually, it actually your reward, your final return decreases with the length of your trajectory. And so there's a natural discounting in terms of the return. And so essentially, by doing many max regret, it ends up prioritizing for those levels where the solutions within reach in the fewest number of steps. And you get this nice curriculum. But because here in all of our approximate single agent regret estimators, we're using a value function, which is bootstrapped off of a generalized advantage estimator, which itself is discounted. You essentially have discounting built into your value function. And so you end up with discounting even if they're even if your environment's a final, you know, sparser word, no discounting naturally in the external reward, you still get discounting because your value function is going to be discounting using gamma. And if you use GA, you have further discounting with lambda. Cool. Yeah, that was one of my one of the things that I didn't exactly understand here in this. Okay, I was like, disregard the discount factors, they're not important. Turns out they're actually one of the most important parts right here to actually make it work. Although you use, you use this, this positive value loss. Now, I think you wrote me in an email before that I got this wrong in the in the paper review. Do you want to add something to that? I mean, I guess we can start from sort of the outside in, I guess, or maybe it makes sense to do the inside out. So basically, the innermost term is essentially just a TD error. It's a one step TD error. Yeah. And it's future facing. So it's from your current time step t until the horizon t capital T. Okay. And essentially, the inner the inner term is essentially the inside out. So it's from your current time step t until the horizon t capital T. Okay. And essentially the inner the inner term except for within the max, that term is basically, if you look at the sum from t to capital T, that's basically the generalized advantage estimator from Schumann et al. And so that one is the most common, that's the advantage estimator used in PPO. It's used in other policy gradient algorithms as well. But essentially, that is essentially the average while trying to do a trade off between one step TD errors being more biased, because it's bootstrapping off a few steps, and longer TD errors being less biased, but having more variance. And so lambda is a discount factor that controls for that. And so in a nutshell, though, this is estimating advantage, which is basically, this is my actual return minus my typical return, which could you can think of as what the value function outputs. And so the zero... Sorry, this is return minus value. Yeah, you can think of it as the return you achieved minus your value prediction at each step in your trajectory, and we average it over the trajectory. And essentially, that's telling us, if that's really high, it means that I'm doing better than what I typically do. And so directionally, this is like in the direction of regret, because it means that in terms of external regret, I can actually get a higher return than I typically do, which means that this is a level where I experience regret. And then we max this with zero, which just means that we are only looking at the positive time steps where at the time steps at which this term is positive. So we're only looking when the agent does better than it typically does. And if on average, when it does better than it typically does is quite high, it means that's a level where the agent can experience a lot of regret in its decision making. How so though, like, my logic was a little bit, if I'm worse than I estimated, that means kind of it's a difficult level. Like, where's my thinking wrong here? So if you're worse than you, if you do worse than you estimated, I think in terms of just the mini max regret framework, it's just a little bit sideways from in terms of measuring the direction of regret. I think if you think of it as looking for cases where you do better than you typically do, that's really just you discovering regret. It's like you discovered a case where you achieve regret relative to your typical self, like you as sort of amortized by this value function that predicts like how well you typically do over time. So with respect to sort of like this average prediction of yourself, you're doing better. And so you're essentially discovering sources of new regret in this level. And that's basically directionally aligned with maximizing regret. While if you were to do the opposite, if you were to say, I want to look for those steps where I do worse than I think I do. I think that's an interesting thing to try actually, but I don't, at least theoretically, it doesn't seem to align with mini max regret as well. Yeah, okay. I can see the logic in that you say, I want to find levels where there's something unexpected positive thing happening. Yeah, I guess it's worth noting as well that in paired, which was the first UD algorithm to use regret, they had a very different approaches, which had a second agent called an antagonist. And the regret was just the difference in performance between those two. And so maybe that's like a bit more intuitive, because if the antagonist can solve a level and the protagonist, the student agent can't, then I guess that's more intuitive in terms of what you expect from regret. But the nice thing about this is it's kind of like a cheap approximate for single agent for red. And we definitely feel like maybe coming up with better metrics for single agent regret is exciting future work that could be improved upon here. But this was taken just from the robust PR paper, and we were surprised how well it worked in quite different environments. So and another detail is in the robust PLR work, another regress meter we used that we explored was what we call maximum Monte Carlo regret estimator. And essentially, it's the same, it's almost the same expression, except the regret target is no longer what you just received inside of a recent episodic rollout. It's for every level, we keep track of the highest return you ever achieved throughout training on that level. And we use that as an estimate for the maximum performance on that level. And then we use that as the target to subtract your value prediction on. And so that's like a more off policy regret, which I think, in some cases might be better because it's less coupled to your current policy. While the positive value loss, it's always what you recently received in a rollout in terms of your target minus your value function prediction. Yeah, is that is that worth because you would introduce some extra variance? Because you're not essentially subtracting your own bait, like use this as a baseline in the advantage estimate? Or am I am I seeing this wrong? So this would introduce extra variance. It's not using the policy update, it's used just to score the levels. Yeah, okay. So essentially, essentially, you're saying the best you've ever done, which might be more, it's going to up and bound your current performance, right? The best you've ever done, including your current performance, versus your value function. So it's slightly nicer in the sense that if you've experienced level many times, maybe you've had some forgetting, then the regret should be higher because you've done well in the past. But but the negative is you have to then score the previous episodes for every level. And then oftentimes, you don't actually have any previous experience. So it's not even that applicable. But there's there's there's a trade off. And I think, again, I think there's something that could be improved in future work. So I mean, especially with procedurally generated content, it's, it's probably hard, like you'd have to, you'd have to build some sort of even a model to estimate the best possible regret given past procedurally generated levels to sort of predict for any new one. And those two models will probably make similar sorts of mistakes, like the mistakes might even be correlated between the okay. So with respect to your your method here, which is is decently simple, what I was surprised by is that you deliberately go away from the teacher being its own agent, right? The teacher here is is, if let's say a fixed algorithm, it has some randomized components with the level editing and so on. But I think the this this differs from a lot of these kind of curriculum approaches where people try to make the teacher deliberately into its own agent and try to sort of frame the adversarial setting in terms of to learning things, doing self play, what what kept you from doing like, are you are you still convinced? Are you still convinced that this might be a good way? Or are you also looking into the direction of making the teacher kind of a learnable component? Yes. So I guess the first thing to say is that when we started this project, we actually did envisage ourselves using a learned editor. And that was like, what personally, what I was really excited about at the beginning was having maybe even a population of editors that make different edits, learn somehow maybe to compete with each other. But the first thing we tried was the simplest thing. And often you hear this in research that the simple thing works surprisingly well. And so we didn't really feel the need to really go beyond when we got results in mini grid initially that were better than anything we've seen before. We felt that it's actually better to go with a simpler approach. And maybe in future, we could consider ways to improve this by adding more learned components because that has been the trend elsewhere. But I think going from random sampling to evolution enough, it was was enough to like significantly improve based on the previous work. So we didn't need to go all the way to land as well. But even she has some additional thoughts on this. Yeah, I totally agree. I think the simplicity of it just was but it was pleasantly surprising that such a simple method could unlock such a big gain in performance. In terms of treating the teacher as an agent, I guess a lot of where this work derives from is this paired method, which did treat the teacher as an agent. And actually the teacher was trained using reinforcement learning. And from based on all the empirical results that we've sort of so far collected in the process of writing these papers, one thing that we have seen is that it seems that RL is not a very efficient way to train an agent to to solve this problem of presenting always the most challenging task for a student. And I think the reason is because it's such a highly non stationary problem. Basically, throughout training, your students going to get better at certain things, maybe get worse at others. And the policy is always evolving. It's very non stationary. So to be able to always track where in the parameter space will correspond to the levels that maximally challenge that of non stationary policy. I think that's a very hard problem for RL to solve, especially given how sample inefficient RL can be. And so what I think one of the reasons why methods like random sampling that PLR does, it works so well, is because it's really able to escape sort of the limitations of RL and just directly sample for points in the space. And you're also not locally bound to the same space. And you're also not locally bound to like just only be able to move a small amount based on a gradient step. You can really just sample anything anywhere in the space because it's randomly searching and then the curator just creates the best ones. So I think that at least within these types of domains we've looked at, this type of like random search plus evolution strategy just definitely outperforms a learned teacher. And in your architecture, I found you mentioned a bunch of times that you are relatively independent of domain specific heuristics and things like this. Specifically, you criticized Poet for choosing like an arbitrary range of returns of, you know, they just select levels between where the agents achieve between 50 and 300, which they claim to be, you know, hard but not hard, but not too hard. And yet I find, for example, in your algorithm, you need something like, well, we only put something into the buffer if the regret is above a certain threshold. Couldn't I leverage kind of the same criticism to you and say, well, probably that threshold is going to be problem specific, right? And it's kind of a hyper parameter that doesn't seem like it's dependent on the environment, but is it? I think you're right that this is dependent on the domain, but I'll say that the specific point about the hyper parameter, that one is actually a bit more benevolent of an issue, I think, because that's actually not a hyper parameter in our method. It's just whatever is the lowest score inside the buffer is the threshold. Okay. But I think that's definitely, I think if someone like you, I think, read it that way, I think we should definitely reward that in the paper. I think that's definitely going to be an improvement to clarity on that point. But it's basically the threshold is basically whatever is the lowest score in the level buffer. And if it's better than the lowest one, we replace it. So it's kind of like a priority queue in terms of the regret. But I agree with you. I think that methods like Excel and methods that basically require you to directly modify levels to construct them. I think these types of methods are always going to be domain specific, because I think at the end of the day, you need to have a way of parameterizing the environment. And that's domain knowledge. And you need to parameterize how you're editing that level. Yeah, I guess the editing itself is also, I think it's more, there's probably more domain knowledge than one cares to admit, because yeah, you think like, okay, in block world, I'm just modifying one block to be there or not, right? But there is a decision of, you know, do I modify one block? Do I modify a block of blocks? Do I place a wall, an entire wall or not? And things like this, and depending on how much you edit, because you have this assumption, right? Which is that if I modify, if I make, like my modifications need to be small enough such that they don't influence the hardness of the level too much, yet they need to be large enough such that they do bring some variation into the picture, right? And that balance, do you think that balance, it might be easy in these kinds of levels? What, like, how do you find this balance in more challenging problems? Like, I don't know if you think further. So I guess in these problems, it's worth noting that for the block situation, the actual domain randomization process places the blocks one at a time. So all we're really doing is kind of saying you have a few more steps with that initial process. So it is fairly aligned with the whole problem there. And then in the bipedal walker setting, we're just making small changes to the encoding vector. And in both settings, we have these details of this in the appendix if you dare to venture. But in both settings, we did sort of a sweep over the number of edits you can make in one go. And in both cases, we found that all the values worked well. We obviously picked the one that was the best performing on our validation sets. But it seemed fairly robust to the number of edits you make. And the thing worth noting again there is that what you could do is if, for example, you don't care as much about the number of samples you use to find a high regret level, you could just try all of these values in one batch. And then because with PLR based methods, you just curate the ones that have high regrets. You could say, okay, I'm going to do some with one edit, some with two, some with three, some with four, or whatever it might be. And you could almost scale the size of the edits. And then just from that batch, just take the high regret ones. And you're probably still going to have more new high regret levels than you would if you randomly sample from the initial distribution. So I think that there is some flexibility to do something like that. And I would argue that you could frame a lot of things in this editing sort of framework. And I think we mentioned a couple of examples like perturbing latent, latent generative model, for example, that may be seen as more general than specific encodings for environments. It is a good point. I want to stick on this a little bit. The types of problems where these methods are applicable, because they seem very general, yet it feels like you need a problem where you can construct such a curriculum and that curriculum needs to be fairly smooth, let's say so that the difficulty increase needs to be manageable and so on. And also, the regret the way you calculate regret with the with the TD error. It means that probably an environment like the walker where I, you know, I get more reward, the further I go, is probably more conducive than something like a Montezuma's revenge, even though I have a TD error and so on that kind of smooths out the loss itself. Can you comment a little bit on what kind of problems would like, where would it start to struggle? Like, where would you probably have trouble applying something like this? And where would it work? Obviously, it works super well on these types of things that you tried it on. But where would it struggle? Yeah, I think you're right. It's got to be a domain where you do have some structure that progressively gets, you know, goes from simpler to more complex. And it's, I guess, one nice benefit of these methods is that you don't need to know ahead of time, what exactly does it mean for a level in this domain to be easier or hard? Because we have this regret based heuristic to tell us that. And if you do have sort of this progressive structure within the domain, then these methods can sort of start to merge that based on this heuristic. But I think that at least with these PLR based methods, because the core is still needle in the haystack, you're looking for high regret levels by random search, and then evolution in Excel just massively augments that in terms of the amount of training data you can get from high regret levels. But the bottleneck step is still sort of like this limitation around at some point, you still have to just get that needle in the haystack. And so I think as the design space, like the dimensionality of your environment gets bigger and bigger, I would expect that these methods become less and less efficient. Do you... Yeah, a couple of... Sorry. I think we have like a one second lag or so. All right, sorry. So I guess one other thing, one perspective of this is it's really just a black box optimization problem where the function returns regret. And so we've gone from random sampling to evolution. But if you look at black box optimization literature, there are plenty of methods that trade off between global and local optimization in a more elegant way. And so what you could do is have some model or a project that's more like a black box optimization. So you could have a model or a project that maybe samples points more like diversity in the space. And then you use something like Excel locally to make edits once you found that needle in the haystack that Min-Shi mentioned. And then the second thing is that I think one place where this might break down is because it is quite a kind of greedy local optimization process is if you haven't got sort of a very clear, like high to low environment, then maybe you need something to encourage diversity. So you could have some sort of like either buffer could be maybe like hierarchical or something or you could try and preserve levels that you think are conducive to edits later on, even if they're not the current high regret levels. And these are all ideas we talked about for future work. I think really what we need is we need to have these more challenging problems to actually break our current methods before we can really think of the hammer for these nails. But yeah, what what is a bit special as well is that you train a single agent, right? Because usually the evolutionary methods they are trying to get a population of agents to work even if they want to end up with a single agent very often. And you encode all of this into a single agent. And that's kind of a PPO really basic agent, if I want to say and I have noticed a little bit that in these demonstrations, no matter what the level is, kind of the strategy tends to be the same, right? It tends to kind of, it tends to hop on this one leg with the other one with the other one out. And that is sort of the best strategy to overcome any and all obstacles. And then kind of rebalance itself once it's yeah, this one see? So maybe maybe we've been walking wrong our whole lives. But no, I mean, it's it's it's obvious if you if you instill this in a single agent, how much of a how much because I also observed some of your results here over time, which was also really cool to see when you compare to the to the poet algorithm in that you do get kind of more challenging levels later on, but they also like they don't dominate it doesn't get more and more and more and more challenging. Right? How much of this is a property of like catastrophic forgetting of the agent itself where you kind of push for the more complicated levels, but all of a sudden, it can't can't solve the easy ones anymore. And therefore, the easy ones become high regret. And then there's kind of this, like how much of this is due to your algorithm? And how much of this is due to the fact that you have a single agent trained with PPO that needs to take care of all of these tasks at the same time? My guess is it's the latter part, because I think that having this buffer that we do have, which in the robust PLR and the previous PLR paper, it does somewhat help with forgetting because you're able to sample things you haven't seen for a while. And if and if you now can't solve them as well, or, or if you now have higher regret in these levels, then you should retrain on them. So it should somewhat eliminate forgetting. But I do think it's worth noting that the this agent is just a two hidden layer neural net policy. It's not not very flexible. It's pretty, like low dimensional. And I think it really is unable to adapt to every different possible behavior. And so I think either having something where you can curve all the architecture as well to maybe make it more flexible as the levels get harder, or even just making your agent be some sort of adaptive agent, like a meta learning algorithm, for example, that does zero shot adaptation. I think these approaches are things that we're excited about maybe for future work. But I think for this, it's sort of an inevitability that if you try and have this like lofty goal of having a generally capable agent, it's going to have some brittleness to some certain components. I think we found a few cases like uphill, it's not particularly good. Yes. When we started visualizing it in this viewer that we have the demo, we noticed that, you know, like, when we were training this thing, all the complexity metrics for like roughness of the ground, it started going up very quickly. But then when we actually printed out a lot of the levels where it's successful, they tend to be levels where it's all downhill, which means that this pogo stick strategy, it's very good at just like hopping down the hill. And it's really robust at landing, like just sticking the landing in terms of like really high clicks. But like, it's like, it's a really good when you start to get more like these rugged hills going uphill, where the slope is positive. That's where it starts to struggle. So that's like a really interesting and I think a very tangible sort of example, where there's sort of a collapse in diversity in a way in the curriculum where because it is a limited, we do replay old levels. But again, it's a limited finite buffer. So you can get, you know, sort of like a buffer overflow in a sense of, you know, lots of levels that collapse in terms of similar challenges. And then maybe the agent just gets too good at going downhill, jumping down really challenging hills. But then it starts to the curriculum starts to forget that also going uphill is also important. And maybe that's what happened in some of these training runs. I like the I like the approach. I think poet or poet v2 had some sort of an approach where they do, of course, have different agents. But they had this metric of ranking the environments that they have in the buffer, right and sort of ranking them with respect to different agents and their conclusion was that if the if the different agents rank the environments in a different way, that kind of indicates a diversity of levels, right? Whereas if they rank them the same way, it's kind of like, well, it's, they're not really diverse. I think much like your regret measure, I'm big fan of these. They're not super domain independent, but they are domain independent enough, right? So that you could like you can kind of disconnect them from the real problem at hand. That's pretty cool. That one is definitely I think more general. Yeah, I think that's quite an exciting approach. Maybe if you wanted to use population, maybe even generate experiences, then that's quite a nice way of evaluating the diversity, I think. So is it fair to say that kind of the end here, like the most, let's say you train this, let's assume this is convergence at 5000 step that this is kind of a representation. Like it's almost like a fingerprint of the agent's ability in the face of a curriculum that tries to push harder and harder, right? Because there's a trade off that the easy levels, not being in the buffer or not being, yeah, not being in the buffer, is kind of like a yeah, not being in the buffer means they're easy. They can be solved, right? But then also, this is it seems like this is the curriculum that's needed for the agent to be as general as possible, not necessarily as good as possible. So yeah, I think it's worth noting as well that Minchi added a really cool feature to the website where you can actually see five seeds of each method. I don't know if you've seen that version, but you can see that the Excel agents are pretty remarkably similar. So they almost all seem to follow quite a similar gate, which makes me think that this is kind of the solution that for this network does cover the space as best as possible. And so it might be the case maybe that to get to get better behavior, better performance, maybe you need to have a Lego shovel seeds, maybe you need to have something that's a little bit more flexible, either something with memory, or I think I think some some implementations by pedal Walker use frame stacking, these types of things, maybe you can get more capacity into the network. That way, I think it's probably possible or likely that Lego is, it's probably quite likely that this this is the best policy you can get with this network to have as many max regret approach. Yeah, well, there is one survivor. Well, we'll see. We'll see. Yeah, excellent. Cool. Yeah, the website is the website is definitely pretty cool. The last interesting thing I found at least for me here was this generalization to the to the maze. And, I mean, it's it's very cool, because you, you train on these on these made up mazes starting from empty rooms, and then you test on these kind of humanoid mazes. Right here, and then you generalize to this giant maze here. Now, you say yourself, the agent seems to follow this kind of bit of a left hand rule. How does something like this emerge? Because it doesn't seem like in the generated levels, a left hand rule would be beneficial, because there's a lot of loops and stuff in that. So, you know, it's a little bit of a puzzle. But it doesn't seem like it's going to be beneficial, because there is a lot of loops and stuff in that, like how does how does a strategy like this emerge? I guess one thing that's quite worth noting in this environment is partially observable. So, you only need to regenerate a small bit of structure within within the grid for it to kind of generalize maybe to larger grids. It only seems the blue area, right? Yeah, exactly. And that actually makes this really hard. So even for a human, if you imagine you didn't know where the green dot was and try and do this as a five hours, humans would not be able to. Yeah, I certainly lost patience with it after a couple of goes, there's like a 5000 step limit. So it's quite long. But if yeah, if you look at the Excel, sort of towards the end of training as well, in the mini grid domain, a lot of the levels, so it ends up converging towards around 60 block count. And that's sort of like the threshold beyond which a lot of the levels where you randomly sample like more than 60 blocks, they tend to be unsolvable. So they tend to have a block preventing you from getting to the goal. And so 60 seems to be like the sweet spot for a 15 by 15 maze. And when you get to that level, you're like, okay, I'm going to set like that amount of saturation of blocks, a lot of the levels tend to actually become effectively single component mazes. And so those are unsolvable by the left hand rule. So I think that's also like just a contributing factor, like some property of the specific dimensionality that we looked at resulted in, you know, the complexity converging to like lots of mazes that are single component. And it helps the agent basically learn this left hand rule. Yeah, it's pretty cool. Do you I didn't dive too much into the experimental results in my review. Is there like, what are some of the things that you might want to highlight across your experimental results, maybe that you you find more interesting than the average person would when they read the paper? I guess for me, it's two things. So the first one is that the complexity is entirely emergent. So we never encourage the agents to actually increase the block count. We never encourage it to increase the stump height and bipedal walker, it just has to do that to increase the grip. So some other papers maybe all works, maybe they have some like ways to encourage this, whereas we actually didn't. So if we were to do that, maybe in the future, that's could even increase even further. And then the second thing is that all of the test cases are zero shot value. So that's a really cool thing. So the agents never seen test levels. And I think it's quite remarkable how robust it is in quite a wide range of settings. So that's probably the two takeaways for me. We also had some results in the appendix where we actually, we also test the final Excel bipedal walker agent on top of the poet levels. So in poet, actually, they publish a few of the rose plots showing the different parameter settings for bipedal walker for some of the crazier environments. And we actually tested bipedal walker, our bipedal walker with Excel on those environments. But it actually it didn't perform very strongly. So it's what's interesting is that we actually tested the bipedal walker with the same parameters. So that's what we're seeing. And then the third thing is, I think what's interesting about this result is, it sort of highlights this duality between like, the goals of these two algorithms, where I kind of see Excel as being on one side of the spectrum, which is about robustness, general robustness to unknown environments, and poet be on the other side of the spectrum, where it's focused on getting specialists for basically finding these agent environment specialist pairs, where this agent just always solves this environment. And so, you know, I think that's a really cool thing. It's kind of an interesting philosophical idea, because it's kind of saying that if you're building an AI system, do you really care about being robust to things that you don't know about? Or do you want to maximize your performance as a specialist? And I think it's a really interesting open question. And the way we navigate this trade off, I think is really full of rich ideas for future research projects. Yeah, especially ideas that could combine some of these things as well. And we've obviously talked about a lot of possible things. But I'm actually used to go a bit, if you go a little bit few pages down, what we did was, we actually took the some of the most complex levels that poet generates. And then we produced, we produced them in our own setting. And that's also 100 by 100 mates, if you're interested. 100 by 100. Did it solve it? Yeah, it has to be odd number for the for the simulators to work. Okay, okay. That one gets the 8% success rate on that one. It's I think a bit above this. Is it a table? Yeah. Higher up higher up. Maybe. What are you looking for? The poet. Yeah, it should be a small it's like a very small table. I think it's down below more search in the paper itself, I guess. We should probably have paper up on our own screen. Well, my bad for for not knowing it too well. Oh, yeah. It's potentially on the next page. This is the like main experiments on the test cases. I think it must be on the next page. I think it's on the next page. Ah, this is the one. Yeah, so 1a to 3b are in the paper towards the end. They have like a rose plot for some of the most extremely challenging levels that each of their seeds generated. So for all three of their seeds, they pick two different levels that they're particularly high values. And we tested our agent zero shot on those. And yeah, the scores are pretty low. But I think the fact that they're above zero is cool. But at the same time, it does make you think that if they can solve those repeatedly, then maybe you do need specialists in some cases to get the most complex things. So some hybrid specialists and generalists might be an even more powerful algorithm than either of them combined. Excellent. So you mentioned a bunch of different and you also have a future work section and so on. What do you think are apart from the things you're going to do next? What are like the big unsolved challenges in the field? Like what's everyone after but no one's been able to do it so far? Well, so the big one is a theme that we as a group have gotten very interested in recently, and we're actually holding a workshop at iClear about this. And essentially, it's about agent environment co-evolution. But in this, in the context of this much older problem called open-endedness. And basically, open-endedness is an idea that kind of came from a group of researchers, Ken Stanley, Joel Lehman, and Jeff Klune. And I think Jeff Klune has this concept of AI generating AI. And it's related to this idea of open-endedness where can you basically create a learning system that essentially ends up evolving just an unbounded amount of novelty and knowledge. And if you can kickstart a process that achieves true open-endedness, then the idea is that maybe you can replicate the emergence of some really complex intelligences, like human level intelligence, because evolution, like the tree of life, this is all sort of the result of an open-ended learning process. And so a lot of where we see this work going is that we see our work as sort of fitting within this bigger theme of open-endedness, and this larger theme of agent environment co-evolution to achieve this open-endedness. And so I think that that sort of to me is one of the most interesting open problems in AI or machine learning, or maybe it goes beyond even these two subjects. Yeah, so I think that if we can actually kick off a process like this, that would be incredible. And I'd be very curious to see what kinds of things fall out of it. Yeah, and for me, the thing I'm really excited about is that, again, tying in with Minshe's is this seems like the only limitation to this really being open-ended is requirement for a simulator. So I'm really excited about whether we can actually learn simulators, for example, world models. So I was obviously very inspired by the Harald Rydhugel work from 2018. But more modern, like offline RL world models. So maybe you have some transformer world model that learns from all this crazy amount of data. And then you can use that to design environments for an RL agent and then collect more data and just keep going. And maybe that's how you really get towards this true open-endedness, because you're not bounded by just the open AI gym environment that you're given. And so this is maybe it's a little bit more of a medium to long term goal, because I think we're a bit away from that right now. But I think that that could be where we can actually learn more about open-endedness. And I think we're a bit away from that right now. But I think that that could be where these different fields intersect and really produce something pretty crazy. Yeah, my issue a little bit with the agent environment co-evolution work is that it just seems to kind of shift the problem away from because, okay, we're evolving the environments right here, but they're still like extremely bounded in an extremely parameterized space, right? And there's only like these many ways that the environment can vary. And the true environment is kind of like the environment generator itself. And it seems like, you know, we could go a level higher and so on. But how is there a method to generally break out of this? I think one way is, you know, it's related to what Jack just described, which is this. So you've heard of SIM to real as the paradigm where you train intelligence in simulation, you transfer to reality. And that's obviously bounded by the fidelity of your simulator for your target domain. There's a new paradigm emerging. And it's like sort of pushed by all these advances in computer vision, which some people have called real to SIM to real. And basically, the idea that you can essentially collect data in a loop where you know, you may have some exploratory agent, maybe it's a hand-coded controller, or maybe it's an RL agent, the one you're training, and you send it out into the wild. It collects lots of data about what the world is like. And then you use that data to essentially enrich your simulator to basically fit your simulator to reality, to all the new things it's learned. And then you get a better, more expansive simulator. You train your agent again in that simulator, and you get a new agent to transfer to reality. And then this loop just keeps repeating. And maybe you can do this in a population of agents doing this. And you get really huge coverage in terms of what's out there. I think that's one promising way to do it. The other though, is I think it kind of just generally the strategy is, like you said, all these simulators are bounded in terms of their parameterization. Like we're looking at 15 by 15 nases, there's a finite number of them. I think what would be really cool is if we started as RL researchers started focusing more on environments that are unbounded in parameterization. So moving into these like more almost non parametric settings, where the environment can just keep growing arbitrarily in its number of parameters. And I actually think the real to sim to real loop is one way to do that, just because the space of possible worlds you can represent as a world model as a neural network is pretty much infinite. But maybe there are other simpler ways you could do this as initial toy tests as well. And then when you have that real sim to real world model, you can then training meaning max regret policy inside it. Yeah. Because then you have like this idea of the population generating this diverse, you know, very high dimensional world model, but then a single agent, maybe that could be generally robust to any possible variation. And so this is maybe a bit of a medium term. But I think for us, it's kind of an all star at the moment. Do you think there will ever be sorry, last question by me, do you think there will ever ever be this distinction between agent and environment will will this continue to be an important distinction? Or is that something that you see in the future vanish and kind of almost become like, let's say interchangeable because people are already like pitting them against each other training them both with RL and so on. Like, why do we even make the distinction? Well, I guess one thing that's interesting is even in the original world models paper, because the world model itself was generated model, the policy was very low dimensional, and it just trained inside the latent state, latent space of the world, the generative model. So then when you're actually interacting with the real environment, you still use the encoder from the world model to process the input so that the policy can then operate. And so in that sense, it's like the world model is the environment at training time offline. But then at test time, when you go back to the real environment, the world models used to process the inputs for the policy. And so they're kind of taking a very, like, I guess, competitive and then a cooperative mindset. So I think maybe there's something like that, where you have well models that are your environment for training time, but then you use them as knowledge bases for test time. I think that's pretty exciting. And it also kind of relates this idea of the cherry on top, because the policy is very small, although I hate to use too many cliches, but it does seem to relate to that sort of self supervised learning large world models, and then RL just for controllers inside that that can operate on the representations. I don't know, Minchi, thanks for that. Well, I think to sort of answer the other side of that question, I think that agent environment, I guess the distinction is, in some ways, it's arbitrary, because you can imagine, you know, like, what part of this learning system actually belongs to the agent? Like, is the agent really like at the activation level? Is it at the observation level? Like, where do you even draw the boundary in terms of the agent? I think that's an interesting question. But I also think that at some point, there's going to be some substrate in which the agent has to operate within. And there seems to be, like, basically, if you wanted to emerge a diverse sort of, you know, tree of life of different RL agents and environments, it seems like there is some sort of asymmetry there in the sense that agents have to operate within an environment, and you can't have it reversed. And so in some, to some extent, I think we'll still have to have this distinction between agents and environments. But it's also possible, you know, like, maybe we could also just learn, you know, joint distributions over agents and environments where you basically just learn, you know, like, the agents parameters themselves are now part of the environment design. And so now you're just emerging agents and environments together inside of a single generative model. I think that's an exciting idea. But and maybe at some point, we'll figure out how to do that. Where can people get started with this if they want to dive into it? So there's a great for open endedness, there's a great primer to it on O'Reilly, I can actually send you the link after. But it's written by some of the original sort of pioneers within this field. And essentially, it's quite long, but it summarizes the whole field. Another, another really interesting work would be I think, just to check out the original mini max regret paper for RL, which is this emergent complexity for zero shot generalization from Michael Dennis and Natasha. Jake jacks, and I would definitely recommend, you know, our line of work with robust pillar checking out this paper. And there's older methods like teacher student curriculum learning from shum shumans group at open AI. And workshop. Yeah. So we're going to have an iClear workshop called agent learning in open endedness, aloe. And that's going to feature a lot of speakers and researchers actively making progress in this field. So if people are really interested, they should check that out. Yeah, that's April 29, April 29. Yeah, Friday. Good. Also, more in a multi agent setting, there's the curriculum learning manifesto from Joel, Joel, Lebo at DeepMind. And we're going to have a lot of people in the audience. So if you're interested, you can check out the poster session. Yeah, that's April 29, April 29, April 29, Friday. Good. Also, more in a multi agent setting, there's the curriculum learning manifesto from Joel, Joel, Lebo at DeepMind. And that has some really nice ideas in terms of automatic curriculum learning, emerging emergent complexity. Cool. Minchi and Jack, thank you very much for being here. This was really cool. Thank you for having us.
[{"start": 0.0, "end": 5.28, "text": " Hi, this is an interview with the authors of the paper evolving curricula with regret based"}, {"start": 5.28, "end": 10.0, "text": " environment design. If you haven't seen it, I've made a review of this paper yesterday,"}, {"start": 10.56, "end": 15.36, "text": " the day before this video is released, and I went over the paper in detail and explained"}, {"start": 15.36, "end": 19.68, "text": " what's inside of it. So if you haven't seen that it would be a good place to start. Today,"}, {"start": 19.68, "end": 26.32, "text": " I'm interviewing the authors of this paper, Jack and Minchi, who are real experts in this domain."}, {"start": 26.32, "end": 31.76, "text": " Now during the interview, we go a lot deeper than I could do myself in the paper review. And you"}, {"start": 31.76, "end": 36.88, "text": " learn a lot more about how things work in this paper. But also in the entire field is a very"}, {"start": 36.88, "end": 41.120000000000005, "text": " exciting field. And it's a real privilege to be able to interview all of these people. I hope"}, {"start": 41.120000000000005, "end": 45.2, "text": " you're having fun. Please let me know in the comments how I can make these videos better for"}, {"start": 45.2, "end": 49.84, "text": " you. And thank you to everyone who does watch who does comment who does share. Thank you to all the"}, {"start": 49.84, "end": 54.64, "text": " supporters on Patreon to all the discord members and to everyone else who is excited by machine"}, {"start": 54.64, "end": 58.4, "text": " learning. I hope you're doing well. Stay hydrated. Now let's get into the interview."}, {"start": 60.8, "end": 69.12, "text": " Jack Parker Holder and Minchi Qiang. Did I get this right? Yeah. Thank you. Welcome very much"}, {"start": 69.12, "end": 76.96000000000001, "text": " to the show. Thanks for having me. It's nice to be here. You I think your paper here, it was of one"}, {"start": 76.96000000000001, "end": 82.88, "text": " one sort an example of a very cool paper, because it's not on a set, it's a bit out of the mainstream,"}, {"start": 82.88, "end": 89.36, "text": " usually reinforcement learning tackles improving the agent as much as possible, where you you go"}, {"start": 89.36, "end": 95.36, "text": " much into this road of poet and work before it improving the environment. But also, I think it's"}, {"start": 95.36, "end": 101.36, "text": " a good lesson in how to kind of put a bit of publicity behind the paper because you made this"}, {"start": 101.36, "end": 106.47999999999999, "text": " this very cool website right here when this with the interactive demo where I can play around with"}, {"start": 106.48, "end": 113.36, "text": " the terrain, right? Okay, if it only works. And you have these these these kind of nice animations"}, {"start": 113.36, "end": 119.92, "text": " of how things develop during training and so on. And I think like, how much do you think something"}, {"start": 119.92, "end": 127.92, "text": " like this helps a paper after it's released? Like, what was your impression of just just kind of or"}, {"start": 127.92, "end": 132.72, "text": " maybe you can tell me a little bit how did you how did you even decide paper aside to make a"}, {"start": 132.72, "end": 139.84, "text": " website like this and presented in a form that's interactive? I think with RL research, especially"}, {"start": 139.84, "end": 144.48, "text": " when you look at curriculum design, you're modifying the environments, there's always really"}, {"start": 144.48, "end": 149.12, "text": " interesting visualizations that you can share. But I think having just like the standard PDF format"}, {"start": 149.12, "end": 154.8, "text": " that everyone publishes on archive in is really, really limiting. And there's just so much, there's"}, {"start": 154.8, "end": 158.72, "text": " so much amazing like assets you can actually share in terms of your agent behavior in terms of the"}, {"start": 158.72, "end": 164.08, "text": " emergent complexity that these algorithms generate. So we really wanted to share that with readers."}, {"start": 164.08, "end": 169.28, "text": " And we thought that would definitely capture more of people's imaginations when they engage with our"}, {"start": 169.28, "end": 174.96, "text": " work. And there's like also just a huge sort of lineage of work that tries to do a similar thing"}, {"start": 174.96, "end": 181.92, "text": " like our template for this website is actually taken from distill. So distill pub has so many"}, {"start": 181.92, "end": 186.8, "text": " great works. And they put so much effort into making such beautiful interactive publications."}, {"start": 186.8, "end": 191.68, "text": " And we definitely took a lot of inspiration from that. David Ha, Google Brain has a bunch of"}, {"start": 191.68, "end": 196.96, "text": " publications like with world models and potential agent that did similar things. Yeah. And then also"}, {"start": 196.96, "end": 201.84, "text": " we use the teach my agent work from the flowers lab as well, which had some of the like building"}, {"start": 201.84, "end": 206.48000000000002, "text": " blocks for this. And that was really cool. But I think the other thing is like, there's always this"}, {"start": 206.48000000000002, "end": 210.88000000000002, "text": " question with these type of methods, if you picked the test environments where your method works,"}, {"start": 210.88000000000002, "end": 215.04000000000002, "text": " and as reviewers ourselves, we're always very cynical of this. And so we kind of thought, what"}, {"start": 215.04, "end": 220.0, "text": " if we just let people try and break into happens? And of course, you can break it pretty easily. And"}, {"start": 220.0, "end": 223.28, "text": " that actually leads to kind of exciting questions of how you can make it better in future work."}, {"start": 223.28, "end": 228.0, "text": " But at the same time, it's kind of nice to see how it does and doesn't work. Because then the day I"}, {"start": 228.0, "end": 232.32, "text": " think we should be more honest about the robustness of our agents. And this is quite a nice tool to not"}, {"start": 232.32, "end": 240.0, "text": " only make it fun, but also kind of demonstrate it. I think more also for not just for readers,"}, {"start": 240.0, "end": 245.2, "text": " but I think just for ourselves as researchers, like in the process of making this tool, and"}, {"start": 245.2, "end": 249.44, "text": " starting to actually run the agent and tons of visualized environments, we actually started to"}, {"start": 249.44, "end": 254.16, "text": " discover certain shortcomings of the agent, like, you can look at all these plots all day long,"}, {"start": 254.16, "end": 257.92, "text": " and you see all the metrics go up into the right, but then you don't actually see sort of the blind"}, {"start": 257.92, "end": 263.52, "text": " spots that come up during training, until you actually visualize it. And we discovered a few"}, {"start": 263.52, "end": 267.92, "text": " interesting motifs that that consistently challenged the agent, even though it's overall"}, {"start": 267.92, "end": 272.32, "text": " quite robust. Yeah, because we're actually going to talk we're talking about maybe like making it so"}, {"start": 272.32, "end": 276.96000000000004, "text": " that it defaulted to levels that we know it can do well on but then we just thought, I kind of"}, {"start": 276.96000000000004, "end": 281.44, "text": " removed the fun. And at the end of the day, if it breaks and someone's inspired to improve it,"}, {"start": 281.44, "end": 287.36, "text": " that's ultimately a good thing. So yeah, I mean, you do you do have the metrics to prove that"}, {"start": 287.36, "end": 293.84000000000003, "text": " it does something well, right? And anything after that is a bonus, essentially. How did you how did"}, {"start": 293.84, "end": 298.32, "text": " you get even into this? How did you get even into this into this field? Do you maybe want to,"}, {"start": 299.2, "end": 304.08, "text": " like give a 30 second bio of yourself? Like, how did you arrive at this point? Sure. So I mean,"}, {"start": 304.08, "end": 309.28, "text": " from my perspective, I'm hoping I'll find PhD and I thought it was really inspirational, really cool"}, {"start": 309.28, "end": 314.47999999999996, "text": " work. But I didn't really know if I'd ever get to work on something like that. And then obviously,"}, {"start": 314.47999999999996, "end": 321.76, "text": " interning last summer at a meta with Tim and Ed and Minchi, who are on paper and weaker as well."}, {"start": 321.76, "end": 326.56, "text": " The group was working on generalization and starting to improve on idea and build on ideas"}, {"start": 326.56, "end": 331.68, "text": " such as like pairs and these algorithms. And so then, so when I came in, we were talking a little"}, {"start": 331.68, "end": 336.0, "text": " bit about, like shortcomings of our methods. And then poet obviously comes up as another example."}, {"start": 336.0, "end": 339.59999999999997, "text": " And we were kind of thinking, how do we take some of the ideas with poet and really incorporate it"}, {"start": 339.59999999999997, "end": 345.12, "text": " into our existing, like regret based curriculum methods. And so then it became the kind of obvious"}, {"start": 345.12, "end": 349.52, "text": " that we want to try this environment and this type of work, I guess it was kind of a fusion of"}, {"start": 349.52, "end": 353.91999999999996, "text": " a fusion of different things. So it was like top down initially, and then also ended up being bottom"}, {"start": 353.91999999999996, "end": 359.12, "text": " up. Yeah. And I guess curriculum learning was something I kind of stumbled on and the first year"}, {"start": 359.12, "end": 365.2, "text": " of my PhD. And basically, I was originally trying a bunch of sort of random ideas of, I always had"}, {"start": 365.2, "end": 370.71999999999997, "text": " this notion that maybe RL could be made more efficient if you train agents on levels that"}, {"start": 370.71999999999997, "end": 375.59999999999997, "text": " were just within reach. And then you basically progressively increased the level complexity in"}, {"start": 375.6, "end": 380.32000000000005, "text": " terms of a curriculum. And so we worked on a prior method as well called prioritize level replay,"}, {"start": 380.32000000000005, "end": 387.12, "text": " which is this pink PLR baseline here. And that one ended up doing quite well, especially when"}, {"start": 387.12, "end": 393.04, "text": " combined with data augmentation on the open AI proc gen benchmark. And so right after that,"}, {"start": 393.84000000000003, "end": 400.08000000000004, "text": " I got in touch with another researcher at UC Berkeley, a fellow named Michael Dennis. And he"}, {"start": 400.08, "end": 407.12, "text": " was one of the first authors on the emerging complexity for zero shot robustness paper that"}, {"start": 407.12, "end": 411.68, "text": " introduced the paired algorithm. And so this is a this is the paper that kind of introduced a lot"}, {"start": 411.68, "end": 416.88, "text": " of the formal theory, decision theory around mini max regret policies in their application within"}, {"start": 416.88, "end": 421.68, "text": " deep RL. And it kind of was the first paper that showed that if you optimize for mini max regret"}, {"start": 421.68, "end": 427.2, "text": " in using deep RL, it makes sense and you get nice experimental results that show robustness and zero"}, {"start": 427.2, "end": 432.8, "text": " shot transfer. And so we started discussing and we realized that actually a lot of theory could be"}, {"start": 432.8, "end": 438.64, "text": " applied to PLR. And that PLR was actually another instantiation of this mini max regret game,"}, {"start": 438.64, "end": 445.12, "text": " which is at the heart of this theory. And an Excel is sort of like, you know, the latest version,"}, {"start": 445.12, "end": 449.28, "text": " it's sort of the culmination of the ideas we've explored so far in this direction."}, {"start": 449.28, "end": 453.12, "text": " Yeah, I guess it's worth noting that we published robust PLR paper in Europe last year."}, {"start": 453.12, "end": 458.08, "text": " So that was really that what was finishing just around June July time when I joined"}, {"start": 458.64, "end": 463.44, "text": " at Meta. And so really, we were looking, we kind of knew that method was very empirically strong"}, {"start": 463.44, "end": 467.28000000000003, "text": " and there has to be nice. But it still maybe lacked something in that it couldn't really have"}, {"start": 467.28000000000003, "end": 471.28000000000003, "text": " some creative process to design its own levels, because it could only sample I think, as you,"}, {"start": 471.28000000000003, "end": 475.68, "text": " as you pointed out in your in your review. So ultimately, if the space is very high dimensional,"}, {"start": 475.68, "end": 479.52, "text": " and you only sample one high regret level, once you've mastered it, you have to then go back to"}, {"start": 479.52, "end": 483.28, "text": " the drawing board. Whereas the nice thing about Excel is that inspired by poet, it can really"}, {"start": 483.28, "end": 488.15999999999997, "text": " kind of build its own complexity over time. And so it really is kind of like a progression through"}, {"start": 489.35999999999996, "end": 494.47999999999996, "text": " a really sequence of papers, I guess. And, and Michael's been on now three of them in a row,"}, {"start": 494.47999999999996, "end": 497.28, "text": " because he was on Paired and then robust PLR and now Excel."}, {"start": 497.28, "end": 504.0, "text": " Can you give like a layman's, a layman's explanation for optimizing for mini max regret?"}, {"start": 504.0, "end": 511.52, "text": " Because there are a bunch of like, it's regret and then max and then min. What's, what does it"}, {"start": 511.52, "end": 513.12, "text": " ultimately boil down to?"}, {"start": 513.12, "end": 522.56, "text": " So, so, so this largely comes from this emerging complexity paper from Michael Daneson and Natasha"}, {"start": 522.56, "end": 529.2, "text": " Jax. Essentially, the theory there is essentially framing, framing a concept called unsupervised"}, {"start": 529.2, "end": 534.32, "text": " environment design as essentially this problem where you want to design environments that"}, {"start": 534.32, "end": 539.36, "text": " maximize for some metric. And that metric is usually some behavioral metric that's associated"}, {"start": 539.36, "end": 544.5600000000001, "text": " with the student agent. And so in this game, in this mini max regret game, we care about maximizing"}, {"start": 544.5600000000001, "end": 550.32, "text": " the regret of the agent. And so if you frame the game as a game where it's a two player game,"}, {"start": 550.32, "end": 555.9200000000001, "text": " it's zero sum, the payoff for the student is the negative regret. And the payoff for the teacher"}, {"start": 555.92, "end": 560.7199999999999, "text": " is the positive regret. Essentially, you have a game where the teacher tries to increase the"}, {"start": 560.7199999999999, "end": 564.64, "text": " regret of the student and the students trying to minimize its regret. So if you think about two"}, {"start": 564.64, "end": 570.0799999999999, "text": " players, zero sum games, they always have a Nash equilibrium. And at the Nash equilibrium of this"}, {"start": 570.0799999999999, "end": 575.1999999999999, "text": " game, it's got to be the policy that the student plays that essentially is a mini max regret policy."}, {"start": 575.1999999999999, "end": 580.24, "text": " It's minimizing its worst case regret, because if it's not doing this, the teacher must be able to"}, {"start": 580.24, "end": 585.76, "text": " change its policy and play more of a certain level that further increases the regret. And so by"}, {"start": 585.76, "end": 591.52, "text": " definition at a Nash equilibrium, neither player has an improving response. So it must be that the"}, {"start": 591.52, "end": 595.44, "text": " student has a mini max regret policy. So what does that mean in layman's terms, it basically means"}, {"start": 595.44, "end": 602.0, "text": " that the student behaves in a way that essentially it's able to do well in any level that's solvable"}, {"start": 602.0, "end": 608.5600000000001, "text": " inside of the parameterized space of tasks that the teacher can use to propose the next level."}, {"start": 608.56, "end": 613.8399999999999, "text": " So the teacher would have..."}, {"start": 613.8399999999999, "end": 622.64, "text": " The teacher's moves would essentially be the levels, like the actions of the teacher would be,"}, {"start": 622.64, "end": 624.0799999999999, "text": " I play this level."}, {"start": 624.0799999999999, "end": 629.52, "text": " Yeah. So it's within this abstraction called a UPOMDP, which is just like a partially observable"}, {"start": 629.52, "end": 634.16, "text": " Markov decision process. But you add an additional set of variables called the free parameters."}, {"start": 634.16, "end": 639.04, "text": " In the papers, we usually use the term theta to denote them. And so those are the positions of"}, {"start": 639.04, "end": 643.68, "text": " where the obstacles are in the maze, in the maze domain. It might be starting position of the agent,"}, {"start": 643.68, "end": 649.1999999999999, "text": " goal position. Inside of a car racing environment, it might be the position of where the tracks are."}, {"start": 650.16, "end": 654.64, "text": " And so these are the design parameters. And so a strategy of the teacher is essentially"}, {"start": 654.64, "end": 661.04, "text": " choose some distribution over choices of the possible free parameters that it can sample"}, {"start": 661.04, "end": 664.48, "text": " as the next level. Sorry, Jack, you go."}, {"start": 664.48, "end": 668.48, "text": " All right. I was going to say like the nice intuitive property of this is that"}, {"start": 668.48, "end": 673.8399999999999, "text": " it makes the agent has to learn to solve all of the simplest solvable environments as well."}, {"start": 673.8399999999999, "end": 678.7199999999999, "text": " So in some other methods, like Poet, they're trying to achieve the maximum complexity,"}, {"start": 679.5999999999999, "end": 683.4399999999999, "text": " which is like, it's very cool and it's well motivated. But this is quite different in that"}, {"start": 683.4399999999999, "end": 687.52, "text": " we're actually happy if even later in training, our agent's training on simple levels,"}, {"start": 687.52, "end": 692.64, "text": " if it means that it can solve all of the simple levels, because we don't really care as much about"}, {"start": 692.64, "end": 698.0799999999999, "text": " solving like crazy complex things if it can break some simple thing, which I think is seems to make"}, {"start": 698.0799999999999, "end": 701.12, "text": " sense, at least to me. Yeah, that was one of my, let's say,"}, {"start": 701.12, "end": 707.6, "text": " worries right here is that if you if you and I framed this a little bit as you are at this"}, {"start": 707.6, "end": 714.24, "text": " zone of proximal development with your agent in that somehow made a drum like you try to"}, {"start": 714.24, "end": 721.2, "text": " reach levels that are just outside of where the agent can handle it. And then you try to edit"}, {"start": 721.2, "end": 726.08, "text": " those a little bit or maybe just where the agent can handle them. And then you try to edit them a"}, {"start": 726.08, "end": 734.16, "text": " little bit. And you try to filter by the ones that pass some threshold in this estimated regret. So"}, {"start": 734.16, "end": 741.6, "text": " my first question would be coming back to this regret, you formulated as the so it's it's a"}, {"start": 741.6, "end": 747.9200000000001, "text": " formulated as the difference to the optimal policy, right? The difference to to the optimal policy,"}, {"start": 747.9200000000001, "end": 755.44, "text": " I'm going to guess on this particular level that you're at. Why doesn't this like disregard the"}, {"start": 755.44, "end": 761.6800000000001, "text": " approximation that you do? If I could calculate this very accurately, wouldn't this select for"}, {"start": 761.6800000000001, "end": 768.0, "text": " super duper difficult levels that that could be solved with the optimal policy, right? Not"}, {"start": 768.0, "end": 774.08, "text": " impossible, but just super difficult ones? That's a great question. I think part of the part of the"}, {"start": 774.08, "end": 780.16, "text": " nuanced detail here is that so one reason that makes this all work is the discount factor. So"}, {"start": 780.16, "end": 787.2, "text": " basically, the so in the original paper that introduced paired and this idea of the mini match"}, {"start": 787.2, "end": 793.6, "text": " regret game, the reward function for that environment actually, it actually your reward,"}, {"start": 793.6, "end": 799.0400000000001, "text": " your final return decreases with the length of your trajectory. And so there's a natural discounting"}, {"start": 799.0400000000001, "end": 804.32, "text": " in terms of the return. And so essentially, by doing many max regret, it ends up prioritizing"}, {"start": 804.32, "end": 808.96, "text": " for those levels where the solutions within reach in the fewest number of steps. And you get this"}, {"start": 808.96, "end": 814.8000000000001, "text": " nice curriculum. But because here in all of our approximate single agent regret estimators, we're"}, {"start": 814.8000000000001, "end": 820.1600000000001, "text": " using a value function, which is bootstrapped off of a generalized advantage estimator, which itself"}, {"start": 820.16, "end": 826.7199999999999, "text": " is discounted. You essentially have discounting built into your value function. And so you end"}, {"start": 826.7199999999999, "end": 831.1999999999999, "text": " up with discounting even if they're even if your environment's a final, you know, sparser word,"}, {"start": 831.1999999999999, "end": 835.8399999999999, "text": " no discounting naturally in the external reward, you still get discounting because your value"}, {"start": 835.8399999999999, "end": 840.3199999999999, "text": " function is going to be discounting using gamma. And if you use GA, you have further discounting"}, {"start": 840.3199999999999, "end": 846.88, "text": " with lambda. Cool. Yeah, that was one of my one of the things that I didn't exactly"}, {"start": 846.88, "end": 853.36, "text": " understand here in this. Okay, I was like, disregard the discount factors, they're not"}, {"start": 853.36, "end": 857.12, "text": " important. Turns out they're actually one of the most important parts right here to actually make"}, {"start": 857.12, "end": 867.76, "text": " it work. Although you use, you use this, this positive value loss. Now, I think you wrote me in"}, {"start": 867.76, "end": 873.92, "text": " an email before that I got this wrong in the in the paper review. Do you want to add something"}, {"start": 873.92, "end": 882.64, "text": " to that? I mean, I guess we can start from sort of the outside in, I guess, or maybe it makes sense"}, {"start": 882.64, "end": 887.8399999999999, "text": " to do the inside out. So basically, the innermost term is essentially just a TD error. It's a one"}, {"start": 887.8399999999999, "end": 892.3199999999999, "text": " step TD error. Yeah. And it's future facing. So it's from your current time step t until the"}, {"start": 892.3199999999999, "end": 898.16, "text": " horizon t capital T. Okay. And essentially, the inner the inner term is essentially the"}, {"start": 898.16, "end": 904.16, "text": " inside out. So it's from your current time step t until the horizon t capital T. Okay. And"}, {"start": 904.16, "end": 911.1999999999999, "text": " essentially the inner the inner term except for within the max, that term is basically, if you"}, {"start": 911.1999999999999, "end": 915.68, "text": " look at the sum from t to capital T, that's basically the generalized advantage estimator"}, {"start": 915.68, "end": 921.68, "text": " from Schumann et al. And so that one is the most common, that's the advantage estimator used in"}, {"start": 921.68, "end": 927.1999999999999, "text": " PPO. It's used in other policy gradient algorithms as well. But essentially, that is essentially"}, {"start": 927.2, "end": 934.08, "text": " the average while trying to do a trade off between one step TD errors being more biased, because"}, {"start": 934.08, "end": 938.8000000000001, "text": " it's bootstrapping off a few steps, and longer TD errors being less biased, but having more"}, {"start": 938.8000000000001, "end": 944.5600000000001, "text": " variance. And so lambda is a discount factor that controls for that. And so in a nutshell, though,"}, {"start": 944.5600000000001, "end": 950.4000000000001, "text": " this is estimating advantage, which is basically, this is my actual return minus my typical return,"}, {"start": 950.4000000000001, "end": 956.0, "text": " which could you can think of as what the value function outputs. And so the zero..."}, {"start": 956.0, "end": 960.56, "text": " Sorry, this is return minus value."}, {"start": 962.32, "end": 966.64, "text": " Yeah, you can think of it as the return you achieved minus your value prediction at each step"}, {"start": 966.64, "end": 971.28, "text": " in your trajectory, and we average it over the trajectory. And essentially, that's telling us,"}, {"start": 971.28, "end": 976.64, "text": " if that's really high, it means that I'm doing better than what I typically do. And so directionally,"}, {"start": 976.64, "end": 981.04, "text": " this is like in the direction of regret, because it means that in terms of external regret,"}, {"start": 981.04, "end": 985.36, "text": " I can actually get a higher return than I typically do, which means that this is a level"}, {"start": 985.36, "end": 991.6, "text": " where I experience regret. And then we max this with zero, which just means that we are only"}, {"start": 991.6, "end": 996.8000000000001, "text": " looking at the positive time steps where at the time steps at which this term is positive. So"}, {"start": 996.8000000000001, "end": 1002.48, "text": " we're only looking when the agent does better than it typically does. And if on average, when it does"}, {"start": 1002.48, "end": 1006.64, "text": " better than it typically does is quite high, it means that's a level where the agent can experience"}, {"start": 1006.64, "end": 1011.84, "text": " a lot of regret in its decision making. How so though, like, my logic was a little"}, {"start": 1011.84, "end": 1019.36, "text": " bit, if I'm worse than I estimated, that means kind of it's a difficult level."}, {"start": 1019.36, "end": 1027.2, "text": " Like, where's my thinking wrong here? So if you're worse than you, if you do worse"}, {"start": 1027.2, "end": 1032.72, "text": " than you estimated, I think in terms of just the mini max regret framework, it's just a little bit"}, {"start": 1034.08, "end": 1039.6000000000001, "text": " sideways from in terms of measuring the direction of regret. I think if you think of it as looking"}, {"start": 1039.6, "end": 1044.32, "text": " for cases where you do better than you typically do, that's really just you discovering regret."}, {"start": 1044.32, "end": 1050.56, "text": " It's like you discovered a case where you achieve regret relative to your typical self, like you as"}, {"start": 1050.56, "end": 1055.52, "text": " sort of amortized by this value function that predicts like how well you typically do over time."}, {"start": 1056.6399999999999, "end": 1062.3999999999999, "text": " So with respect to sort of like this average prediction of yourself, you're doing better."}, {"start": 1062.3999999999999, "end": 1065.9199999999998, "text": " And so you're essentially discovering sources of new regret in this level."}, {"start": 1065.92, "end": 1072.16, "text": " And that's basically directionally aligned with maximizing regret. While if you were to do the"}, {"start": 1072.16, "end": 1077.52, "text": " opposite, if you were to say, I want to look for those steps where I do worse than I think I do."}, {"start": 1078.72, "end": 1082.5600000000002, "text": " I think that's an interesting thing to try actually, but I don't, at least theoretically,"}, {"start": 1082.5600000000002, "end": 1085.68, "text": " it doesn't seem to align with mini max regret as well."}, {"start": 1085.68, "end": 1091.44, "text": " Yeah, okay. I can see the logic in that you say, I want to find levels where there's something"}, {"start": 1091.44, "end": 1098.0, "text": " unexpected positive thing happening. Yeah, I guess it's worth noting as well that in paired,"}, {"start": 1098.0, "end": 1102.48, "text": " which was the first UD algorithm to use regret, they had a very different approaches, which had"}, {"start": 1102.48, "end": 1106.3200000000002, "text": " a second agent called an antagonist. And the regret was just the difference in performance"}, {"start": 1106.3200000000002, "end": 1112.0, "text": " between those two. And so maybe that's like a bit more intuitive, because if the antagonist"}, {"start": 1112.0, "end": 1116.8, "text": " can solve a level and the protagonist, the student agent can't, then I guess that's more intuitive"}, {"start": 1116.8, "end": 1120.16, "text": " in terms of what you expect from regret. But the nice thing about this is it's kind of like"}, {"start": 1120.16, "end": 1126.24, "text": " a cheap approximate for single agent for red. And we definitely feel like maybe coming up with"}, {"start": 1126.24, "end": 1131.28, "text": " better metrics for single agent regret is exciting future work that could be improved upon here."}, {"start": 1131.28, "end": 1135.28, "text": " But this was taken just from the robust PR paper, and we were surprised how well it worked in"}, {"start": 1135.28, "end": 1141.68, "text": " quite different environments. So and another detail is in the robust PLR work, another regress"}, {"start": 1141.68, "end": 1147.52, "text": " meter we used that we explored was what we call maximum Monte Carlo regret estimator. And"}, {"start": 1147.52, "end": 1154.72, "text": " essentially, it's the same, it's almost the same expression, except the regret target is no longer"}, {"start": 1154.72, "end": 1160.4, "text": " what you just received inside of a recent episodic rollout. It's for every level, we keep track of"}, {"start": 1160.4, "end": 1165.52, "text": " the highest return you ever achieved throughout training on that level. And we use that as an"}, {"start": 1165.52, "end": 1169.6, "text": " estimate for the maximum performance on that level. And then we use that as the target to"}, {"start": 1169.6, "end": 1174.24, "text": " subtract your value prediction on. And so that's like a more off policy regret, which I think,"}, {"start": 1174.24, "end": 1178.56, "text": " in some cases might be better because it's less coupled to your current policy. While the positive"}, {"start": 1178.56, "end": 1183.84, "text": " value loss, it's always what you recently received in a rollout in terms of your target minus your"}, {"start": 1183.84, "end": 1189.44, "text": " value function prediction. Yeah, is that is that worth because you would introduce some extra"}, {"start": 1189.44, "end": 1195.2, "text": " variance? Because you're not essentially subtracting your own bait, like use this as a"}, {"start": 1195.2, "end": 1200.64, "text": " baseline in the advantage estimate? Or am I am I seeing this wrong? So this would introduce extra"}, {"start": 1200.64, "end": 1206.8000000000002, "text": " variance. It's not using the policy update, it's used just to score the levels. Yeah, okay. So"}, {"start": 1206.8000000000002, "end": 1211.5200000000002, "text": " essentially, essentially, you're saying the best you've ever done, which might be more, it's going"}, {"start": 1211.5200000000002, "end": 1215.44, "text": " to up and bound your current performance, right? The best you've ever done, including your current"}, {"start": 1215.44, "end": 1221.2, "text": " performance, versus your value function. So it's slightly nicer in the sense that if you've"}, {"start": 1221.2, "end": 1224.96, "text": " experienced level many times, maybe you've had some forgetting, then the regret should be higher"}, {"start": 1224.96, "end": 1228.88, "text": " because you've done well in the past. But but the negative is you have to then score the"}, {"start": 1228.88, "end": 1232.72, "text": " previous episodes for every level. And then oftentimes, you don't actually have any previous"}, {"start": 1232.72, "end": 1237.0400000000002, "text": " experience. So it's not even that applicable. But there's there's there's a trade off. And I"}, {"start": 1237.0400000000002, "end": 1240.64, "text": " think, again, I think there's something that could be improved in future work. So"}, {"start": 1241.2, "end": 1247.2800000000002, "text": " I mean, especially with procedurally generated content, it's, it's probably hard, like you'd have"}, {"start": 1247.2800000000002, "end": 1253.68, "text": " to, you'd have to build some sort of even a model to estimate the best possible regret given past"}, {"start": 1253.68, "end": 1259.52, "text": " procedurally generated levels to sort of predict for any new one. And those two models will probably"}, {"start": 1259.52, "end": 1266.64, "text": " make similar sorts of mistakes, like the mistakes might even be correlated between the okay. So with"}, {"start": 1266.64, "end": 1272.3200000000002, "text": " respect to your your method here, which is is decently simple, what I was surprised by is that"}, {"start": 1272.3200000000002, "end": 1280.48, "text": " you deliberately go away from the teacher being its own agent, right? The teacher here is is,"}, {"start": 1280.48, "end": 1285.76, "text": " if let's say a fixed algorithm, it has some randomized components with the level editing and"}, {"start": 1285.76, "end": 1291.52, "text": " so on. But I think the this this differs from a lot of these kind of curriculum approaches where"}, {"start": 1291.84, "end": 1298.48, "text": " people try to make the teacher deliberately into its own agent and try to sort of frame the"}, {"start": 1298.48, "end": 1304.0, "text": " adversarial setting in terms of to learning things, doing self play, what what kept you from"}, {"start": 1304.0, "end": 1311.28, "text": " doing like, are you are you still convinced? Are you still convinced that this might be a good way?"}, {"start": 1311.28, "end": 1316.48, "text": " Or are you also looking into the direction of making the teacher kind of a learnable component?"}, {"start": 1317.6, "end": 1323.36, "text": " Yes. So I guess the first thing to say is that when we started this project, we actually did"}, {"start": 1323.36, "end": 1328.0, "text": " envisage ourselves using a learned editor. And that was like, what personally, what I was really"}, {"start": 1328.0, "end": 1331.76, "text": " excited about at the beginning was having maybe even a population of editors that make different"}, {"start": 1331.76, "end": 1336.8799999999999, "text": " edits, learn somehow maybe to compete with each other. But the first thing we tried was the"}, {"start": 1336.8799999999999, "end": 1341.76, "text": " simplest thing. And often you hear this in research that the simple thing works surprisingly well."}, {"start": 1342.48, "end": 1347.6, "text": " And so we didn't really feel the need to really go beyond when we got results in mini grid"}, {"start": 1347.6, "end": 1353.04, "text": " initially that were better than anything we've seen before. We felt that it's actually better"}, {"start": 1353.04, "end": 1357.12, "text": " to go with a simpler approach. And maybe in future, we could consider ways to improve this"}, {"start": 1357.12, "end": 1362.3999999999999, "text": " by adding more learned components because that has been the trend elsewhere. But I think going"}, {"start": 1362.3999999999999, "end": 1368.32, "text": " from random sampling to evolution enough, it was was enough to like significantly improve"}, {"start": 1368.32, "end": 1373.36, "text": " based on the previous work. So we didn't need to go all the way to land as well. But"}, {"start": 1373.36, "end": 1378.8799999999999, "text": " even she has some additional thoughts on this. Yeah, I totally agree. I think the"}, {"start": 1378.8799999999999, "end": 1384.08, "text": " simplicity of it just was but it was pleasantly surprising that such a simple method could unlock"}, {"start": 1384.08, "end": 1390.72, "text": " such a big gain in performance. In terms of treating the teacher as an agent, I guess a lot"}, {"start": 1390.72, "end": 1397.52, "text": " of where this work derives from is this paired method, which did treat the teacher as an agent."}, {"start": 1398.08, "end": 1404.24, "text": " And actually the teacher was trained using reinforcement learning. And from based on all"}, {"start": 1404.24, "end": 1408.6399999999999, "text": " the empirical results that we've sort of so far collected in the process of writing these papers,"}, {"start": 1408.64, "end": 1413.5200000000002, "text": " one thing that we have seen is that it seems that RL is not a very efficient way to train an agent"}, {"start": 1413.5200000000002, "end": 1420.0, "text": " to to solve this problem of presenting always the most challenging task for a student. And I think"}, {"start": 1420.0, "end": 1425.44, "text": " the reason is because it's such a highly non stationary problem. Basically, throughout training,"}, {"start": 1425.44, "end": 1429.76, "text": " your students going to get better at certain things, maybe get worse at others. And the policy"}, {"start": 1429.76, "end": 1434.16, "text": " is always evolving. It's very non stationary. So to be able to always track where in the parameter"}, {"start": 1434.16, "end": 1439.6000000000001, "text": " space will correspond to the levels that maximally challenge that of non stationary policy. I think"}, {"start": 1439.6000000000001, "end": 1445.0400000000002, "text": " that's a very hard problem for RL to solve, especially given how sample inefficient RL can be."}, {"start": 1445.0400000000002, "end": 1450.24, "text": " And so what I think one of the reasons why methods like random sampling that PLR does,"}, {"start": 1450.24, "end": 1456.5600000000002, "text": " it works so well, is because it's really able to escape sort of the limitations of RL and just"}, {"start": 1456.5600000000002, "end": 1461.52, "text": " directly sample for points in the space. And you're also not locally bound to the same"}, {"start": 1461.52, "end": 1466.8, "text": " space. And you're also not locally bound to like just only be able to move a small amount based"}, {"start": 1466.8, "end": 1471.44, "text": " on a gradient step. You can really just sample anything anywhere in the space because it's"}, {"start": 1471.44, "end": 1476.96, "text": " randomly searching and then the curator just creates the best ones. So I think that at least"}, {"start": 1476.96, "end": 1483.04, "text": " within these types of domains we've looked at, this type of like random search plus evolution"}, {"start": 1483.04, "end": 1486.8799999999999, "text": " strategy just definitely outperforms a learned teacher."}, {"start": 1486.88, "end": 1496.64, "text": " And in your architecture, I found you mentioned a bunch of times that you are relatively"}, {"start": 1497.7600000000002, "end": 1502.8000000000002, "text": " independent of domain specific heuristics and things like this. Specifically, you criticized"}, {"start": 1502.8000000000002, "end": 1510.48, "text": " Poet for choosing like an arbitrary range of returns of, you know, they just select levels"}, {"start": 1510.48, "end": 1516.5600000000002, "text": " between where the agents achieve between 50 and 300, which they claim to be, you know, hard but"}, {"start": 1516.56, "end": 1523.12, "text": " not hard, but not too hard. And yet I find, for example, in your algorithm, you need something"}, {"start": 1523.12, "end": 1530.1599999999999, "text": " like, well, we only put something into the buffer if the regret is above a certain threshold."}, {"start": 1530.1599999999999, "end": 1535.36, "text": " Couldn't I leverage kind of the same criticism to you and say, well, probably that threshold is"}, {"start": 1535.36, "end": 1542.72, "text": " going to be problem specific, right? And it's kind of a hyper parameter that doesn't seem like"}, {"start": 1542.72, "end": 1548.96, "text": " it's dependent on the environment, but is it? I think you're right that this is dependent on"}, {"start": 1548.96, "end": 1554.48, "text": " the domain, but I'll say that the specific point about the hyper parameter, that one is actually"}, {"start": 1554.48, "end": 1561.1200000000001, "text": " a bit more benevolent of an issue, I think, because that's actually not a hyper parameter"}, {"start": 1561.1200000000001, "end": 1566.16, "text": " in our method. It's just whatever is the lowest score inside the buffer is the threshold."}, {"start": 1566.16, "end": 1573.3600000000001, "text": " Okay. But I think that's definitely, I think if someone like you, I think, read it that way,"}, {"start": 1573.3600000000001, "end": 1576.88, "text": " I think we should definitely reward that in the paper. I think that's definitely going to be an"}, {"start": 1576.88, "end": 1581.28, "text": " improvement to clarity on that point. But it's basically the threshold is basically whatever is"}, {"start": 1581.28, "end": 1586.3200000000002, "text": " the lowest score in the level buffer. And if it's better than the lowest one, we replace it. So it's"}, {"start": 1586.3200000000002, "end": 1592.64, "text": " kind of like a priority queue in terms of the regret. But I agree with you. I think that"}, {"start": 1592.64, "end": 1598.96, "text": " methods like Excel and methods that basically require you to directly modify levels to construct"}, {"start": 1598.96, "end": 1603.68, "text": " them. I think these types of methods are always going to be domain specific, because I think at"}, {"start": 1603.68, "end": 1608.0, "text": " the end of the day, you need to have a way of parameterizing the environment. And that's domain"}, {"start": 1608.0, "end": 1611.3600000000001, "text": " knowledge. And you need to parameterize how you're editing that level."}, {"start": 1614.4, "end": 1620.0, "text": " Yeah, I guess the editing itself is also, I think it's more, there's probably more domain"}, {"start": 1620.0, "end": 1625.84, "text": " knowledge than one cares to admit, because yeah, you think like, okay, in block world,"}, {"start": 1625.84, "end": 1631.68, "text": " I'm just modifying one block to be there or not, right? But there is a decision of, you know,"}, {"start": 1631.68, "end": 1637.84, "text": " do I modify one block? Do I modify a block of blocks? Do I place a wall, an entire wall or not?"}, {"start": 1637.84, "end": 1642.24, "text": " And things like this, and depending on how much you edit, because you have this assumption, right?"}, {"start": 1642.24, "end": 1650.32, "text": " Which is that if I modify, if I make, like my modifications need to be small enough such that"}, {"start": 1650.32, "end": 1655.84, "text": " they don't influence the hardness of the level too much, yet they need to be large enough such"}, {"start": 1655.84, "end": 1661.1200000000001, "text": " that they do bring some variation into the picture, right? And that balance, do you think"}, {"start": 1661.1200000000001, "end": 1669.28, "text": " that balance, it might be easy in these kinds of levels? What, like, how do you find this balance"}, {"start": 1669.28, "end": 1673.84, "text": " in more challenging problems? Like, I don't know if you think further."}, {"start": 1673.84, "end": 1681.2, "text": " So I guess in these problems, it's worth noting that for the block situation, the actual domain"}, {"start": 1681.2, "end": 1685.76, "text": " randomization process places the blocks one at a time. So all we're really doing is kind of saying"}, {"start": 1685.76, "end": 1690.72, "text": " you have a few more steps with that initial process. So it is fairly aligned with the whole"}, {"start": 1690.72, "end": 1696.6399999999999, "text": " problem there. And then in the bipedal walker setting, we're just making small changes to the"}, {"start": 1696.64, "end": 1702.0800000000002, "text": " encoding vector. And in both settings, we have these details of this in the appendix if you"}, {"start": 1702.0800000000002, "end": 1706.24, "text": " dare to venture. But in both settings, we did sort of a sweep over the number of edits you can make"}, {"start": 1706.24, "end": 1712.64, "text": " in one go. And in both cases, we found that all the values worked well. We obviously"}, {"start": 1712.64, "end": 1718.48, "text": " picked the one that was the best performing on our validation sets. But it seemed fairly robust"}, {"start": 1718.48, "end": 1723.2, "text": " to the number of edits you make. And the thing worth noting again there is that what you could do"}, {"start": 1723.2, "end": 1727.28, "text": " is if, for example, you don't care as much about the number of samples you use to find a high"}, {"start": 1727.28, "end": 1732.64, "text": " regret level, you could just try all of these values in one batch. And then because with PLR"}, {"start": 1732.64, "end": 1736.96, "text": " based methods, you just curate the ones that have high regrets. You could say, okay, I'm going to do"}, {"start": 1736.96, "end": 1740.4, "text": " some with one edit, some with two, some with three, some with four, or whatever it might be."}, {"start": 1740.96, "end": 1745.2, "text": " And you could almost scale the size of the edits. And then just from that batch, just take the high"}, {"start": 1745.2, "end": 1749.1200000000001, "text": " regret ones. And you're probably still going to have more new high regret levels than you would"}, {"start": 1749.12, "end": 1753.4399999999998, "text": " if you randomly sample from the initial distribution. So I think that there is some"}, {"start": 1753.4399999999998, "end": 1758.7199999999998, "text": " flexibility to do something like that. And I would argue that you could frame a lot of things"}, {"start": 1759.28, "end": 1764.1599999999999, "text": " in this editing sort of framework. And I think we mentioned a couple of examples like perturbing"}, {"start": 1764.1599999999999, "end": 1770.0, "text": " latent, latent generative model, for example, that may be seen as more general than specific"}, {"start": 1770.0, "end": 1776.6399999999999, "text": " encodings for environments. It is a good point. I want to stick on this a little bit. The types"}, {"start": 1776.64, "end": 1782.64, "text": " of problems where these methods are applicable, because they seem very general, yet it feels like"}, {"start": 1782.64, "end": 1788.48, "text": " you need a problem where you can construct such a curriculum and that curriculum needs to be fairly"}, {"start": 1789.1200000000001, "end": 1796.3200000000002, "text": " smooth, let's say so that the difficulty increase needs to be manageable and so on. And also, the"}, {"start": 1796.3200000000002, "end": 1802.3200000000002, "text": " regret the way you calculate regret with the with the TD error. It means that probably an"}, {"start": 1802.32, "end": 1811.28, "text": " environment like the walker where I, you know, I get more reward, the further I go, is probably"}, {"start": 1811.28, "end": 1816.48, "text": " more conducive than something like a Montezuma's revenge, even though I have a TD error and so on"}, {"start": 1816.48, "end": 1823.6799999999998, "text": " that kind of smooths out the loss itself. Can you comment a little bit on what kind of problems"}, {"start": 1823.6799999999998, "end": 1830.8799999999999, "text": " would like, where would it start to struggle? Like, where would you probably have trouble"}, {"start": 1830.88, "end": 1834.88, "text": " applying something like this? And where would it work? Obviously, it works super well on these"}, {"start": 1834.88, "end": 1840.16, "text": " types of things that you tried it on. But where would it struggle? Yeah, I think you're right."}, {"start": 1840.16, "end": 1847.3600000000001, "text": " It's got to be a domain where you do have some structure that progressively gets, you know,"}, {"start": 1847.3600000000001, "end": 1853.6000000000001, "text": " goes from simpler to more complex. And it's, I guess, one nice benefit of these methods is that"}, {"start": 1853.6000000000001, "end": 1859.0400000000002, "text": " you don't need to know ahead of time, what exactly does it mean for a level in this domain to be"}, {"start": 1859.04, "end": 1865.68, "text": " easier or hard? Because we have this regret based heuristic to tell us that. And if you do have sort"}, {"start": 1865.68, "end": 1870.8799999999999, "text": " of this progressive structure within the domain, then these methods can sort of start to merge that"}, {"start": 1871.44, "end": 1876.56, "text": " based on this heuristic. But I think that at least with these PLR based methods, because the core is"}, {"start": 1876.56, "end": 1881.92, "text": " still needle in the haystack, you're looking for high regret levels by random search, and then"}, {"start": 1881.92, "end": 1887.6, "text": " evolution in Excel just massively augments that in terms of the amount of training data you can get"}, {"start": 1887.6, "end": 1893.76, "text": " from high regret levels. But the bottleneck step is still sort of like this limitation around at"}, {"start": 1893.76, "end": 1899.1999999999998, "text": " some point, you still have to just get that needle in the haystack. And so I think as the design space,"}, {"start": 1899.1999999999998, "end": 1905.28, "text": " like the dimensionality of your environment gets bigger and bigger, I would expect that these"}, {"start": 1905.28, "end": 1911.04, "text": " methods become less and less efficient. Do you... Yeah, a couple of... Sorry."}, {"start": 1911.04, "end": 1940.08, "text": " I think we have like a one second lag or so. All right, sorry. So I guess one other thing, one perspective of this is it's really just a black box optimization problem where the function returns regret. And so we've gone from random sampling to evolution. But if you look at black box optimization literature, there are plenty of methods that trade off between global and local optimization in a more elegant way. And so what you could do is have some model or a project that's more like a black box optimization."}, {"start": 1941.04, "end": 1970.08, "text": " So you could have a model or a project that maybe samples points more like diversity in the space. And then you use something like Excel locally to make edits once you found that needle in the haystack that Min-Shi mentioned. And then the second thing is that I think one place where this might break down is because it is quite a kind of greedy local optimization process is if you haven't got sort of a very clear, like high to low environment, then maybe you need something to encourage diversity."}, {"start": 1970.08, "end": 1993.6799999999998, "text": " So you could have some sort of like either buffer could be maybe like hierarchical or something or you could try and preserve levels that you think are conducive to edits later on, even if they're not the current high regret levels. And these are all ideas we talked about for future work. I think really what we need is we need to have these more challenging problems to actually break our current methods before we can really think of the hammer for these nails."}, {"start": 1993.68, "end": 2023.28, "text": " But yeah, what what is a bit special as well is that you train a single agent, right? Because usually the evolutionary methods they are trying to get a population of agents to work even if they want to end up with a single agent very often. And you encode all of this into a single agent. And that's kind of a PPO really basic agent, if I want to say and I have noticed a little bit that in these demonstrations, no matter what the level is, kind of the"}, {"start": 2023.28, "end": 2042.6399999999999, "text": " strategy tends to be the same, right? It tends to kind of, it tends to hop on this one leg with the other one with the other one out. And that is sort of the best strategy to overcome any and all obstacles. And then kind of rebalance itself once it's yeah, this one see? So"}, {"start": 2042.64, "end": 2072.2400000000002, "text": " maybe maybe we've been walking wrong our whole lives. But no, I mean, it's it's it's obvious if you if you instill this in a single agent, how much of a how much because I also observed some of your results here over time, which was also really cool to see when you compare to the to the poet algorithm in that you do get kind of more challenging levels later on, but they also like they don't dominate it doesn't get more and more and more and more challenging."}, {"start": 2072.64, "end": 2098.16, "text": " Right? How much of this is a property of like catastrophic forgetting of the agent itself where you kind of push for the more complicated levels, but all of a sudden, it can't can't solve the easy ones anymore. And therefore, the easy ones become high regret. And then there's kind of this, like how much of this is due to your algorithm? And how much of this is due to the fact that you have a single agent trained with PPO that needs to take care of all of these tasks at the same time?"}, {"start": 2098.16, "end": 2100.16, "text": " My guess is it's the latter part, because I think that having this buffer that we do have, which in the robust PLR and the previous PLR paper, it does somewhat help with forgetting because you're able to sample things you haven't seen for a while. And if and if you now can't solve them as well, or, or if you now have higher regret in these levels, then you should retrain on them. So it should somewhat eliminate forgetting. But I do think it's worth noting that the"}, {"start": 2128.96, "end": 2156.04, "text": " this agent is just a two hidden layer neural net policy. It's not not very flexible. It's pretty, like low dimensional. And I think it really is unable to adapt to every different possible behavior. And so I think either having something where you can curve all the architecture as well to maybe make it more flexible as the levels get harder, or even just making your agent be some sort of adaptive agent, like a meta learning algorithm, for example, that does zero shot adaptation."}, {"start": 2156.04, "end": 2173.04, "text": " I think these approaches are things that we're excited about maybe for future work. But I think for this, it's sort of an inevitability that if you try and have this like lofty goal of having a generally capable agent, it's going to have some brittleness to some certain components. I think we found a few cases like uphill, it's not particularly good."}, {"start": 2173.04, "end": 2203.04, "text": " Yes. When we started visualizing it in this viewer that we have the demo, we noticed that, you know, like, when we were training this thing, all the complexity metrics for like roughness of the ground, it started going up very quickly. But then when we actually printed out a lot of the levels where it's successful, they tend to be levels where it's all downhill, which means that this pogo stick strategy, it's very good at just like hopping down the hill. And it's really robust at landing, like just sticking the landing in terms of like really high clicks. But like, it's like, it's a really good"}, {"start": 2203.04, "end": 2233.04, "text": " when you start to get more like these rugged hills going uphill, where the slope is positive. That's where it starts to struggle. So that's like a really interesting and I think a very tangible sort of example, where there's sort of a collapse in diversity in a way in the curriculum where because it is a limited, we do replay old levels. But again, it's a limited finite buffer. So you can get, you know, sort of like a buffer overflow in a sense of, you know, lots of levels that collapse in terms of similar challenges. And then maybe the agent just gets too"}, {"start": 2233.04, "end": 2244.24, "text": " good at going downhill, jumping down really challenging hills. But then it starts to the curriculum starts to forget that also going uphill is also important. And maybe that's what happened in some of these training runs."}, {"start": 2245.16, "end": 2262.96, "text": " I like the I like the approach. I think poet or poet v2 had some sort of an approach where they do, of course, have different agents. But they had this metric of ranking the environments that they have in the buffer, right and sort of ranking them with respect to different agents and their"}, {"start": 2262.96, "end": 2280.16, "text": " conclusion was that if the if the different agents rank the environments in a different way, that kind of indicates a diversity of levels, right? Whereas if they rank them the same way, it's kind of like, well, it's, they're not really diverse. I think much like your regret"}, {"start": 2280.16, "end": 2293.56, "text": " measure, I'm big fan of these. They're not super domain independent, but they are domain independent enough, right? So that you could like you can kind of disconnect them from the real problem at hand. That's pretty cool."}, {"start": 2294.04, "end": 2305.64, "text": " That one is definitely I think more general. Yeah, I think that's quite an exciting approach. Maybe if you wanted to use population, maybe even generate experiences, then that's quite a nice way of evaluating the diversity, I think."}, {"start": 2305.64, "end": 2334.08, "text": " So is it fair to say that kind of the end here, like the most, let's say you train this, let's assume this is convergence at 5000 step that this is kind of a representation. Like it's almost like a fingerprint of the agent's ability in the face of a curriculum that tries to push harder and harder, right? Because there's a trade off that the easy levels, not being in the buffer or not being, yeah, not being in the buffer, is kind of like a"}, {"start": 2334.08, "end": 2350.2, "text": " yeah, not being in the buffer means they're easy. They can be solved, right? But then also, this is it seems like this is the curriculum that's needed for the agent to be as general as possible, not necessarily as good as possible."}, {"start": 2350.2, "end": 2372.8799999999997, "text": " So yeah, I think it's worth noting as well that Minchi added a really cool feature to the website where you can actually see five seeds of each method. I don't know if you've seen that version, but you can see that the Excel agents are pretty remarkably similar. So they almost all seem to follow quite a similar gate, which makes me think that this is kind of the solution that for this network does cover the space as best as possible."}, {"start": 2372.88, "end": 2393.2000000000003, "text": " And so it might be the case maybe that to get to get better behavior, better performance, maybe you need to have a Lego shovel seeds, maybe you need to have something that's a little bit more flexible, either something with memory, or I think I think some some implementations by pedal Walker use frame stacking, these types of things, maybe you can get more capacity into the network."}, {"start": 2393.2, "end": 2408.56, "text": " That way, I think it's probably possible or likely that Lego is, it's probably quite likely that this this is the best policy you can get with this network to have as many max regret approach."}, {"start": 2410.2799999999997, "end": 2412.04, "text": " Yeah, well, there is one survivor."}, {"start": 2412.64, "end": 2414.08, "text": " Well, we'll see. We'll see."}, {"start": 2414.08, "end": 2439.48, "text": " Yeah, excellent. Cool. Yeah, the website is the website is definitely pretty cool. The last interesting thing I found at least for me here was this generalization to the to the maze. And, I mean, it's it's very cool, because you, you train on these on these made up mazes starting from empty rooms, and then you test on these kind of humanoid mazes."}, {"start": 2439.48, "end": 2464.88, "text": " Right here, and then you generalize to this giant maze here. Now, you say yourself, the agent seems to follow this kind of bit of a left hand rule. How does something like this emerge? Because it doesn't seem like in the generated levels, a left hand rule would be beneficial, because there's a lot of loops and stuff in that. So, you know, it's a little bit of a puzzle."}, {"start": 2464.88, "end": 2474.48, "text": " But it doesn't seem like it's going to be beneficial, because there is a lot of loops and stuff in that, like how does how does a strategy like this emerge?"}, {"start": 2476.6400000000003, "end": 2489.56, "text": " I guess one thing that's quite worth noting in this environment is partially observable. So, you only need to regenerate a small bit of structure within within the grid for it to kind of generalize maybe to larger grids."}, {"start": 2489.76, "end": 2491.8, "text": " It only seems the blue area, right?"}, {"start": 2491.8, "end": 2507.1600000000003, "text": " Yeah, exactly. And that actually makes this really hard. So even for a human, if you imagine you didn't know where the green dot was and try and do this as a five hours, humans would not be able to. Yeah, I certainly lost patience with it after a couple of goes, there's like a 5000 step limit. So it's quite long."}, {"start": 2507.16, "end": 2537.16, "text": " But if yeah, if you look at the Excel, sort of towards the end of training as well, in the mini grid domain, a lot of the levels, so it ends up converging towards around 60 block count. And that's sort of like the threshold beyond which a lot of the levels where you randomly sample like more than 60 blocks, they tend to be unsolvable. So they tend to have a block preventing you from getting to the goal. And so 60 seems to be like the sweet spot for a 15 by 15 maze. And when you get to that level, you're like, okay, I'm going to"}, {"start": 2537.16, "end": 2563.04, "text": " set like that amount of saturation of blocks, a lot of the levels tend to actually become effectively single component mazes. And so those are unsolvable by the left hand rule. So I think that's also like just a contributing factor, like some property of the specific dimensionality that we looked at resulted in, you know, the complexity converging to like lots of mazes that are single component. And it helps the agent basically learn this left hand rule."}, {"start": 2563.04, "end": 2583.36, "text": " Yeah, it's pretty cool. Do you I didn't dive too much into the experimental results in my review. Is there like, what are some of the things that you might want to highlight across your experimental results, maybe that you you find more interesting than the average person would when they read the paper?"}, {"start": 2583.36, "end": 2611.56, "text": " I guess for me, it's two things. So the first one is that the complexity is entirely emergent. So we never encourage the agents to actually increase the block count. We never encourage it to increase the stump height and bipedal walker, it just has to do that to increase the grip. So some other papers maybe all works, maybe they have some like ways to encourage this, whereas we actually didn't. So if we were to do that, maybe in the future, that's could even increase even further. And then the second thing is that all of the test cases are zero shot value. So that's a really cool thing."}, {"start": 2613.7200000000003, "end": 2624.1200000000003, "text": " So the agents never seen test levels. And I think it's quite remarkable how robust it is in quite a wide range of settings. So that's probably the two takeaways for me."}, {"start": 2624.12, "end": 2654.12, "text": " We also had some results in the appendix where we actually, we also test the final Excel bipedal walker agent on top of the poet levels. So in poet, actually, they publish a few of the rose plots showing the different parameter settings for bipedal walker for some of the crazier environments. And we actually tested bipedal walker, our bipedal walker with Excel on those environments. But it actually it didn't perform very strongly. So it's what's interesting is that we actually tested the"}, {"start": 2654.12, "end": 2682.16, "text": " bipedal walker with the same parameters. So that's what we're seeing. And then the third thing is, I think what's interesting about this result is, it sort of highlights this duality between like, the goals of these two algorithms, where I kind of see Excel as being on one side of the spectrum, which is about robustness, general robustness to unknown environments, and poet be on the other side of the spectrum, where it's focused on getting specialists for basically finding these agent environment specialist pairs, where this agent just always solves this environment. And so, you know, I think that's a really cool thing."}, {"start": 2684.12, "end": 2707.2, "text": " It's kind of an interesting philosophical idea, because it's kind of saying that if you're building an AI system, do you really care about being robust to things that you don't know about? Or do you want to maximize your performance as a specialist? And I think it's a really interesting open question. And the way we navigate this trade off, I think is really full of rich ideas for future research projects."}, {"start": 2707.2, "end": 2728.2, "text": " Yeah, especially ideas that could combine some of these things as well. And we've obviously talked about a lot of possible things. But I'm actually used to go a bit, if you go a little bit few pages down, what we did was, we actually took the some of the most complex levels that poet generates. And then we produced, we produced them in our own setting. And that's also 100 by 100 mates, if you're interested."}, {"start": 2729.2, "end": 2731.2, "text": " 100 by 100. Did it solve it?"}, {"start": 2731.2, "end": 2742.2, "text": " Yeah, it has to be odd number for the for the simulators to work. Okay, okay. That one gets the 8% success rate on that one. It's I think a bit above this."}, {"start": 2743.2, "end": 2744.2, "text": " Is it a table?"}, {"start": 2745.2, "end": 2749.2, "text": " Yeah. Higher up higher up. Maybe."}, {"start": 2751.2, "end": 2752.2, "text": " What are you looking for?"}, {"start": 2752.2, "end": 2753.2, "text": " The poet."}, {"start": 2753.2, "end": 2761.2, "text": " Yeah, it should be a small it's like a very small table. I think it's down below more search in the paper itself, I guess."}, {"start": 2763.2, "end": 2766.2, "text": " We should probably have paper up on our own screen."}, {"start": 2767.2, "end": 2770.2, "text": " Well, my bad for for not knowing it too well."}, {"start": 2771.2, "end": 2773.2, "text": " Oh, yeah. It's potentially on the next page."}, {"start": 2774.2, "end": 2777.2, "text": " This is the like main experiments on the test cases."}, {"start": 2777.2, "end": 2778.2, "text": " I think it must be on the next page."}, {"start": 2778.2, "end": 2779.2, "text": " I think it's on the next page."}, {"start": 2779.2, "end": 2783.2, "text": " Ah, this is the one."}, {"start": 2783.2, "end": 2793.2, "text": " Yeah, so 1a to 3b are in the paper towards the end. They have like a rose plot for some of the most extremely challenging levels that each of their seeds generated."}, {"start": 2794.2, "end": 2803.2, "text": " So for all three of their seeds, they pick two different levels that they're particularly high values. And we tested our agent zero shot on those."}, {"start": 2803.2, "end": 2816.2, "text": " And yeah, the scores are pretty low. But I think the fact that they're above zero is cool. But at the same time, it does make you think that if they can solve those repeatedly, then maybe you do need specialists in some cases to get the most complex things."}, {"start": 2816.2, "end": 2821.2, "text": " So some hybrid specialists and generalists might be an even more powerful algorithm than either of them combined."}, {"start": 2823.2, "end": 2824.2, "text": " Excellent."}, {"start": 2824.2, "end": 2842.2, "text": " So you mentioned a bunch of different and you also have a future work section and so on. What do you think are apart from the things you're going to do next? What are like the big unsolved challenges in the field? Like what's everyone after but no one's been able to do it so far?"}, {"start": 2842.2, "end": 2861.2, "text": " Well, so the big one is a theme that we as a group have gotten very interested in recently, and we're actually holding a workshop at iClear about this. And essentially, it's about agent environment co-evolution. But in this, in the context of this much older problem called open-endedness."}, {"start": 2861.2, "end": 2886.2, "text": " And basically, open-endedness is an idea that kind of came from a group of researchers, Ken Stanley, Joel Lehman, and Jeff Klune. And I think Jeff Klune has this concept of AI generating AI. And it's related to this idea of open-endedness where can you basically create a learning system that essentially ends up evolving just an unbounded amount of novelty and knowledge."}, {"start": 2886.2, "end": 2903.2, "text": " And if you can kickstart a process that achieves true open-endedness, then the idea is that maybe you can replicate the emergence of some really complex intelligences, like human level intelligence, because evolution, like the tree of life, this is all sort of the result of an open-ended learning process."}, {"start": 2903.2, "end": 2925.2, "text": " And so a lot of where we see this work going is that we see our work as sort of fitting within this bigger theme of open-endedness, and this larger theme of agent environment co-evolution to achieve this open-endedness. And so I think that that sort of to me is one of the most interesting open problems in AI or machine learning, or maybe it goes beyond even these two subjects."}, {"start": 2925.2, "end": 2933.2, "text": " Yeah, so I think that if we can actually kick off a process like this, that would be incredible. And I'd be very curious to see what kinds of things fall out of it."}, {"start": 2935.2, "end": 2946.2, "text": " Yeah, and for me, the thing I'm really excited about is that, again, tying in with Minshe's is this seems like the only limitation to this really being open-ended is requirement for a simulator."}, {"start": 2946.2, "end": 2963.2, "text": " So I'm really excited about whether we can actually learn simulators, for example, world models. So I was obviously very inspired by the Harald Rydhugel work from 2018. But more modern, like offline RL world models. So maybe you have some transformer world model that learns from all this crazy amount of data."}, {"start": 2963.2, "end": 2976.2, "text": " And then you can use that to design environments for an RL agent and then collect more data and just keep going. And maybe that's how you really get towards this true open-endedness, because you're not bounded by just the open AI gym environment that you're given."}, {"start": 2977.2, "end": 2990.2, "text": " And so this is maybe it's a little bit more of a medium to long term goal, because I think we're a bit away from that right now. But I think that that could be where we can actually learn more about open-endedness."}, {"start": 2990.2, "end": 2998.2, "text": " And I think we're a bit away from that right now. But I think that that could be where these different fields intersect and really produce something pretty crazy."}, {"start": 2998.2, "end": 3019.2, "text": " Yeah, my issue a little bit with the agent environment co-evolution work is that it just seems to kind of shift the problem away from because, okay, we're evolving the environments right here, but they're still like extremely bounded in an extremely parameterized space, right? And there's only like these many ways that the environment can vary."}, {"start": 3019.2, "end": 3032.2, "text": " And the true environment is kind of like the environment generator itself. And it seems like, you know, we could go a level higher and so on. But how is there a method to generally break out of this?"}, {"start": 3033.2, "end": 3047.2, "text": " I think one way is, you know, it's related to what Jack just described, which is this. So you've heard of SIM to real as the paradigm where you train intelligence in simulation, you transfer to reality."}, {"start": 3047.2, "end": 3059.2, "text": " And that's obviously bounded by the fidelity of your simulator for your target domain. There's a new paradigm emerging. And it's like sort of pushed by all these advances in computer vision, which some people have called real to SIM to real."}, {"start": 3060.2, "end": 3072.2, "text": " And basically, the idea that you can essentially collect data in a loop where you know, you may have some exploratory agent, maybe it's a hand-coded controller, or maybe it's an RL agent, the one you're training, and you send it out into the wild."}, {"start": 3072.2, "end": 3084.2, "text": " It collects lots of data about what the world is like. And then you use that data to essentially enrich your simulator to basically fit your simulator to reality, to all the new things it's learned. And then you get a better, more expansive simulator."}, {"start": 3085.2, "end": 3094.2, "text": " You train your agent again in that simulator, and you get a new agent to transfer to reality. And then this loop just keeps repeating. And maybe you can do this in a population of agents doing this."}, {"start": 3094.2, "end": 3107.2, "text": " And you get really huge coverage in terms of what's out there. I think that's one promising way to do it. The other though, is I think it kind of just generally the strategy is, like you said, all these simulators are bounded in terms of their parameterization."}, {"start": 3108.2, "end": 3119.2, "text": " Like we're looking at 15 by 15 nases, there's a finite number of them. I think what would be really cool is if we started as RL researchers started focusing more on environments that are unbounded in parameterization."}, {"start": 3119.2, "end": 3127.2, "text": " So moving into these like more almost non parametric settings, where the environment can just keep growing arbitrarily in its number of parameters."}, {"start": 3128.2, "end": 3137.2, "text": " And I actually think the real to sim to real loop is one way to do that, just because the space of possible worlds you can represent as a world model as a neural network is pretty much infinite."}, {"start": 3138.2, "end": 3143.2, "text": " But maybe there are other simpler ways you could do this as initial toy tests as well."}, {"start": 3143.2, "end": 3148.2, "text": " And then when you have that real sim to real world model, you can then training meaning max regret policy inside it."}, {"start": 3149.2, "end": 3149.2, "text": " Yeah."}, {"start": 3150.2, "end": 3161.2, "text": " Because then you have like this idea of the population generating this diverse, you know, very high dimensional world model, but then a single agent, maybe that could be generally robust to any possible variation."}, {"start": 3162.2, "end": 3167.2, "text": " And so this is maybe a bit of a medium term. But I think for us, it's kind of an all star at the moment."}, {"start": 3167.2, "end": 3178.2, "text": " Do you think there will ever be sorry, last question by me, do you think there will ever ever be this distinction between agent and environment will will this continue to be an important distinction?"}, {"start": 3178.2, "end": 3191.2, "text": " Or is that something that you see in the future vanish and kind of almost become like, let's say interchangeable because people are already like pitting them against each other training them both with RL and so on."}, {"start": 3191.2, "end": 3193.2, "text": " Like, why do we even make the distinction?"}, {"start": 3193.2, "end": 3206.2, "text": " Well, I guess one thing that's interesting is even in the original world models paper, because the world model itself was generated model, the policy was very low dimensional, and it just trained inside the latent state, latent space of the world, the generative model."}, {"start": 3207.2, "end": 3214.2, "text": " So then when you're actually interacting with the real environment, you still use the encoder from the world model to process the input so that the policy can then operate."}, {"start": 3214.2, "end": 3224.2, "text": " And so in that sense, it's like the world model is the environment at training time offline. But then at test time, when you go back to the real environment, the world models used to process the inputs for the policy."}, {"start": 3224.2, "end": 3237.2, "text": " And so they're kind of taking a very, like, I guess, competitive and then a cooperative mindset. So I think maybe there's something like that, where you have well models that are your environment for training time, but then you use them as knowledge bases for test time."}, {"start": 3237.2, "end": 3255.2, "text": " I think that's pretty exciting. And it also kind of relates this idea of the cherry on top, because the policy is very small, although I hate to use too many cliches, but it does seem to relate to that sort of self supervised learning large world models, and then RL just for controllers inside that that can operate on the representations."}, {"start": 3256.2, "end": 3258.2, "text": " I don't know, Minchi, thanks for that."}, {"start": 3258.2, "end": 3273.2, "text": " Well, I think to sort of answer the other side of that question, I think that agent environment, I guess the distinction is, in some ways, it's arbitrary, because you can imagine, you know, like, what part of this learning system actually belongs to the agent?"}, {"start": 3273.2, "end": 3287.2, "text": " Like, is the agent really like at the activation level? Is it at the observation level? Like, where do you even draw the boundary in terms of the agent? I think that's an interesting question. But I also think that at some point, there's going to be some substrate in which the agent has to operate within."}, {"start": 3287.2, "end": 3305.2, "text": " And there seems to be, like, basically, if you wanted to emerge a diverse sort of, you know, tree of life of different RL agents and environments, it seems like there is some sort of asymmetry there in the sense that agents have to operate within an environment, and you can't have it reversed."}, {"start": 3305.2, "end": 3325.2, "text": " And so in some, to some extent, I think we'll still have to have this distinction between agents and environments. But it's also possible, you know, like, maybe we could also just learn, you know, joint distributions over agents and environments where you basically just learn, you know, like, the agents parameters themselves are now part of the environment design."}, {"start": 3325.2, "end": 3336.2, "text": " And so now you're just emerging agents and environments together inside of a single generative model. I think that's an exciting idea. But and maybe at some point, we'll figure out how to do that."}, {"start": 3338.2, "end": 3341.2, "text": " Where can people get started with this if they want to dive into it?"}, {"start": 3341.2, "end": 3354.2, "text": " So there's a great for open endedness, there's a great primer to it on O'Reilly, I can actually send you the link after. But it's written by some of the original sort of pioneers within this field. And essentially, it's quite long, but it summarizes the whole field."}, {"start": 3354.2, "end": 3371.2, "text": " Another, another really interesting work would be I think, just to check out the original mini max regret paper for RL, which is this emergent complexity for zero shot generalization from Michael Dennis and Natasha."}, {"start": 3371.2, "end": 3387.2, "text": " Jake jacks, and I would definitely recommend, you know, our line of work with robust pillar checking out this paper. And there's older methods like teacher student curriculum learning from shum shumans group at open AI. And"}, {"start": 3387.2, "end": 3404.2, "text": " workshop. Yeah. So we're going to have an iClear workshop called agent learning in open endedness, aloe. And that's going to feature a lot of speakers and researchers actively making progress in this field. So if people are really interested, they should check that out."}, {"start": 3404.2, "end": 3421.2, "text": " Yeah, that's April 29, April 29. Yeah, Friday. Good. Also, more in a multi agent setting, there's the curriculum learning manifesto from Joel, Joel, Lebo at DeepMind. And we're going to have a lot of people in the audience. So if you're interested, you can check out the poster session."}, {"start": 3421.2, "end": 3442.2, "text": " Yeah, that's April 29, April 29, April 29, Friday. Good. Also, more in a multi agent setting, there's the curriculum learning manifesto from Joel, Joel, Lebo at DeepMind. And that has some really nice ideas in terms of automatic curriculum learning, emerging emergent complexity."}, {"start": 3443.2, "end": 3444.2, "text": " Cool."}, {"start": 3445.2, "end": 3449.2, "text": " Minchi and Jack, thank you very much for being here. This was really cool."}, {"start": 3449.2, "end": 3450.2, "text": " Thank you for having us."}]