Greg Brockman: OpenAI and AGI | Lex Fridman Podcast #17
 The following is a conversation with Greg Brockman. He's the co-founder and CTO of OpenAI, a world-class research organization developing ideas in AI with the goal of eventually creating a safe and friendly artificial general intelligence, one that benefits and empowers humanity. OpenAI is not only a source of publications, algorithms, tools, and data sets. Their mission is a catalyst for an important public discourse about our future with both narrow and general intelligence systems. This conversation is part of the Artificial Intelligence podcast at MIT and beyond. If you enjoy it, subscribe on YouTube, iTunes, or simply connect with me on Twitter at Lex Friedman, spelled F-R-I-D. And now, here's my conversation with Greg Brockman. So in high school and right after you wrote a draft of a chemistry textbook, I saw that. That covers everything from basic structure of the atom to quantum mechanics. So it's clear you have an intuition and a passion for both the physical world with chemistry and now robotics to the digital world with AI, deep learning, reinforcement learning, and so on. Do you see the physical world and the digital world as different? And what do you think is the gap? A lot of it actually boils down to iteration speed, right? That I think that a lot of what really motivates me is building things, right? Is the, you know, think about mathematics, for example, where you think really hard about a problem, you understand it, you write it down in this very obscure form that we call proof. But then this is in humanity's library, right? It's there forever. This is some truth that we've discovered. You know, maybe only five people in your field will ever read it, but somehow you've kind of moved humanity forward. And so I actually used to really think that I was going to be a mathematician. And then I actually started writing this chemistry textbook. One of my friends told me, you'll never publish it because you don't have a PhD. So instead, I decided to build a website and tried to promote my ideas that way. And then I discovered programming. And you know, that in programming, you think hard about a problem, you understand it, you write it down in a very obscure form that we call a program. But then once again, it's in humanity's library, right? And anyone can get the benefit from it. And the scalability is massive. And so I think that the thing that really appeals to me about the digital world is that you can have this insane leverage, right? A single individual with an idea is able to affect the entire planet. And that's something I think is really hard to do if you're moving around physical atoms. But you said mathematics. So if you look at the wet thing over here, our mind, do you ultimately see it as just math, as just information processing? Or is there some other magic as you've seen, if you've seen through biology and chemistry and so on? I think it's really interesting to think about humans as just information processing systems. And that it seems like it's actually a pretty good way of describing a lot of kind of how the world works or a lot of what we're capable of to think that, again, if you just look at technological innovations over time, that in some ways, the most transformative innovation that we've had is has been the computer, right? In some ways, the internet, what has the internet done, right? The internet is not about these physical cables. It's about the fact that I am suddenly able to instantly communicate with any other human on the planet. I'm able to retrieve any piece of knowledge that in some ways the human race has ever had. And that those are these insane transformations. Do you see the, our society as a whole, the collective as another extension of the intelligence of the human being? So if you look at the human being as an information processing system, you mentioned the internet, the networking. Do you see us all together as a civilization as a kind of intelligent system? Yeah, I think this is actually a really interesting perspective to take and to think about that you sort of have this collective intelligence of all of society. The economy itself is this superhuman machine that is optimizing something, right? And it's almost in some ways a company has a will of its own, right? That you have all these individuals who are all pursuing their own individual goals and thinking really hard and thinking about the right things to do. But somehow the company does something that is this emergent thing and that is a really useful abstraction. And so I think that in some ways, we think of ourselves as the most intelligent things on the planet and the most powerful things on the planet. But there are things that are bigger than us that are these systems that we all contribute to. And so I think actually, it's interesting to think about if you've read Isaac Isamov's foundation, right? That there's this concept of psycho history in there, which is effectively this, that if you have trillions or quadrillions of beings, then maybe you could actually predict what that being, that huge macro being will do and almost independent of what the individuals want. And I actually have a second angle on this that I think is interesting, which is thinking about technological determinism. One thing that I actually think a lot about with OpenAI, is that we're kind of coming on onto this insanely transformational technology of general intelligence that will happen at some point. And there's a question of how can you take actions that will actually steer it to go better rather than worse? And that I think one question you need to ask is, as a scientist, as an inventor, as a creator, what impact can you have in general? You look at things like the telephone invented by two people on the same day. Like, what does that mean? Like, what does that mean about the shape of innovation? And I think that what's going on is everyone's building on the shoulders of the same giants. And so you can kind of, you can't really hope to create something no one else ever would. If Einstein wasn't born, someone else would have come up with relativity. You change the timeline a bit, right? That maybe it would have taken another 20 years, but it wouldn't be that fundamentally humanity would never discover these fundamental truths. There's some kind of invisible momentum that some people like Einstein or OpenAI is plugging into, that anybody else can also plug into and ultimately that wave takes us into a certain direction. That's what you mean by determinism. That's right, that's right. And this kind of seems to play out in a bunch of different ways, that there's some exponential that is being written and that the exponential itself, which one it is, changes. Think about Moore's law, an entire industry set, it's clocked to it for 50 years. Like, how can that be? How is that possible? And yet somehow it happened. And so I think you can't hope to ever invent something that no one else will. Maybe you can change the timeline a little bit, but if you really want to make a difference, I think that the thing that you really have to do, the only real degree of freedom you have is to set the initial conditions under which a technology is born. And so you think about the internet, right? That there are lots of other competitors trying to build similar things and the internet won and that the initial conditions were that it was created by this group that really valued people being able to be, anyone being able to plug in this very academic mindset of being open and connected. And I think that the internet for the next 40 years really played out that way. Maybe today, things are starting to shift in a different direction, but I think that those initial conditions were really important to determine the next 40 years worth of progress. That's really beautifully put. So another example of that I think about, I recently looked at it. I looked at Wikipedia, the formation of Wikipedia, and I wondered what the internet would be like if Wikipedia had ads. There's an interesting argument that why they chose not to put advertisement on Wikipedia. I think Wikipedia is one of the greatest resources we have on the internet. It's extremely surprising how well it works and how well it was able to aggregate all this kind of good information. And essentially, the creator of Wikipedia, I don't know, there's probably some debates there, but set the initial conditions and how it carried itself forward. That's really interesting. So the way you're thinking about AGI or artificial intelligence is you're focused on setting the initial conditions for the progress. That's right. That's powerful. Okay, so look into the future. If you create an AGI system, like one that can ace the Turing test, natural language, what do you think would be the interactions you would have with it? What do you think are the questions you would ask? What would be the first question you would ask it, her, him? That's right. I think that at that point, if you've really built a powerful system that is capable of shaping the future of humanity, the first question that you really should ask is how do we make sure that this plays out well? And so that's actually the first question that I would ask a powerful AGI system is. So you wouldn't ask your colleague, you wouldn't ask like Ilya, you would ask the AGI system. Oh, we've already had the conversation with Ilya, right? And everyone here. And so you want as many perspectives and a piece of wisdom as you can for answering this question. So I don't think you necessarily defer to whatever your powerful system tells you, but you use it as one input to try to figure out what to do. But I guess fundamentally, what it really comes down to is if you've built something really powerful and you think about, for example, the creation of shortly after the creation of nuclear weapons, right? The most important question in the world was what's the world order going to be like? How do we set ourselves up in a place where we're going to be able to survive as a species? With AGI, I think the question is slightly different, right? That there is a question of how do we make sure that we don't get the negative effects? But there's also the positive side, right? You imagine that, you know, like what will an AGI be like? Like what will it be capable of? And I think that one of the core reasons that an AGI can be powerful and transformative is actually due to technological development, right? If you have something that's capable as a human and that it's much more scalable, that you absolutely want that thing to go read the whole scientific literature and think about how to create cures for all the diseases, right? You want it to think about how to go and build technologies to help us create material abundance and to figure out societal problems that we have trouble with, like, how are we supposed to clean up the environment? And, you know, maybe you want this to go and invent a bunch of little robots that will go out and be biodegradable and turn ocean debris into harmless molecules. And I think that that positive side is something that I think people miss sometimes when thinking about what an AGI will be like. And so I think that if you have a system that's capable of all of that, you absolutely want its advice about how do I make sure that we're using your capabilities in a positive way for humanity. So what do you think about that psychology that looks at all the different possible trajectories of an AGI system, many of which, perhaps the majority of which are positive, and nevertheless focuses on the negative trajectories? I mean, you get to interact with folks, you get to think about this, maybe within yourself as well. You look at Sam Harris and so on. It seems to be, sorry to put it this way, but almost more fun to think about the negative possibilities. Whatever that's deep in our psychology, what do you think about that? And how do we deal with it? Because we want AI to help us. So I think there's kind of two problems entailed in that question. The first is more of the question of how can you even picture what a world with a new technology will be like? Now imagine we're in 1950, and I'm trying to describe Uber to someone. Apps and the internet. Yeah, I mean, that's going to be extremely complicated, but it's imaginable. It's imaginable, right? But, and now imagine being in 1950 and predicting Uber, right? And you need to describe the internet, you need to describe GPS, you need to describe the fact that everyone's going to have this phone in their pocket. And so I think that just the first truth is that it is hard to picture how a transformative technology will play out in the world. We've seen that before with technologies that are far less transformative than AGI will be. And so I think that one piece is that it's just even hard to imagine and to really put yourself in a world where you can predict what that positive vision would be like. And I think the second thing is that I think it is always easier to support the negative side than the positive side. It's always easier to destroy than create. And less in a physical sense and more just in an intellectual sense, right? Because I think that with creating something you need to just get a bunch of things right and to destroy you just need to get one thing wrong. And so I think that what that means is that I think a lot of people's thinking dead ends as soon as they see the negative story. But that being said, I actually have some hope, right? I think that the positive vision is something that I think can be, is something that we can talk about. And I think that just simply saying this fact of, yeah, there's positive, there's negatives, everyone likes to dwell on the negative. People actually respond well to that message and say, huh, you're right, there's a part of this that we're not talking about, not thinking about. And that's actually something that's, I think, really been a key part of how we think about AGI at OpenAI. You can kind of look at it as like, okay, OpenAI talks about the fact that there are risks and yet they're trying to build this system. How do you square those two facts? So do you share the intuition that some people have, I mean, from Sam Harris to even Elon Musk himself, that it's tricky as you develop AGI to keep it from slipping into the existential threats, into the negative? What's your intuition about how hard is it to keep AI development on the positive track? What's your intuition there? To answer the question, you can really look at how we structure OpenAI. So we really have three main arms. We have capabilities, which is actually doing the technical work and pushing forward what these systems can do. There's safety, which is working on technical mechanisms to ensure that the systems we build are aligned with human values. And then there's policy, which is making sure that we have governance mechanisms, answering that question of, well, whose values? And so I think that the technical safety one is the one that people kind of talk about the most, right? You talk about, like think about, you know, all of the dystopic AI movies, a lot of that is about not having good technical safety in place. And what we've been finding is that, you know, I think that actually a lot of people look at the technical safety problem and think it's just intractable. Right, this question of what do humans want? How am I supposed to write that down? Can I even write down what I want? No way. And then they stop there. But the thing is we've already built systems that are able to learn things that humans can't specify. You know, even the rules for how to recognize if there's a cat or a dog in an image. Turns out it's intractable to write that down. And yet we're able to learn it. And that what we're seeing with systems we build at OpenAI, and they're still in early proof of concept stage, is that you are able to learn human preferences. You're able to learn what humans want from data. And so that's kind of the core focus for our technical safety team. And I think that there actually we've had some pretty encouraging updates in terms of what we've been able to make work. So you have an intuition and a hope that from data, you know, looking at the value alignment problem, from data, we can build systems that align with the collective better angels of our nature. So align with the ethics and the morals of human beings. To even say this in a different way, I mean, think about how do we align humans, right? Think about like a human baby can grow up to be an evil person or a great person. And a lot of that is from learning from data, right? That you have some feedback as a child is growing up, they get to see positive examples. And so I think that just like the only example we have of a general intelligence that is able to learn from data to align with human values and to learn values, I think we shouldn't be surprised that we can do the same sorts of techniques or whether the same sort of techniques end up being how we solve value alignment for AGI's. So let's go even higher. I don't know if you've read the book Sapiens, but there's an idea that, you know, that as a collective, as us human beings, we kind of develop together ideas that we hold. There's no, in that context, objective truth. We just kind of all agree to certain ideas and hold them as a collective. Did you have a sense that there is, in the world of good and evil, do you have a sense that to the first approximation, there are some things that are good and that you could teach systems to behave to be good? So I think that this actually blends into our third team, right, which is the policy team. And this is the one, the aspect I think people really talk about way less than they should, right? Because imagine that we build super powerful systems that we've managed to figure out all the mechanisms for these things to do whatever the operator wants. The most important question becomes who's the operator, what do they want, and how is that going to affect everyone else, right? And I think that this question of what is good, what are those values, I mean, I think you don't even have to go to those very grand existential places to start, to realize how hard this problem is. You just look at different countries and cultures across the world and that there's a very different conception of how the world works and what kinds of ways that society wants to operate. And so I think that the really core question is actually very concrete. And I think it's not a question that we have ready answers to, right? It's how do you have a world where all the different countries that we have, United States, China, Russia, and the hundreds of other countries out there are able to continue to not just operate in the way that they see fit, but in the world that emerges in these, where you have these very powerful systems operating alongside humans ends up being something that empowers humans more, that makes human existence be a more meaningful thing and that people are happier and wealthier and able to live more fulfilling lives. It's not an obvious thing for how to design that world once you have that very powerful system. So if we take a little step back, and we're having like a fascinating conversation and OpenAI is in many ways a tech leader in the world, and yet we're thinking about these big existential questions which is fascinating and really important. I think you're a leader in that space and that's a really important space of just thinking how AI affects society in a big picture view. So Oscar Wilde said, we're all in the gutter, but some of us are looking at the stars and I think OpenAI has a charter that looks to the stars, I would say, to create intelligence, to create general intelligence, make it beneficial, safe, and collaborative. Can you tell me how that came about, how a mission like that and the path to creating a mission like that at OpenAI was founded? Yeah, so I think that in some ways it really boils down to taking a look at the landscape. So if you think about the history of AI, that basically for the past 60 or 70 years, people have thought about this goal of what could happen if you could automate human intellectual labor. Imagine you can build a computer system that could do that, what becomes possible? We have a lot of sci-fi that tells stories of various dystopias and increasingly you have movies like Herd that tell you a little bit about maybe more of a little bit utopic vision. You think about the impacts that we've seen from being able to have bicycles for our minds and computers and I think that the impact of computers and the internet has just far outstripped what anyone really could have predicted. And so I think that it's very clear that if you can build an AGI, it will be the most transformative technology that humans will ever create. And so what it boils down to then is a question of, well, is there a path? Is there hope? Is there a way to build such a system? And I think that for 60 or 70 years, that people got excited and that ended up not being able to deliver on the hopes that people had pinned on them. And I think that then, after two winters of AI development, that people I think kind of almost stopped daring to dream. Really talking about AGI or thinking about AGI became almost this taboo in the community. But I actually think that people took the wrong lesson from AI history. And if you look back, starting in 1959 is when the Perceptron was released. And this is basically one of the earliest neural networks. It was released to what was perceived as this massive overhype. So in the New York Times in 1959, you have this article saying that the Perceptron will one day recognize people, call out their names, instantly translate speech between languages. And people at the time looked at this and said, this is, your system can't do any of that. And basically spent 10 years trying to discredit the whole Perceptron direction and succeeded. And all the funding dried up and people kind of went in other directions. And in the 80s, there was a resurgence. And I'd always heard that the resurgence in the 80s was due to the invention of back propagation and these algorithms that got people excited. But actually the causality was due to people building larger computers. That you can find these articles from the 80s saying that the democratization of computing power suddenly meant that you could run these larger neural networks and then people started to do all these amazing things. The back propagation algorithm was invented. And the neural nets people were running were these tiny little 20 neuron neural nets. What are you supposed to learn with 20 neurons? And so of course, they weren't able to get great results. And it really wasn't until 2012 that this approach, that's almost the most simple natural approach that people have come up with in the 50s. In some ways even in the 40s before there were computers with the Pitz McCollum neuron, suddenly this became the best way of solving problems. I think there are three core properties that deep learning has that I think are very worth paying attention to. The first is generality. We have a very small number of deep learning tools. SGD, deep neural net, maybe some RL. And it solves this huge variety of problems. Speech recognition, machine translation, game playing, all of these problems, small set of tools. So there's the generality. There's a second piece, which is the competence. You wanna solve any of those problems throughout 40 years worth of normal computer vision research replace it with a deep neural net, it's gonna work better. And there's a third piece, which is the scalability. One thing that has been shown time and time again is that if you have a larger neural network, throw more compute, more data at it, it will work better. Those three properties together feel like essential parts of building a general intelligence. Now it doesn't just mean that if we scale up what we have, that we will have an AGI, right? There are clearly missing pieces, there are missing ideas. We need to have answers for reasoning. But I think that the core here is that for the first time, it feels that we have a paradigm that gives us hope that general intelligence can be achievable. And so as soon as you believe that, everything else becomes comes into focus, right? If you imagine that you may be able to, and that the timeline I think remains uncertain, but I think that certainly within our lifetimes and possibly within a much shorter period of time than people would expect, if you can really build the most transformative technology that will ever exist, you stop thinking about yourself so much, right? And you start thinking about just like, how do you have a world where this goes well? And that you need to think about the practicalities of how do you build an organization and get together a bunch of people and resources, and to make sure that people feel motivated and ready to do it. But I think that then you start thinking about, well, what if we succeed? And how do we make sure that when we succeed, that the world is actually the place that we want ourselves to exist in? And almost in the Ralsi and Bale sense of the word. And so that's kind of the broader landscape. And OpenAI was really formed in 2015 with that high level picture of AGI might be possible sooner than people think, and that we need to try to do our best to make sure it's going to go well. And then we spent the next couple of years really trying to figure out what does that mean? How do we do it? And I think that typically with a company, you start out very small, see you in a co-founder, and you build a product, you get some users, you get product market fit. Then at some point you raise some money, you hire people, you scale, and then down the road, then the big companies realize you exist and try to kill you. And for OpenAI, it was basically everything in exactly the opposite order. Ah. Ah. Ah. Let me just pause for a second. You said a lot of things. And let me just admire the jarring aspect of what OpenAI stands for, which is daring to dream. I mean, you said it, it's pretty powerful. It caught me off guard because I think that's very true. The step of just daring to dream about the possibilities of creating intelligence in a positive, in a safe way, but just even creating intelligence is a much needed, refreshing catalyst for the AI community. So that's the starting point. Okay, so then formation of OpenAI. I would just say that when we were starting OpenAI, that kind of the first question that we had is, is it too late to start a lab with a bunch of the best people? Right, is that even possible? That was an actual question. That was really, that was the core question of, we had this dinner in July of 2015, and that was really what we spent the whole time talking about. And, you know, because it's, you think about kind of where AI was, is that it had transitioned from being an academic pursuit to an industrial pursuit. And so a lot of the best people were in these big research labs, and that we wanted to start our own one that no matter how much resources we could accumulate would be pale in comparison to the big tech companies. And we knew that. And there was a question of, are we going to be actually able to get this thing off the ground? You need a critical mass. You can't just do you and a co-founder build a product. Right, you really need to have a group of, you know, five to 10 people. And we kind of concluded it wasn't obviously impossible. So it seemed worth trying. Well, you're also a dreamer, so who knows, right? That's right. Okay, so speaking of that, competing with the big players, let's talk about some of the tricky things as you think through this process of growing, of seeing how you can develop these systems at scale that competes. So you recently formed OpenAI LP, a new cap profit company that now carries the name OpenAI. So OpenAI is now this official company. The original non-profit company still exists and carries the OpenAI non-profit name. So can you explain what this company is, what the purpose of its creation is, and how did you arrive at the decision to create it? OpenAI, the whole entity and OpenAI LP as a vehicle is trying to accomplish the mission of ensuring that artificial general intelligence benefits everyone. And the main way that we're trying to do that is by actually trying to build general intelligence to ourselves and make sure the benefits are distributed to the world. That's the primary way. We're also fine if someone else does this, right? It doesn't have to be us. If someone else is going to build an AGI and make sure that the benefits don't get locked up in one company or with one set of people, we're actually fine with that. And so those ideas are baked into our charter, which is kind of the foundational document that describes kind of our values and how we operate. It was also really baked into the structure of OpenAI LP. And so the way that we've set up OpenAI LP is that in the case where we succeed, right? If we actually build what we're trying to build, then investors are able to get a return. But that return is something that is capped. And so if you think of AGI in terms of the value that you could really create, you're talking about the most transformative technology ever created. It's gonna create orders of magnitude more value than any existing company. And that all of that value will be owned by the world, like legally titled to the nonprofit to fulfill that mission. And so that's the structure. So the mission is a powerful one and it's one that I think most people would agree with. It's how we would hope AI progresses. And so how do you tie yourself to that mission? How do you make sure you do not deviate from that mission? That other incentives that are profit driven don't interfere with the mission. So this was actually a really core question for us for the past couple of years, because I'd say that like the way that our history went was that for the first year, we were getting off the ground, right? We had this high level picture, but we didn't know exactly how we wanted to accomplish it. And really two years ago, it's when we first started realizing in order to build AGI, we're just going to need to raise way more money than we can as a nonprofit. And we're talking many billions of dollars. And so the first question is, how are you supposed to do that and stay true to this mission? And we looked at every legal structure out there and included none of them were quite right for what we wanted to do. And I guess it shouldn't be too surprising if you're going to do some like crazy unprecedented technology that you're going to have to come up with some crazy unprecedented structure to do it in. And a lot of our conversation was with people at OpenAI, right, the people who really joined because they believe so much in this mission and thinking about how do we actually raise the resources to do it and also stay true to what we stand for. And the place you got to start is to really align on what is it that we stand for, right? What are those values? What's really important to us? And so I'd say that we spent about a year really compiling the OpenAI charter and that that determines, and if you even look at the first line item in there, it says that, look, we expect we're going to have to marshal huge amounts of resources, but we're going to make sure that we minimize conflict of interest with the mission. And that kind of aligning on all of those pieces was the most important step towards figuring out how do we structure a company that can actually raise the resources to do what we need to do. I imagine OpenAI, the decision to create OpenAI LP was a really difficult one, and there was a lot of discussions, as you mentioned, for a year, and there was different ideas, perhaps detractors within OpenAI, sort of different paths that you could have taken. What were those concerns? What were the different paths considered? What was that process of making that decision like? Yep, so if you look actually at the OpenAI charter, that there's almost two paths embedded within it. There is, we are primarily trying to build AGI ourselves, but we're also okay if someone else does it. And this is a weird thing for a company. It's really interesting, actually. Yeah. There is an element of competition that you do want to be the one that does it, but at the same time, you're okay if somebody else doesn't. We'll talk about that a little bit, that trade-off, that dance, that's really interesting. And I think this was the core tension as we were designing OpenAI LP, and really the OpenAI strategy, is how do you make sure that both you have a shot at being a primary actor, which really requires building an organization, raising massive resources, and really having the will to go and execute on some really, really hard vision, right? You need to really sign up for a long period to go and take on a lot of pain and a lot of risk. And to do that, normally, you just import the startup mindset, right? And that you think about, okay, how do we out-execute everyone? You have this very competitive angle. But you also have this second angle of saying that, well, the true mission isn't for OpenAI to build AGI. The true mission is for AGI to go well for humanity. And so, how do you take all of those first actions and make sure you don't close the door on outcomes that would actually be positive and fulfill the mission? And so I think it's a very delicate balance, right? And I think that going 100% one direction or the other is clearly not the correct answer. And so I think that even in terms of just how we talk about OpenAI and think about it, there's just like one thing that's always in the back of my mind is to make sure that we're not just saying OpenAI's goal is to build AGI, right, that it's actually much broader than that, right? That first of all, it's not just AGI, it's safe AGI that's very important. But secondly, our goal isn't to be the ones to build it, our goal is to make sure it goes well for the world. And so I think that figuring out how do you balance all of those and to get people to really come to the table and compile a single document that encompasses all of that wasn't trivial. So part of the challenge here is your mission is, I would say, beautiful, empowering, and a beacon of hope for people in the research community and just people thinking about AI. So your decisions are scrutinized more than, I think, a regular profit-driven company. Do you feel the burden of this in the creation of the charter and just in the way you operate? Yes. So why do you lean into the burden by creating such a charter? Why not keep it quiet? I mean, it just boils down to the mission, right? Like I'm here and everyone else is here because we think this is the most important mission. Dare to dream. All right, so do you think you can be good for the world or create an AGI system that's good when you're a for-profit company? From my perspective, I don't understand why profit interferes with positive impact on society. I don't understand why Google, that makes most of its money from ads, can't also do good for the world or other companies, Facebook, anything. I don't understand why those have to interfere. You can, profit isn't the thing in my view that affects the impact of a company. What affects the impact of the company is the charter, is the culture, is the people inside, and profit is the thing that just fuels those people. What are your views there? Yeah, so I think that's a really good question and there's some real long-standing debates in human society that are wrapped up in it. The way that I think about it is just think about what are the most impactful non-profits in the world? What are the most impactful for-profits in the world? Right, it's much easier to list the for-profits. That's right, and I think that there's some real truth here that the system that we set up, the system for kind of how today's world is organized, is one that really allows for huge impact. And that kind of part of that is that you need to be, that for-profits are self-sustaining and able to kind of build on their own momentum. And I think that's a really powerful thing. It's something that when it turns out that we haven't set the guardrails correctly, causes problems, right? Think about logging companies that go into a forest, the rainforest, that's really bad, we don't want that. And it's actually really interesting to me that kind of this question of how do you get positive benefits out of a for-profit company, it's actually very similar to how do you get positive benefits out of an AGI, right? That you have this like very powerful system that's more powerful than any human and is kind of autonomous in some ways, it's superhuman in a lot of axes, and somehow you have to set the guardrails to get good things to happen. But when you do, the benefits are massive. And so I think that when I think about non-profit versus for-profit, I think just not enough happens in non-profits, they're very pure, but it's just kind of, it's just hard to do things there. In for-profits in some ways, like too much happens, but if kind of shaped in the right way, it can actually be very positive. And so with OpenAILP, we're picking a road in between. Now, the thing that I think is really important to recognize is that the way that we think about OpenAILP is that in the world where AGI actually happens, right? In a world where we are successful, we build the most transformative technology ever, the amount of value we're gonna create will be astronomical. And so then in that case, that the cap that we have will be a small fraction of the value we create. And the amount of value that goes back to investors and employees looks pretty similar to what would happen in a pretty successful startup. And that's really the case that we're optimizing for, right? That we're thinking about in the success case, making sure that the value we create doesn't get locked up. And I expect that in other for-profit companies that it's possible to do something like that. I think it's not obvious how to do it, right? And I think that as a for-profit company, you have a lot of fiduciary duty to your shareholders and that there are certain decisions that you just cannot make in our structure. We've set it up so that we have a fiduciary duty to the charter, that we always get to make the decision that is right for the charter, rather than even if it comes at the expense of our own stakeholders. And so I think that when I think about what's really important, it's not really about nonprofit versus for-profit. It's really a question of if you build AGI and you kind of, you know, humanity is now in this new age, who benefits, whose lives are better? And I think that what's really important is to have an answer that is everyone. Yeah, which is one of the core aspects of the charter. So one concern people have, not just with OpenAI, but with Google, Facebook, Amazon, anybody really that's creating impact at scale, is how do we avoid, as your charter says, avoid enabling the use of AI or AGI to unduly concentrate power? Why would not a company like OpenAI keep all the power of an AGI system to itself? The charter. The charter, so how does the charter actualize itself in day to day? So I think that the first to zoom out, right, that the way that we structure the company is so that the power for sort of dictating the actions that OpenAI takes ultimately rests with the board, right? The board of the nonprofit. And the board is set up in certain ways, with certain restrictions that you can read about in the OpenAI LP blog post. But effectively the board is the governing body for OpenAI LP. And the board has a duty to fulfill the mission of the nonprofit. And so that's kind of how we tie, how we thread all these things together. Now there's a question of, so day to day, how do people, the individuals, who in some ways are the most empowered ones, right? The board sort of gets to call the shots at the high level, but the people who are actually executing are the employees, right? The people here on a day to day basis who have the keys to the technical kingdom. And there I think that the answer looks a lot like, well, how does any company's values get actualized, right? And I think that a lot of that comes down to that you need people who are here because they really believe in that mission, and they believe in the charter, and that they are willing to take actions that maybe are worse for them, but are better for the charter. And that's something that's really baked into the culture. And honestly, I think it's, I think that that's one of the things that we really have to work to preserve as time goes on. And that's a really important part of how we think about hiring people and bringing people into OpenAI. So there's people here, there's people here who could speak up and say, like, hold on a second, this is totally against what we stand for, culture-wise. Yeah, yeah, for sure. I mean, I think that we actually have, I think that's like a pretty important part of how we operate and how we have, even again, with designing the charter and designing OpenAILP in the first place, that there has been a lot of conversation with employees here, and a lot of times where employees said, wait a second, this seems like it's going in the wrong direction, and let's talk about it. And so I think one thing that's, I think, a really, and here's actually one thing that I think is very unique about us as a small company, is that if you're at a massive tech giant, it's a little bit hard for someone who's a line employee to go and talk to the CEO and say, I think that we're doing this wrong. And you look at companies like Google that have had some collective action from employees to make ethical change around things like Maven. And so maybe there are mechanisms that other companies that work, but here, super easy for anyone to pull me aside, to pull Sam aside, to pull Ilya aside, and people do it all the time. One of the interesting things in the charter is this idea that it'd be great if you could try to describe or untangle switching from competition to collaboration and late stage AGI development. It was really interesting, this dance between competition and collaboration. How do you think about that? Yeah, assuming that you can actually do the technical side of AGI development, I think there's going to be two key problems with figuring out how do you actually deploy it, make it go well. The first one of these is the run up to building the first AGI. You look at how self-driving cars are being developed, and it's a competitive race. And the thing that always happens in competitive race is that you have huge amounts of pressure to get rid of safety. And so that's one thing we're very concerned about, right? Is that people, multiple teams figuring out we can actually get there, but if we took the slower path that is more guaranteed to be safe, we will lose. And so we're going to take the fast path. And so the more that we can both ourselves be in a position where we don't generate that competitive race, where we say if the race is being run and that someone else is further ahead than we are, we're not going to try to leapfrog, we're going to actually work with them, right? We will help them succeed as long as what they're trying to do is to fulfill our mission, then we're good. We don't have to build AGI ourselves. And I think that's a really important commitment from us, but it can't just be unilateral, right? I think that it's really important that other players who are serious about building AGI make similar commitments, right? And I think that, you know, again, to the extent that everyone believes that AGI should be something to benefit everyone, then it actually really shouldn't matter which company builds it. And we should all be concerned about the case where we just race so hard to get there that something goes wrong. So what role do you think government, our favorite entity, has in setting policy and rules about this domain, from research to the development to early stage to late stage AI and AGI development? So I think that, first of all, it's really important that government's in there, right? In some way, shape, or form. You know, at the end of the day, we're talking about building technology that will shape how the world operates, and that there needs to be government as part of that answer. And so that's why we've done a number of different congressional testimonies. We interact with a number of different lawmakers, and that, you know, right now, a lot of our message to them is that it's not the time for regulation, it is the time for measurement, right? That our main policy recommendation is that people, and, you know, the government does this all the time with bodies like NIST, spend time trying to figure out just where the technology is, how fast it's moving, and can really become literate and up to speed with respect to what to expect. So I think that today, the answer really is about measurement, and I think that there will be a time and place where that will change. And I think it's a little bit hard to predict exactly what exactly that trajectory should look like. So there will be a point at which regulation, federal and the United States, the government steps in and helps be the, I don't want to say the adult in the room, to make sure that there is strict rules, maybe conservative rules that nobody can cross. Well, I think there's kind of maybe two angles to it. So today, with narrow AI applications, that I think there are already existing bodies that are responsible and should be responsible for regulation. You think about, for example, with self-driving cars, that you want the National Highway, exactly to be regulating that. That makes sense, right? That basically what we're saying is that we're going to have these technological systems that are going to be performing applications that humans already do. Great, we already have ways of thinking about standards and safety for those. So I think actually empowering those regulators today is also pretty important. And then I think for AGI, that there's going to be a point where we'll have better answers. And I think that maybe a similar approach of first measurement and start thinking about what the rules should be. I think it's really important that we don't prematurely squash progress. I think it's very easy to smother a budding field. And I think that's something to really avoid. But I don't think it's the right way of doing it is to say, let's just try to blaze ahead and not involve all these other stakeholders. So you've recently released a paper on GPT-2 language modeling, but did not release the full model because you had concerns about the possible negative effects of the availability of such model. It's outside of just that decision, it's super interesting because of the discussion at a societal level, the discourse it creates. So it's fascinating in that aspect. But if you think at the specifics here at first, what are some negative effects that you envisioned? And of course, what are some of the positive effects? Yeah, so again, I think to zoom out, the way that we thought about GPT-2 is that with language modeling, we are clearly on a trajectory right now where we scale up our models and we get qualitatively better performance. GPT-2 itself was actually just a scale up of a model that we've released in the previous June, right? And we just ran it at much larger scale and we got these results where suddenly starting to write coherent prose, which was not something we'd seen previously. And what are we doing now? Well, we're gonna scale up GPT-2 by 10x, by 100x, by 1000x, and we don't know what we're gonna get. And so it's very clear that the model that we released last June, I think it's kind of like, it's a good academic toy. It's not something that we think is something that can really have negative applications or to the extent that it can, that the positive of people being able to play with it is far outweighs the possible harms. You fast forward to not GPT-2, but GPT-20, and you think about what that's gonna be like. And I think that the capabilities are going to be substantive. And so there needs to be a point in between the two where you say, this is something where we are drawing the line and that we need to start thinking about the safety aspects. And I think for GPT-2, we could have gone either way. And in fact, when we had conversations internally, that we had a bunch of pros and cons, and it wasn't clear which one outweighed the other. And I think that when we announced that, hey, we decide not to release this model, then there was a bunch of conversation where various people said, it's so obvious that you should have just released it. There are other people said, it's so obvious you should not have released it. And I think that that almost definitionally means that holding it back was the correct decision. If it's not obvious whether something is beneficial or not, you should probably default to caution. And so I think that the overall landscape for how we think about it is that this decision could have gone either way. There are great arguments in both directions, but for future models down the road and possibly sooner than you'd expect because scaling these things up doesn't actually take that long, those ones you're definitely not going to want to release into the wild. And so I think that we almost view this as a test case and to see, can we even design, how do you have a society, how do you have a system that goes from having no concept of responsible disclosure where the mere idea of not releasing something for safety reasons is unfamiliar to a world where you say, okay, we have a powerful model, let's at least think about it, let's go through some process. And you think about the security community, it took them a long time to design responsible disclosure. You think about this question of, well, I have a security exploit, I send it to the company, the company is like tries to prosecute me or just ignores it, what do I do? And so the alternatives of, oh, I just always publish your exploits, that doesn't seem good either. And so it really took a long time and it was bigger than any individual, it's really about building a whole community that believe that, okay, we'll have this process where you send it to the company, if they don't act in a certain time, then you can go public and you're not a bad person, you've done the right thing. And I think that in AI, part of the response at GPT-2 just proves that we don't have any concept of this. So that's the high level picture. And so I think that, I think this was a really important move to make and we could have maybe delayed it for GPT-3, but I'm really glad we did it for GPT-2. And so now you look at GPT-2 itself and you think about the substance of, okay, what are potential negative applications? So you have this model that's been trained on the internet, which it's also going to be a bunch of very biased data, a bunch of very offensive content in there. And you can ask it to generate content for you on basically any topic, right? You just give it a prompt and it'll just start writing and it writes content like you see on the internet, even down to like saying advertisement in the middle of some of its generations. And you think about the possibilities for generating fake news or abusive content. And it's interesting seeing what people have done with, we released a smaller version of GPT-2 and the people have done things like try to generate, take my own Facebook message history and generate more Facebook messages like me and people generating fake politician content or there's a bunch of things there where you at least have to think, is this going to be good for the world? There's the flip side, which is I think that there's a lot of awesome applications that we really want to see, like creative applications in terms of, if you have sci-fi authors that can work with this tool and come with cool ideas, like that seems awesome if we can write better sci-fi through the use of these tools. And we've actually had a bunch of people write into us asking, hey, can we use it for, a variety of different creative applications? So the positive are actually pretty easy to imagine. The usual NLP applications are really interesting, but let's go there. It's kind of interesting to think about a world where, look at Twitter, where not just fake news, but smarter and smarter bots being able to spread in an interesting complex networking way information that just floods out us regular human beings with our original thoughts. So what are your views of this world with GPT-20, right? How do we think about it? Again, it's like one of those things about in the 50s trying to describe the internet or the smartphone. What do you think about that world, the nature of information? One possibility is that we'll always try to design systems that identify robot versus human, and we'll do so successfully, and so we'll authenticate that we're still human. And the other world is that we just accept the fact that we're swimming in a sea of fake news and just learn to swim there. Well, have you ever seen the, there's a popular meme of a robot with a physical arm and pen clicking the I'm not a robot button? Yeah. I think that the truth is that really trying to distinguish between robot and human is a losing battle. Ultimately, you think it's a losing battle. I think it's a losing battle ultimately, right? I think that that is that in terms of the content, in terms of the actions that you can take, I mean, think about how captures have gone, right? The captures used to be a very nice, simple, you just have this image, all of our OCR is terrible, you put a couple of artifacts in it, humans are gonna be able to tell what it is, an AI system wouldn't be able to. Today, I can barely do captures. And I think that this is just kind of where we're going. I think captures were a moment in time thing, and as AI systems become more powerful, that there being human capabilities that can be measured in a very easy automated way that AIs will not be capable of, I think that's just like, it's just an increasingly hard technical battle. But it's not that all hope is lost, right? You think about how do we already authenticate ourselves, right? We have systems, we have social security numbers, if you're in the US or you have ways of identifying individual people and having real world identity tied to digital identity seems like a step towards authenticating the source of content rather than the content itself. Now, there are problems with that. How can you have privacy and anonymity in a world where the only content you can really trust is, or the only way you can trust content is by looking at where it comes from. And so I think that building out good reputation networks, maybe one possible solution. But yeah, I think that this question is not an obvious one. And I think that we, maybe sooner than we think we'll be in a world where today, I often will read a tweet and be like, do I feel like a real human wrote this? Or do I feel like this is like genuine? Do I feel like I can kind of judge the content a little bit? And I think in the future, it just won't be the case. You will get, for example, the FCC comments on net neutrality. It came out later that millions of those were auto-generated and that the researchers were able to do various statistical techniques to do that. What do you do in a world where those statistical techniques don't exist? It's just impossible to tell the difference between humans and AIs. And in fact, the most persuasive arguments are written by AI. All that stuff, it's not sci-fi anymore. You look at GPT-2 making a great argument for why recycling is bad for the world. You gotta read that and be like, huh, you're right. We are addressing those symptoms. Yeah, that's quite interesting. I mean, ultimately it boils down to the physical world being the last frontier of proving, so you said like basically networks of people, humans vouching for humans in the physical world. And somehow the authentication ends there. I mean, if I had to ask you, I mean, you're way too eloquent for a human. So if I had to ask you to authenticate, like prove how do I know you're not a robot and how do you know I'm not a robot? I think that's so far where in this space, this conversation we just had, the physical movements we did is the biggest gap between us and AI systems is the physical manipulation. So maybe that's the last frontier. Well, here's another question is why is solving this problem important? What aspects are really important to us? I think that probably where we'll end up is we'll hone in on what do we really want out of knowing if we're talking to a human. And I think that again, this comes down to identity. And so I think that the internet of the future, I expect to be one that will have lots of agents out there that will interact with you. But I think that the question of is this real flesh and blood human or is this an automated system may actually just be less important. Let's actually go there. It's GPT-2 is impressive and let's look at GPT-20. Why is it so bad that all my friends are GPT-20? Why is it so important on the internet, do you think, to interact with only human beings? Why can't we live in a world where ideas can come from models trained on human data? Yeah, I think this is actually a really interesting question. This comes back to the how do you even picture a world with some new technology? And I think that one thing that I think is important is let's say honesty. And I think that if you have almost in the Turing test style sense of technology, you have AIs that are pretending to be humans and deceiving you. I think that feels like a bad thing. I think that it's really important that we feel like we're in control of our environment, that we understand who we're interacting with and if it's an AI or a human, that that's not something that we're being deceived about. But I think that the flip side of can I have as meaningful of an interaction with an AI as I can with a human? Well, I actually think here you can turn to Sci-Fi and her I think is a great example of asking this very question. One thing I really love about her is it really starts out almost by asking how meaningful are human virtual relationships, right? And then you have a human who has a relationship with an AI and that you really start to be drawn into that, right? And that all of your emotional buttons get triggered in the same way as if there was a real human that was on the other side of that phone. And so I think that this is one way of thinking about it is that I think that we can have meaningful interactions and that if there's a funny joke, some sense it doesn't really matter if it was written by a human or an AI, but what you don't want in a way where I think we should really draw hard lines is deception. And I think that as long as we're in a world where, why do we build AI systems at all, right? The reason we want to build them is to enhance human lives, to make humans be able to do more things, to have humans feel more fulfilled. And if we can build AI systems that do that, sign me up. So the process of language modeling, how far do you think it take us? Let's look at movie Her. Do you think dialogue, natural language conversation is formulated by the Turing test, for example, do you think that process could be achieved through this kind of unsupervised language modeling? So I think the Turing test in its real form isn't just about language, right? It's really about reasoning too, right? To really pass the Turing test, I should be able to teach calculus to whoever's on the other side and have it really understand calculus and be able to go and solve new calculus problems. And so I think that to really solve the Turing test, we need more than what we're seeing with language models. We need some way of plugging in reasoning. Now, how different will that be from what we already do? That's an open question, right? It might be that we need some sequence of totally radical new ideas, or it might be that we just need to kind of shape our existing systems in a slightly different way. But I think that in terms of how far language modeling will go, it's already gone way further than many people would have expected, right? I think that things like, and I think there's a lot of really interesting angles to poke in terms of how much does GPT-2 understand physical world? Like, you know, you read a little bit about fire underwater in GPT-2, so it's like, okay, maybe it doesn't quite understand what these things are. But at the same time, I think that you also see various things like smoke coming from flame and a bunch of these things that GPT-2, it has no body, it has no physical experience. It's just statically read data. And I think that the answer is like, we don't know yet. And these questions, though, we're starting to be able to actually ask them to physical systems, to real systems that exist, and that's very exciting. Do you think, what's your intuition? Do you think if you just scale language modeling, like significantly scale, that reasoning can emerge from the same exact mechanisms? I think it's unlikely that if we just scale GPT-2 that we'll have reasoning in the full-fledged way. And I think that there's like, you know, the type signature's a little bit wrong, right? That like, there's something we do with, that we call thinking, right? Where we spend a lot of compute, like a variable amount of compute, to get to better answers, right? I think a little bit harder, I get a better answer. And that that kind of type signature isn't quite encoded in a GPT, right? GPT will kind of like, it's spent a long time, and it's like evolutionary history, baking in all this information, getting very, very good at this predictive process. And then at runtime, I just kind of do one forward pass, and I'm able to generate stuff. And so, you know, there might be small tweaks to what we do in order to get the type signature right. For example, well, you know, it's not really one forward pass, right? You know, you generate symbol by symbol, and so maybe you generate like a whole sequence of thoughts, and you only keep like the last bit or something. But I think that at the very least, I would expect you have to make changes like that. Yeah, just exactly how we, you said, think, is the process of generating thought by thought in the same kind of way, like you said, keep the last bit, the thing that we converge towards. Yep. And I think there's another piece, which is interesting, which is this out of distribution generalization, right? That like thinking somehow lets us do that, right? That we haven't experienced a thing, and yet somehow we just kind of keep refining our mental model of it. This is again, something that feels tied to whatever reasoning is. And maybe it's a small tweak to what we do, maybe it's many ideas and we'll take as many decades. Yeah, so the assumption there, generalization out of distribution is that it's possible to create new ideas. It's possible that nobody's ever created any new ideas, and then with scaling GPT-2 to GPT-20, you would essentially generalize to all possible thoughts that us humans can have. Just to play devil's advocate. I mean, how many new story ideas have we come up with since Shakespeare, right? Yeah, exactly. It's just all different forms of love and drama and so on. Okay, not sure if you read Biddle Lesson, a recent blog post by Ray Sutton. Yep, I have. He basically says something that echoes some of the ideas that you've been talking about, which is he says the biggest lesson that can be read from so many years of AI research is that general methods that leverage computation are ultimately going to ultimately win out. Do you agree with this? So basically, open AI in general, but the ideas you're exploring, well, coming up with methods, whether it's GPT-2 modeling or whether it's open AI-5 playing Dota, or a general method is better than a more fine-tuned, expert-tuned method. Yeah, so I think that, well, one thing that I think was really interesting about the reaction to that blog post was that a lot of people have read this as saying that compute is all that matters. And that's a very threatening idea, right? And I don't think it's a true idea either, right? It's very clear that we have algorithmic ideas that have been very important for making progress and to really build AGI, you wanna push as far as you can on the computational scale, and you wanna push as far as you can on human ingenuity. And so I think you need both. But I think the way that you phrase the question is actually very good, right? That it's really about what kind of ideas should we be striving for? And absolutely, if you can find a scalable idea, you pour more data into it, it gets better. Like, that's the real holy grail. And so I think that the answer to the question, I think, is yes. That that's really how we think about it, and that part of why we're excited about the power of deep learning, the potential for building AGI, is because we look at the systems that exist in the most successful AI systems, and we realize that you scale those up, they're gonna work better. And I think that that scalability is something that really gives us hope for being able to build transformative systems. So I'll tell you, this is partially an emotional, you know, a thing that, a response that people often have is compute is so important for state of the art performance. You know, individual developers, maybe a 13 year old sitting somewhere in Kansas or something like that, you know, they're sitting, they might not even have a GPU, or may have a single GPU, a 1080 or something like that. And there's this feeling like, well, how can I possibly compete or contribute to this world of AI if scale is so important? So if you can comment on that, and in general, do you think we need to also in the future focus on democratizing compute resources more, or as much as we democratize the algorithms? Well, so the way that I think about it is that there's this space of possible progress, right? There's a space of ideas and sort of systems that will work, that will move us forward. And there's a portion of that space, and to some extent, an increasingly significant portion of that space that does just require massive compute resources. And for that, that I think that the answer is kind of clear, and that part of why we have the structure that we do is because we think it's really important to be pushing the scale and to be building these large clusters and systems. But there's another portion of the space that isn't about the large scale compute, that are these ideas that, and again, I think that for the ideas to really be impactful and really shine, that they should be ideas that if you scale them up, would work way better than they do at small scale. But that you can discover them without massive computational resources. And if you look at the history of recent developments, you think about things like the GAN or the VAE, that these are ones that I think you could come up with them without having, and in practice, people did come up with them without having massive, massive computational resources. Right, I just talked to Ian Goodfellow, but the thing is, the initial GAN produced pretty terrible results, right? So only because it was in a very specific, only because they're smart enough to know that this is quite surprising and generate anything that they know. Do you see a world, or is that too optimistic and dreamer-like to imagine that the compute resources are something that's owned by governments and provided as a utility? Actually, to some extent, this question reminds me of a blog post from one of my former professors at Harvard, this guy, Matt Welsh, who was a systems professor. I remember sitting in his tenure talk, right, and he had literally just gotten tenure. He went to Google for the summer and then decided he wasn't going back to academia, right? And that kind of in his blog post, he makes this point that, look, as a systems researcher, that I come up with these cool system ideas, right, and I kind of build a little proof of concept, and the best thing I can hope for is that the people at Google or Yahoo, which was around at the time, will implement it and actually make it work at scale, right? That's like the dream for me, right? I built the little thing and they turned it into the big thing that's actually working. And for him, he said, I'm done with that. I want to be the person who's actually doing building and deploying, and I think that there's a similar dichotomy here, right? I think that there are people who really actually find value, and I think it is a valuable thing to do, to be the person who produces those ideas, right, who builds the proof of concept. And yeah, you don't get to generate the coolest possible GAN images, but you invented the GAN, right? And so there's a real trade-off there, and I think that's a very personal choice, but I think there's value in both sides. So do you think creating AGI, something or some new models, we would see echoes of the brilliance, even at the prototype level? So you would be able to develop those ideas without scale, the initial seeds? So take a look at, you know, I always like to look at examples that exist, right? Look at real precedent. And so take a look at the June 2018 model that we released, that we scaled up to turn into GPT-2. And you can see that at small scale, it set some records, right? This was the original GPT. We actually had some cool generations. They weren't nearly as amazing and really stunning as the GPT-2 ones, but it was promising, it was interesting. And so I think it is the case that with a lot of these ideas, that you see promise at small scale. But there is an asterisk here, a very big asterisk, which is sometimes we see behaviors that emerge that are qualitatively different from anything we saw at small scale. And that the original inventor of whatever algorithm looks at and says, I didn't think it could do that. This is what we saw in Dota, right? So PPO was created by John Schulman, who's a researcher here. And with Dota, we basically just ran PPO at massive, massive scale. And there's some tweaks in order to make it work, but fundamentally it's PPO at the core. And we were able to get this long-term planning, these behaviors to really play out on a time scale that we just thought was not possible. And John looked at that and was like, I didn't think it could do that. That's what happens when you're at three orders of magnitude more scale than you tested at. Yeah, but it still has the same flavors of, at least echoes of the expected billions. Although I suspect with GPT, it's scaled more and more. You might get surprising things. So yeah, you're right. It's interesting. It's difficult to see how far an idea will go when it's scaled. It's an open question. Well, so to that point with Dota and PPO, I mean, here's a very concrete one, right? It's actually, one thing that's very surprising about Dota that I think people don't really pay that much attention to is the degree of generalization out of distribution that happens, right? That you have this AI that's trained against other bots for its entirety, the entirety of its existence. Sorry to take a step back. Can you talk through a story of Dota, a story of leading up to opening AI-5 and that past, and what was the process of self-play and so on of training on that? Yeah, yeah, yeah. So with Dota. What is Dota? Yeah, Dota is a complex video game. And we started training, we started trying to solve Dota because we felt like this was a step towards the real world relative to other games like Chess or Go, right? Those very cerebral games where you just kind of have this board of very discreet moves. Dota starts to be much more continuous time that you have this huge variety of different actions, that you have a 45 minute game with all these different units and it's got a lot of messiness to it that really hasn't been captured by previous games. And famously, all of the hard-coded bots for Dota were terrible, right? Just impossible to write anything good for it because it's so complex. And so this seemed like a really good place to push what's the state of the art in reinforcement learning. And so we started by focusing on the one versus one version of the game and we're able to solve that. We're able to beat the world champions and the learning, the skill curve was this crazy exponential, right? And it was like constantly we were just scaling up that we were fixing bugs and that you look at the skill curve and it was really a very, very smooth one. This is actually really interesting to see how that human iteration loop yielded very steady exponential progress. And to one side note, first of all, it's an exceptionally popular video game. The side effect is that there's a lot of incredible human experts at that video game. So the benchmark that you're trying to reach is very high. And the other, can you talk about the approach that was used initially and throughout training these agents to play this game? Yep, and so the approach that we used is self-play. And so you have two agents that don't know anything. They battle each other. They discover something a little bit good and now they both know it. And they just get better and better and better without bound. And that's a really powerful idea, right? That we then went from the one versus one version of the game and scaled up to five versus five, right? So you think about kind of like with basketball where you have this like team sport and you need to do all this coordination and we were able to push the same idea, the same self-play to really get to the professional level at the full five versus five version of the game. And the things I think are really interesting here is that these agents, in some ways, they're almost like an insect-like intelligence, right? Where they have a lot in common with how an insect is trained, right? Insect kind of lives in this environment for a very long time, or the ancestors of this insect have been around for a long time and had a lot of experience that gets baked into this agent. And it's not really smart in the sense of a human, right? It's not able to go and learn calculus, but it's able to navigate its environment extremely well. It's able to handle unexpected things in the environment that it's never seen before pretty well. And we see the same sort of thing with our Dota bots, right, that they're able to, within this game, they're able to play against humans, which is something that never existed in its evolutionary environment, totally different play styles from humans versus the bots. And yet it's able to handle it extremely well. And that's something that I think was very surprising to us, was something that doesn't really emerge from what we've seen with PPO at smaller scale, right? And the kind of scale we're running this stuff at was, I could say like 100,000 CPU cores, running with like hundreds of GPUs, it was probably about something like hundreds of years of experience going into this bot every single real day. And so that scale is massive, and we start to see very different kinds of behaviors out of the algorithms that we all know and love. And Dota, you mentioned, beat the world expert 1v1. And then you weren't able to win 5v5 this year at the best players in the world. So what's the comeback story? First of all, talk through that, that was an exceptionally exciting event. And what's the following months in this year look like? Yeah, yeah, so well, one thing that's interesting is that we lose all the time. Because we play- Who's we here? The Dota team at OpenAI, we play the bot against better players than our system all the time, or at least we used to, right? Like, the first time we lost publicly was we went up on stage at the International, and we played against some of the best teams in the world, and we ended up losing both games, but we gave them a run for their money, right? That both games were kind of 30 minutes, 25 minutes, and they went back and forth, back and forth, back and forth. And so I think that really shows that we're at the professional level, and that kind of looking at those games, we think that the coin could have gone a different direction, and we could have had some wins, and so that was actually very encouraging for us. And you know, it's interesting because the International was at a fixed time, right? So we knew exactly what day we were going to be playing, and we pushed as far as we could, as fast as we could. Two weeks later, we had a bot that had an 80% win rate versus the one that played at TI. So the march of progress, you know, you should think of as a snapshot rather than as an end state. And so in fact, we'll be announcing our finals pretty soon. I actually think that we'll announce our final match prior to this podcast being released. So there should be, we'll be playing against the world champions. And you know, for us, it's really less about, like the way that we think about what's upcoming is the final milestone, the final competitive milestone for the project, right? That our goal in all of this isn't really about beating humans at Dota. Our goal is to push the state of the art in reinforcement learning, and we've done that, right? And we've actually learned a lot from our system and that we have, you know, I think a lot of exciting next steps that we wanna take. And so, you know, kind of as a final showcase of what we built, we're going to do this match. But for us, it's not really the success or failure to see, you know, do we have the coin flip go in our direction or against. Where do you see the field of deep learning heading in the next few years? Where do you see the work and reinforcement learning perhaps heading, and more specifically with OpenAI, all the exciting projects that you're working on, what does 2019 hold for you? Massive scale. Scale. I will put an atris on that and just say, you know, I think that it's about ideas plus scale. You need both. So that's a really good point. So the question in terms of ideas, you have a lot of projects that are exploring different areas of intelligence. And the question is, when you think of scale, do you think about growing the scale of those individual projects, or do you think about adding new projects? And sorry, if you were thinking about adding new projects, or if you look at the past, what's the process of coming up with new projects and new ideas? Yep. So we really have a life cycle of project here. So we start with a few people just working on a small scale idea, and language is actually a very good example of this. There was really, you know, one person here who was pushing on language for a long time. I mean, then you get signs of life, right? And so this is like, let's say, you know, with the original GPT, we had something that was interesting. And we said, okay, it's time to scale this, right? It's time to put more people on it, put more computational resources behind it. And then we just kind of keep pushing and keep pushing. And the end state is something that looks like Dota or robotics, where you have a large team of, you know, 10 or 15 people that are running things at very large scale, and that you're able to really have material engineering and, you know, sort of machine learning science coming together to make systems that work and get material results that just would have been impossible otherwise. So we do that whole life cycle. We've done it a number of times, you know, typically end to end, it's probably two years or so to do it. I know the organization has been around for three years, so maybe we'll find that we also have longer life cycle projects, but, you know, we will work up to those. We have, so one team that we were actually just starting, Ilya and I are kicking off a new team called the Reasoning Team, and that this is to really try to tackle how do you get neural networks to reason. And we think that this will be a long term project and it's one that we're very excited about. In terms of reasoning, super exciting topic, what kind of benchmarks, what kind of tests of reasoning do you envision? What would, if you set back with whatever drink and you would be impressed that this system is able to do something, what would that look like? Theorem proving. Theorem proving. So some kind of logic and especially mathematical logic. I think so, right? And I think that there's kind of other problems that are dual to theorem proving in particular. You know, you think about programming, you think about even like security analysis of code, that these all kind of capture the same sorts of core reasoning and being able to do some out of distribution generalization. It would be quite exciting if OpenAI reasoning team was able to prove that P equals NP. That would be very nice. It would be very, very, very exciting, especially if it turns out that P equals NP, that'll be interesting too. Hey, but it would be ironic and humorous. So what problem stands out to you as the most exciting and challenging impact for the work for us as a community in general and for OpenAI this year? You mentioned reasoning. I think that's a heck of a problem. Yeah, so I think reasoning is an important one. I think it's gonna be hard to get good results in 2019. You know, again, just like we think about the life cycle, takes time. I think for 2019, language modeling seems to be kind of on that ramp, right? It's at the point that we have a technique that works. We wanna scale 100X, 1000X, see what happens. Awesome. Do you think we're living in a simulation? I think it's hard to have a real opinion about it. You know, it's actually interesting. I separate out things that I think can have like, you know, yield materially different predictions about the world from ones that are just kind of, you know, fun to speculate about. And I kind of view simulation as more like, is there a flying teapot between Mars and Jupiter? Like, maybe, but it's a little bit hard to know what that would mean for my life. So there is something actionable. So some of the best work OpenAI has done is in the field of reinforcement learning. And some of the success of reinforcement learning come from being able to simulate the problem you're trying to solve. So do you have a hope for reinforcement, for the future of reinforcement learning and for the future of simulation? Like whether we're talking about autonomous vehicles or any kind of system, do you see that scaling to be able to simulate systems and hence, be able to create a simulator that echoes our real world and proving once and for all, even though you're denying it, that we're living in a simulation? I feel like there's two separate questions, right? So, you know, kind of at the core there of like, can we use simulation for self-driving cars? Take a look at our robotic system, Dactyl, right? That was trained in simulation using the Dota system, in fact, and it transfers to a physical robot. And I think everyone looks at our Dota system, they're like, okay, it's just a game. How are you ever gonna escape to the real world? And the answer is, well, we did it with the physical robot that no one could program. And so I think the answer is simulation goes a lot further than you think if you apply the right techniques to it. Now there's a question of, you know, are the beings in that simulation gonna wake up and have consciousness? I think that one seems a lot harder to, again, reason about. I think that, you know, you really should think about like, where exactly does human consciousness come from in our own self-awareness? And, you know, is it just that like, once you have like a complicated enough neural net, do you have to worry about the agents feeling pain? And I think there's like interesting speculation to do there, but, you know, again, I think it's a little bit hard to know for sure. Well, let me just keep with the speculation. Do you think to create intelligence, general intelligence, you need one, consciousness, and two, a body? Do you think any of those elements are needed, or is intelligence something that's orthogonal to those? I'll stick to the kind of like, the non-grand answer first, right? So the non-grand answer is just to look at, you know, what are we already making work? You'll get GPT-2, a lot of people would have said that to even get these kinds of results, you need real-world experience, you need a body, you need grounding. How are you supposed to reason about any of these things? How are you supposed to like even kind of know about smoke and fire and those things if you've never experienced them? And GPT-2 shows that you can actually go way further than that kind of reasoning would predict. So I think that in terms of, do we need consciousness? Do we need a body? It seems the answer is probably not, right? That we could probably just continue to push kind of the systems we have. They already feel general. They're not as competent or as general or able to learn as quickly as an AGI would, but you know, they're at least like kind of proto-AGI in some way and they don't need any of those things. Now, let's move to the grand answer, which is, you know, if our neural nets conscious already, would we ever know, how can we tell, right? And you know, here's where the speculation starts to become, you know, at least interesting or fun and maybe a little bit disturbing, depending on where you take it. But it certainly seems that when we think about animals, that there's some continuum of consciousness. You know, my cat, I think is conscious in some way, right? You know, not as conscious as a human. And you could imagine that you could build a little consciousness meter, right? You point at a cat, it gives you a little reading. Point at a human, it gives you much bigger reading. Well, what would happen if you pointed one of those at a donor neural net? And if you're training in this massive simulation, do the neural nets feel pain? You know, it becomes pretty hard to know that the answer is no. And it becomes pretty hard to really think about what that would mean if the answer were yes. And it's very possible, you know, for example, you could imagine that maybe the reason that humans have consciousness is because it's a convenient computational shortcut, right? If you think about it, if you have a being that wants to avoid pain, which seems pretty important to survive in this environment and wants to like, you know, eat food, then that maybe the best way of doing it is to have a being that's conscious, right? That, you know, in order to succeed in the environment, you need to have those properties, and how are you supposed to implement them? And maybe this consciousness's way of doing that. If that's true, then actually maybe we should expect that really competent reinforcement learning agents will also have consciousness. But you know, it's a big if, and I think there are a lot of other arguments they can make in other directions. I think that's a really interesting idea that even GPT-2 has some degree of consciousness. That's something, it's actually not as crazy to think about, it's useful to think about as we think about what it means to create intelligence of a dog, intelligence of a cat, and the intelligence of a human. So last question, do you think we will ever fall in love, like in the movie Her, with an artificial intelligence system or an artificial intelligence system falling in love with a human? I hope so. If there's any better way to end it, is on love, so Greg, thanks so much for talking today. Thank you for having me.
 The following is a conversation with Greg Brockman.
 He's the co-founder and CTO of OpenAI,
 a world-class research organization
 developing ideas in AI with the goal
 of eventually creating a safe and friendly
 artificial general intelligence,
 one that benefits and empowers humanity.
 OpenAI is not only a source of publications,
 algorithms, tools, and data sets.
 Their mission is a catalyst for an important public discourse
 about our future with both narrow
 and general intelligence systems.
 This conversation is part of the
 Artificial Intelligence podcast at MIT and beyond.
 If you enjoy it, subscribe on YouTube, iTunes,
 or simply connect with me on Twitter at Lex Friedman,
 spelled F-R-I-D.
 And now, here's my conversation with Greg Brockman.
 So in high school and right after you wrote a draft
 of a chemistry textbook, I saw that.
 That covers everything from basic structure of the atom
 to quantum mechanics.
 So it's clear you have an intuition and a passion
 for both the physical world with chemistry and now robotics
 to the digital world with AI, deep learning,
 reinforcement learning, and so on.
 Do you see the physical world
 and the digital world as different?
 And what do you think is the gap?
 A lot of it actually boils down to iteration speed, right?
 That I think that a lot of what really motivates me
 is building things, right?
 Is the, you know, think about mathematics, for example,
 where you think really hard about a problem,
 you understand it, you write it down
 in this very obscure form that we call proof.
 But then this is in humanity's library, right?
 It's there forever.
 This is some truth that we've discovered.
 You know, maybe only five people in your field
 will ever read it, but somehow you've kind of
 moved humanity forward.
 And so I actually used to really think
 that I was going to be a mathematician.
 And then I actually started writing this chemistry textbook.
 One of my friends told me, you'll never publish it
 because you don't have a PhD.
 So instead, I decided to build a website
 and tried to promote my ideas that way.
 And then I discovered programming.
 And you know, that in programming,
 you think hard about a problem,
 you understand it, you write it down
 in a very obscure form that we call a program.
 But then once again, it's in humanity's library, right?
 And anyone can get the benefit from it.
 And the scalability is massive.
 And so I think that the thing that really appeals to me
 about the digital world is that you can have
 this insane leverage, right?
 A single individual with an idea
 is able to affect the entire planet.
 And that's something I think is really hard to do
 if you're moving around physical atoms.
 But you said mathematics.
 So if you look at the wet thing over here, our mind,
 do you ultimately see it as just math,
 as just information processing?
 Or is there some other magic as you've seen,
 if you've seen through biology and chemistry and so on?
 I think it's really interesting to think about
 humans as just information processing systems.
 And that it seems like it's actually a pretty good
 way of describing a lot of kind of how the world works
 or a lot of what we're capable of to think that,
 again, if you just look at technological innovations
 over time, that in some ways,
 the most transformative innovation that we've had
 is has been the computer, right?
 In some ways, the internet,
 what has the internet done, right?
 The internet is not about these physical cables.
 It's about the fact that I am suddenly able
 to instantly communicate with any other human on the planet.
 I'm able to retrieve any piece of knowledge
 that in some ways the human race has ever had.
 And that those are these insane transformations.
 Do you see the, our society as a whole,
 the collective as another extension of the intelligence
 of the human being?
 So if you look at the human being
 as an information processing system,
 you mentioned the internet, the networking.
 Do you see us all together as a civilization
 as a kind of intelligent system?
 Yeah, I think this is actually a really interesting
 perspective to take and to think about
 that you sort of have this collective intelligence
 of all of society.
 The economy itself is this superhuman machine
 that is optimizing something, right?
 And it's almost in some ways a company has a will
 of its own, right?
 That you have all these individuals
 who are all pursuing their own individual goals
 and thinking really hard
 and thinking about the right things to do.
 But somehow the company does something
 that is this emergent thing
 and that is a really useful abstraction.
 And so I think that in some ways,
 we think of ourselves as the most intelligent things
 on the planet and the most powerful things on the planet.
 But there are things that are bigger than us
 that are these systems that we all contribute to.
 And so I think actually, it's interesting to think about
 if you've read Isaac Isamov's foundation, right?
 That there's this concept of psycho history in there,
 which is effectively this,
 that if you have trillions or quadrillions of beings,
 then maybe you could actually predict what that being,
 that huge macro being will do
 and almost independent of what the individuals want.
 And I actually have a second angle on this
 that I think is interesting,
 which is thinking about technological determinism.
 One thing that I actually think a lot about with OpenAI,
 is that we're kind of coming on
 onto this insanely transformational technology
 of general intelligence that will happen at some point.
 And there's a question of how can you take actions
 that will actually steer it to go better rather than worse?
 And that I think one question you need to ask is,
 as a scientist, as an inventor, as a creator,
 what impact can you have in general?
 You look at things like the telephone invented
 by two people on the same day.
 Like, what does that mean?
 Like, what does that mean about the shape of innovation?
 And I think that what's going on is everyone's building
 on the shoulders of the same giants.
 And so you can kind of, you can't really hope
 to create something no one else ever would.
 If Einstein wasn't born,
 someone else would have come up with relativity.
 You change the timeline a bit, right?
 That maybe it would have taken another 20 years,
 but it wouldn't be that fundamentally humanity
 would never discover these fundamental truths.
 There's some kind of invisible momentum
 that some people like Einstein or OpenAI is plugging into,
 that anybody else can also plug into
 and ultimately that wave takes us into a certain direction.
 That's what you mean by determinism.
 That's right, that's right.
 And this kind of seems to play out
 in a bunch of different ways,
 that there's some exponential that is being written
 and that the exponential itself, which one it is, changes.
 Think about Moore's law, an entire industry set,
 it's clocked to it for 50 years.
 Like, how can that be?
 How is that possible?
 And yet somehow it happened.
 And so I think you can't hope to ever invent something
 that no one else will.
 Maybe you can change the timeline a little bit,
 but if you really want to make a difference,
 I think that the thing that you really have to do,
 the only real degree of freedom you have
 is to set the initial conditions
 under which a technology is born.
 And so you think about the internet, right?
 That there are lots of other competitors
 trying to build similar things and the internet won
 and that the initial conditions
 were that it was created by this group
 that really valued people being able to be,
 anyone being able to plug in this very academic mindset
 of being open and connected.
 And I think that the internet for the next 40 years
 really played out that way.
 Maybe today, things are starting to shift
 in a different direction,
 but I think that those initial conditions
 were really important to determine
 the next 40 years worth of progress.
 That's really beautifully put.
 So another example of that I think about,
 I recently looked at it.
 I looked at Wikipedia, the formation of Wikipedia,
 and I wondered what the internet would be like
 if Wikipedia had ads.
 There's an interesting argument that why they chose not
 to put advertisement on Wikipedia.
 I think Wikipedia is one of the greatest resources
 we have on the internet.
 It's extremely surprising how well it works
 and how well it was able to aggregate
 all this kind of good information.
 And essentially, the creator of Wikipedia, I don't know,
 there's probably some debates there,
 but set the initial conditions
 and how it carried itself forward.
 That's really interesting.
 So the way you're thinking about AGI
 or artificial intelligence is you're focused
 on setting the initial conditions for the progress.
 That's right.
 That's powerful.
 Okay, so look into the future.
 If you create an AGI system,
 like one that can ace the Turing test, natural language,
 what do you think would be the interactions
 you would have with it?
 What do you think are the questions you would ask?
 What would be the first question you would ask it, her, him?
 That's right.
 I think that at that point,
 if you've really built a powerful system
 that is capable of shaping the future of humanity,
 the first question that you really should ask
 is how do we make sure that this plays out well?
 And so that's actually the first question
 that I would ask a powerful AGI system is.
 So you wouldn't ask your colleague,
 you wouldn't ask like Ilya,
 you would ask the AGI system.
 Oh, we've already had the conversation with Ilya, right?
 And everyone here.
 And so you want as many perspectives
 and a piece of wisdom as you can
 for answering this question.
 So I don't think you necessarily defer
 to whatever your powerful system tells you,
 but you use it as one input
 to try to figure out what to do.
 But I guess fundamentally,
 what it really comes down to is
 if you've built something really powerful
 and you think about, for example,
 the creation of shortly after
 the creation of nuclear weapons, right?
 The most important question in the world was
 what's the world order going to be like?
 How do we set ourselves up in a place
 where we're going to be able to survive as a species?
 With AGI, I think the question is slightly different, right?
 That there is a question of how do we make sure
 that we don't get the negative effects?
 But there's also the positive side, right?
 You imagine that, you know, like what will an AGI be like?
 Like what will it be capable of?
 And I think that one of the core reasons
 that an AGI can be powerful and transformative
 is actually due to technological development, right?
 If you have something that's capable as a human
 and that it's much more scalable,
 that you absolutely want that thing
 to go read the whole scientific literature
 and think about how to create cures for all the diseases,
 right?
 You want it to think about how to go
 and build technologies to help us create material abundance
 and to figure out societal problems
 that we have trouble with, like,
 how are we supposed to clean up the environment?
 And, you know, maybe you want this to go
 and invent a bunch of little robots that will go out
 and be biodegradable and turn ocean debris
 into harmless molecules.
 And I think that that positive side
 is something that I think people miss sometimes
 when thinking about what an AGI will be like.
 And so I think that if you have a system
 that's capable of all of that,
 you absolutely want its advice about how do I make sure
 that we're using your capabilities
 in a positive way for humanity.
 So what do you think about that psychology
 that looks at all the different possible trajectories
 of an AGI system, many of which,
 perhaps the majority of which are positive,
 and nevertheless focuses on the negative trajectories?
 I mean, you get to interact with folks,
 you get to think about this,
 maybe within yourself as well.
 You look at Sam Harris and so on.
 It seems to be, sorry to put it this way,
 but almost more fun to think about the negative possibilities.
 Whatever that's deep in our psychology,
 what do you think about that?
 And how do we deal with it?
 Because we want AI to help us.
 So I think there's kind of two problems
 entailed in that question.
 The first is more of the question of
 how can you even picture what a world
 with a new technology will be like?
 Now imagine we're in 1950,
 and I'm trying to describe Uber to someone.
 Apps and the internet.
 Yeah, I mean, that's going to be extremely complicated,
 but it's imaginable.
 It's imaginable, right?
 But, and now imagine being in 1950
 and predicting Uber, right?
 And you need to describe the internet,
 you need to describe GPS,
 you need to describe the fact that
 everyone's going to have this phone in their pocket.
 And so I think that just the first truth
 is that it is hard to picture
 how a transformative technology will play out in the world.
 We've seen that before with technologies
 that are far less transformative than AGI will be.
 And so I think that one piece is that
 it's just even hard to imagine
 and to really put yourself in a world
 where you can predict what that positive vision
 would be like.
 And I think the second thing is that
 I think it is always easier to support
 the negative side than the positive side.
 It's always easier to destroy than create.
 And less in a physical sense
 and more just in an intellectual sense, right?
 Because I think that with creating something
 you need to just get a bunch of things right
 and to destroy you just need to get one thing wrong.
 And so I think that what that means
 is that I think a lot of people's thinking dead ends
 as soon as they see the negative story.
 But that being said, I actually have some hope, right?
 I think that the positive vision
 is something that I think can be,
 is something that we can talk about.
 And I think that just simply saying this fact of,
 yeah, there's positive, there's negatives,
 everyone likes to dwell on the negative.
 People actually respond well to that message
 and say, huh, you're right, there's a part of this
 that we're not talking about, not thinking about.
 And that's actually something that's, I think,
 really been a key part of how we think about AGI at OpenAI.
 You can kind of look at it as like, okay,
 OpenAI talks about the fact that there are risks
 and yet they're trying to build this system.
 How do you square those two facts?
 So do you share the intuition that some people have,
 I mean, from Sam Harris to even Elon Musk himself,
 that it's tricky as you develop AGI
 to keep it from slipping into the existential threats,
 into the negative?
 What's your intuition about how hard is it
 to keep AI development on the positive track?
 What's your intuition there?
 To answer the question,
 you can really look at how we structure OpenAI.
 So we really have three main arms.
 We have capabilities,
 which is actually doing the technical work
 and pushing forward what these systems can do.
 There's safety, which is working on technical mechanisms
 to ensure that the systems we build
 are aligned with human values.
 And then there's policy,
 which is making sure that we have governance mechanisms,
 answering that question of, well, whose values?
 And so I think that the technical safety one
 is the one that people kind of talk about the most, right?
 You talk about, like think about,
 you know, all of the dystopic AI movies,
 a lot of that is about not having
 good technical safety in place.
 And what we've been finding is that,
 you know, I think that actually a lot of people
 look at the technical safety problem
 and think it's just intractable.
 Right, this question of what do humans want?
 How am I supposed to write that down?
 Can I even write down what I want?
 No way.
 And then they stop there.
 But the thing is we've already built systems
 that are able to learn things that humans can't specify.
 You know, even the rules for how to recognize
 if there's a cat or a dog in an image.
 Turns out it's intractable to write that down.
 And yet we're able to learn it.
 And that what we're seeing with systems we build at OpenAI,
 and they're still in early proof of concept stage,
 is that you are able to learn human preferences.
 You're able to learn what humans want from data.
 And so that's kind of the core focus
 for our technical safety team.
 And I think that there actually
 we've had some pretty encouraging updates
 in terms of what we've been able to make work.
 So you have an intuition and a hope that from data,
 you know, looking at the value alignment problem,
 from data, we can build systems that align
 with the collective better angels of our nature.
 So align with the ethics and the morals of human beings.
 To even say this in a different way,
 I mean, think about how do we align humans, right?
 Think about like a human baby can grow up
 to be an evil person or a great person.
 And a lot of that is from learning from data, right?
 That you have some feedback as a child is growing up,
 they get to see positive examples.
 And so I think that just like the only example we have
 of a general intelligence that is able to learn from data
 to align with human values and to learn values,
 I think we shouldn't be surprised
 that we can do the same sorts of techniques
 or whether the same sort of techniques end up being
 how we solve value alignment for AGI's.
 So let's go even higher.
 I don't know if you've read the book Sapiens,
 but there's an idea that, you know,
 that as a collective, as us human beings,
 we kind of develop together ideas that we hold.
 There's no, in that context, objective truth.
 We just kind of all agree to certain ideas
 and hold them as a collective.
 Did you have a sense that there is,
 in the world of good and evil,
 do you have a sense that to the first approximation,
 there are some things that are good
 and that you could teach systems to behave to be good?
 So I think that this actually blends into our third team,
 right, which is the policy team.
 And this is the one, the aspect I think people really
 talk about way less than they should, right?
 Because imagine that we build super powerful systems
 that we've managed to figure out all the mechanisms
 for these things to do whatever the operator wants.
 The most important question becomes who's the operator,
 what do they want, and how is that going to affect
 everyone else, right?
 And I think that this question of what is good,
 what are those values, I mean,
 I think you don't even have to go to those
 very grand existential places to start,
 to realize how hard this problem is.
 You just look at different countries and cultures
 across the world and that there's a very different
 conception of how the world works
 and what kinds of ways that society wants to operate.
 And so I think that the really core question
 is actually very concrete.
 And I think it's not a question
 that we have ready answers to, right?
 It's how do you have a world where all the different
 countries that we have, United States, China, Russia,
 and the hundreds of other countries out there
 are able to continue to not just operate
 in the way that they see fit,
 but in the world that emerges in these,
 where you have these very powerful systems
 operating alongside humans ends up being something
 that empowers humans more,
 that makes human existence be a more meaningful thing
 and that people are happier and wealthier
 and able to live more fulfilling lives.
 It's not an obvious thing for how to design that world
 once you have that very powerful system.
 So if we take a little step back,
 and we're having like a fascinating conversation
 and OpenAI is in many ways a tech leader in the world,
 and yet we're thinking about these big existential
 questions which is fascinating and really important.
 I think you're a leader in that space
 and that's a really important space
 of just thinking how AI affects society
 in a big picture view.
 So Oscar Wilde said, we're all in the gutter,
 but some of us are looking at the stars
 and I think OpenAI has a charter that looks to the stars,
 I would say, to create intelligence,
 to create general intelligence,
 make it beneficial, safe, and collaborative.
 Can you tell me how that came about,
 how a mission like that and the path
 to creating a mission like that at OpenAI was founded?
 Yeah, so I think that in some ways
 it really boils down to taking a look at the landscape.
 So if you think about the history of AI,
 that basically for the past 60 or 70 years,
 people have thought about this goal
 of what could happen if you could automate
 human intellectual labor.
 Imagine you can build a computer system that could do that,
 what becomes possible?
 We have a lot of sci-fi that tells stories
 of various dystopias and increasingly you have movies
 like Herd that tell you a little bit about
 maybe more of a little bit utopic vision.
 You think about the impacts that we've seen
 from being able to have bicycles for our minds
 and computers and I think that the impact
 of computers and the internet has just far outstripped
 what anyone really could have predicted.
 And so I think that it's very clear
 that if you can build an AGI,
 it will be the most transformative technology
 that humans will ever create.
 And so what it boils down to then is a question of,
 well, is there a path?
 Is there hope?
 Is there a way to build such a system?
 And I think that for 60 or 70 years,
 that people got excited and that ended up
 not being able to deliver on the hopes
 that people had pinned on them.
 And I think that then, after two winters of AI development,
 that people I think kind of almost stopped daring to dream.
 Really talking about AGI or thinking about AGI
 became almost this taboo in the community.
 But I actually think that people took the wrong lesson
 from AI history.
 And if you look back, starting in 1959
 is when the Perceptron was released.
 And this is basically one of the earliest neural networks.
 It was released to what was perceived
 as this massive overhype.
 So in the New York Times in 1959,
 you have this article saying that the Perceptron
 will one day recognize people, call out their names,
 instantly translate speech between languages.
 And people at the time looked at this and said,
 this is, your system can't do any of that.
 And basically spent 10 years trying to discredit
 the whole Perceptron direction and succeeded.
 And all the funding dried up and people kind of
 went in other directions.
 And in the 80s, there was a resurgence.
 And I'd always heard that the resurgence in the 80s
 was due to the invention of back propagation
 and these algorithms that got people excited.
 But actually the causality was due to people
 building larger computers.
 That you can find these articles from the 80s
 saying that the democratization of computing power
 suddenly meant that you could run these larger neural networks
 and then people started to do all these amazing things.
 The back propagation algorithm was invented.
 And the neural nets people were running
 were these tiny little 20 neuron neural nets.
 What are you supposed to learn with 20 neurons?
 And so of course, they weren't able to get great results.
 And it really wasn't until 2012 that this approach,
 that's almost the most simple natural approach
 that people have come up with in the 50s.
 In some ways even in the 40s before there were computers
 with the Pitz McCollum neuron,
 suddenly this became the best way of solving problems.
 I think there are three core properties
 that deep learning has that I think
 are very worth paying attention to.
 The first is generality.
 We have a very small number of deep learning tools.
 SGD, deep neural net, maybe some RL.
 And it solves this huge variety of problems.
 Speech recognition, machine translation, game playing,
 all of these problems, small set of tools.
 So there's the generality.
 There's a second piece, which is the competence.
 You wanna solve any of those problems
 throughout 40 years worth of normal computer vision research
 replace it with a deep neural net, it's gonna work better.
 And there's a third piece, which is the scalability.
 One thing that has been shown time and time again
 is that if you have a larger neural network,
 throw more compute, more data at it, it will work better.
 Those three properties together feel like essential parts
 of building a general intelligence.
 Now it doesn't just mean that if we scale up what we have,
 that we will have an AGI, right?
 There are clearly missing pieces, there are missing ideas.
 We need to have answers for reasoning.
 But I think that the core here is that for the first time,
 it feels that we have a paradigm that gives us hope
 that general intelligence can be achievable.
 And so as soon as you believe that,
 everything else becomes comes into focus, right?
 If you imagine that you may be able to,
 and that the timeline I think remains uncertain,
 but I think that certainly within our lifetimes
 and possibly within a much shorter period of time
 than people would expect, if you can really build
 the most transformative technology that will ever exist,
 you stop thinking about yourself so much, right?
 And you start thinking about just like,
 how do you have a world where this goes well?
 And that you need to think about the practicalities
 of how do you build an organization
 and get together a bunch of people and resources,
 and to make sure that people feel motivated
 and ready to do it.
 But I think that then you start thinking about,
 well, what if we succeed?
 And how do we make sure that when we succeed,
 that the world is actually the place
 that we want ourselves to exist in?
 And almost in the Ralsi and Bale sense of the word.
 And so that's kind of the broader landscape.
 And OpenAI was really formed in 2015
 with that high level picture of AGI might be possible
 sooner than people think,
 and that we need to try to do our best
 to make sure it's going to go well.
 And then we spent the next couple of years
 really trying to figure out what does that mean?
 How do we do it?
 And I think that typically with a company,
 you start out very small, see you in a co-founder,
 and you build a product, you get some users,
 you get product market fit.
 Then at some point you raise some money,
 you hire people, you scale, and then down the road,
 then the big companies realize you exist
 and try to kill you.
 And for OpenAI, it was basically everything
 in exactly the opposite order.
 Ah.
 Ah.
 Ah.
 Let me just pause for a second.
 You said a lot of things.
 And let me just admire the jarring aspect
 of what OpenAI stands for, which is daring to dream.
 I mean, you said it, it's pretty powerful.
 It caught me off guard because I think that's very true.
 The step of just daring to dream about the possibilities
 of creating intelligence in a positive, in a safe way,
 but just even creating intelligence
 is a much needed, refreshing catalyst
 for the AI community.
 So that's the starting point.
 Okay, so then formation of OpenAI.
 I would just say that when we were starting OpenAI,
 that kind of the first question that we had is,
 is it too late to start a lab
 with a bunch of the best people?
 Right, is that even possible?
 That was an actual question.
 That was really, that was the core question of,
 we had this dinner in July of 2015,
 and that was really what we spent
 the whole time talking about.
 And, you know, because it's,
 you think about kind of where AI was,
 is that it had transitioned from being an academic pursuit
 to an industrial pursuit.
 And so a lot of the best people
 were in these big research labs,
 and that we wanted to start our own one
 that no matter how much resources we could accumulate
 would be pale in comparison to the big tech companies.
 And we knew that.
 And there was a question of,
 are we going to be actually able
 to get this thing off the ground?
 You need a critical mass.
 You can't just do you and a co-founder build a product.
 Right, you really need to have a group of,
 you know, five to 10 people.
 And we kind of concluded it wasn't obviously impossible.
 So it seemed worth trying.
 Well, you're also a dreamer, so who knows, right?
 That's right.
 Okay, so speaking of that,
 competing with the big players,
 let's talk about some of the tricky things
 as you think through this process of growing,
 of seeing how you can develop these systems
 at scale that competes.
 So you recently formed OpenAI LP,
 a new cap profit company that now carries the name OpenAI.
 So OpenAI is now this official company.
 The original non-profit company still exists
 and carries the OpenAI non-profit name.
 So can you explain what this company is,
 what the purpose of its creation is,
 and how did you arrive at the decision to create it?
 OpenAI, the whole entity and OpenAI LP as a vehicle
 is trying to accomplish the mission
 of ensuring that artificial general intelligence
 benefits everyone.
 And the main way that we're trying to do that
 is by actually trying to build general intelligence
 to ourselves and make sure the benefits
 are distributed to the world.
 That's the primary way.
 We're also fine if someone else does this, right?
 It doesn't have to be us.
 If someone else is going to build an AGI
 and make sure that the benefits don't get locked up
 in one company or with one set of people,
 we're actually fine with that.
 And so those ideas are baked into our charter,
 which is kind of the foundational document
 that describes kind of our values and how we operate.
 It was also really baked into the structure of OpenAI LP.
 And so the way that we've set up OpenAI LP
 is that in the case where we succeed, right?
 If we actually build what we're trying to build,
 then investors are able to get a return.
 But that return is something that is capped.
 And so if you think of AGI in terms of the value
 that you could really create,
 you're talking about the most transformative technology
 ever created.
 It's gonna create orders of magnitude more value
 than any existing company.
 And that all of that value will be owned by the world,
 like legally titled to the nonprofit to fulfill that mission.
 And so that's the structure.
 So the mission is a powerful one
 and it's one that I think most people would agree with.
 It's how we would hope AI progresses.
 And so how do you tie yourself to that mission?
 How do you make sure you do not deviate from that mission?
 That other incentives that are profit driven
 don't interfere with the mission.
 So this was actually a really core question for us
 for the past couple of years,
 because I'd say that like the way that our history went
 was that for the first year,
 we were getting off the ground, right?
 We had this high level picture,
 but we didn't know exactly how we wanted to accomplish it.
 And really two years ago,
 it's when we first started realizing
 in order to build AGI,
 we're just going to need to raise way more money
 than we can as a nonprofit.
 And we're talking many billions of dollars.
 And so the first question is,
 how are you supposed to do that
 and stay true to this mission?
 And we looked at every legal structure out there
 and included none of them were quite right
 for what we wanted to do.
 And I guess it shouldn't be too surprising
 if you're going to do some like crazy unprecedented
 technology that you're going to have to come up
 with some crazy unprecedented structure to do it in.
 And a lot of our conversation was with people at OpenAI,
 right, the people who really joined
 because they believe so much in this mission
 and thinking about how do we actually raise the resources
 to do it and also stay true to what we stand for.
 And the place you got to start is to really align
 on what is it that we stand for, right?
 What are those values?
 What's really important to us?
 And so I'd say that we spent about a year
 really compiling the OpenAI charter
 and that that determines,
 and if you even look at the first line item in there,
 it says that, look, we expect we're going to have
 to marshal huge amounts of resources,
 but we're going to make sure that we minimize conflict
 of interest with the mission.
 And that kind of aligning on all of those pieces
 was the most important step towards figuring out
 how do we structure a company
 that can actually raise the resources
 to do what we need to do.
 I imagine OpenAI, the decision to create OpenAI LP
 was a really difficult one,
 and there was a lot of discussions,
 as you mentioned, for a year,
 and there was different ideas,
 perhaps detractors within OpenAI,
 sort of different paths that you could have taken.
 What were those concerns?
 What were the different paths considered?
 What was that process of making that decision like?
 Yep, so if you look actually at the OpenAI charter,
 that there's almost two paths embedded within it.
 There is, we are primarily trying to build AGI ourselves,
 but we're also okay if someone else does it.
 And this is a weird thing for a company.
 It's really interesting, actually.
 Yeah.
 There is an element of competition
 that you do want to be the one that does it,
 but at the same time, you're okay if somebody else doesn't.
 We'll talk about that a little bit, that trade-off,
 that dance, that's really interesting.
 And I think this was the core tension
 as we were designing OpenAI LP,
 and really the OpenAI strategy,
 is how do you make sure that both you have a shot
 at being a primary actor,
 which really requires building an organization,
 raising massive resources,
 and really having the will to go
 and execute on some really, really hard vision, right?
 You need to really sign up for a long period
 to go and take on a lot of pain and a lot of risk.
 And to do that, normally,
 you just import the startup mindset, right?
 And that you think about, okay,
 how do we out-execute everyone?
 You have this very competitive angle.
 But you also have this second angle of saying that,
 well, the true mission isn't for OpenAI to build AGI.
 The true mission is for AGI to go well for humanity.
 And so, how do you take all of those first actions
 and make sure you don't close the door on outcomes
 that would actually be positive and fulfill the mission?
 And so I think it's a very delicate balance, right?
 And I think that going 100% one direction or the other
 is clearly not the correct answer.
 And so I think that even in terms of just how we talk about
 OpenAI and think about it,
 there's just like one thing that's always
 in the back of my mind is to make sure
 that we're not just saying OpenAI's goal is to build AGI,
 right, that it's actually much broader than that, right?
 That first of all, it's not just AGI,
 it's safe AGI that's very important.
 But secondly, our goal isn't to be the ones to build it,
 our goal is to make sure it goes well for the world.
 And so I think that figuring out
 how do you balance all of those
 and to get people to really come to the table
 and compile a single document that encompasses all of that
 wasn't trivial.
 So part of the challenge here is your mission is,
 I would say, beautiful, empowering,
 and a beacon of hope for people in the research community
 and just people thinking about AI.
 So your decisions are scrutinized more than,
 I think, a regular profit-driven company.
 Do you feel the burden of this in the creation of the charter
 and just in the way you operate?
 Yes.
 So why do you lean into the burden
 by creating such a charter?
 Why not keep it quiet?
 I mean, it just boils down to the mission, right?
 Like I'm here and everyone else is here
 because we think this is the most important mission.
 Dare to dream.
 All right, so do you think you can be good for the world
 or create an AGI system that's good
 when you're a for-profit company?
 From my perspective, I don't understand why profit
 interferes with positive impact on society.
 I don't understand why Google,
 that makes most of its money from ads,
 can't also do good for the world
 or other companies, Facebook, anything.
 I don't understand why those have to interfere.
 You can, profit isn't the thing in my view
 that affects the impact of a company.
 What affects the impact of the company is the charter,
 is the culture, is the people inside,
 and profit is the thing that just fuels those people.
 What are your views there?
 Yeah, so I think that's a really good question
 and there's some real long-standing debates
 in human society that are wrapped up in it.
 The way that I think about it is just think about
 what are the most impactful non-profits in the world?
 What are the most impactful for-profits in the world?
 Right, it's much easier to list the for-profits.
 That's right, and I think that there's some real truth here
 that the system that we set up,
 the system for kind of how today's world is organized,
 is one that really allows for huge impact.
 And that kind of part of that is that you need to be,
 that for-profits are self-sustaining
 and able to kind of build on their own momentum.
 And I think that's a really powerful thing.
 It's something that when it turns out
 that we haven't set the guardrails correctly,
 causes problems, right?
 Think about logging companies that go into a forest,
 the rainforest, that's really bad, we don't want that.
 And it's actually really interesting to me
 that kind of this question of how do you get
 positive benefits out of a for-profit company,
 it's actually very similar to how do you get
 positive benefits out of an AGI, right?
 That you have this like very powerful system
 that's more powerful than any human
 and is kind of autonomous in some ways,
 it's superhuman in a lot of axes,
 and somehow you have to set the guardrails
 to get good things to happen.
 But when you do, the benefits are massive.
 And so I think that when I think about
 non-profit versus for-profit,
 I think just not enough happens in non-profits,
 they're very pure, but it's just kind of,
 it's just hard to do things there.
 In for-profits in some ways, like too much happens,
 but if kind of shaped in the right way,
 it can actually be very positive.
 And so with OpenAILP, we're picking a road in between.
 Now, the thing that I think is really important to recognize
 is that the way that we think about OpenAILP
 is that in the world where AGI actually happens, right?
 In a world where we are successful,
 we build the most transformative technology ever,
 the amount of value we're gonna create will be astronomical.
 And so then in that case,
 that the cap that we have will be a small fraction
 of the value we create.
 And the amount of value that goes back to investors
 and employees looks pretty similar to what would happen
 in a pretty successful startup.
 And that's really the case that we're optimizing for, right?
 That we're thinking about in the success case,
 making sure that the value we create doesn't get locked up.
 And I expect that in other for-profit companies
 that it's possible to do something like that.
 I think it's not obvious how to do it, right?
 And I think that as a for-profit company,
 you have a lot of fiduciary duty to your shareholders
 and that there are certain decisions
 that you just cannot make in our structure.
 We've set it up so that we have a fiduciary duty
 to the charter, that we always get to make the decision
 that is right for the charter,
 rather than even if it comes at the expense
 of our own stakeholders.
 And so I think that when I think about
 what's really important,
 it's not really about nonprofit versus for-profit.
 It's really a question of if you build AGI
 and you kind of, you know, humanity is now in this new age,
 who benefits, whose lives are better?
 And I think that what's really important
 is to have an answer that is everyone.
 Yeah, which is one of the core aspects of the charter.
 So one concern people have, not just with OpenAI,
 but with Google, Facebook, Amazon, anybody really
 that's creating impact at scale,
 is how do we avoid, as your charter says,
 avoid enabling the use of AI or AGI
 to unduly concentrate power?
 Why would not a company like OpenAI
 keep all the power of an AGI system to itself?
 The charter.
 The charter, so how does the charter
 actualize itself in day to day?
 So I think that the first to zoom out, right,
 that the way that we structure the company
 is so that the power for sort of dictating the actions
 that OpenAI takes ultimately rests with the board, right?
 The board of the nonprofit.
 And the board is set up in certain ways,
 with certain restrictions that you can read about
 in the OpenAI LP blog post.
 But effectively the board is the governing body
 for OpenAI LP.
 And the board has a duty to fulfill the mission
 of the nonprofit.
 And so that's kind of how we tie,
 how we thread all these things together.
 Now there's a question of, so day to day,
 how do people, the individuals,
 who in some ways are the most empowered ones, right?
 The board sort of gets to call the shots at the high level,
 but the people who are actually executing
 are the employees, right?
 The people here on a day to day basis
 who have the keys to the technical kingdom.
 And there I think that the answer looks a lot like,
 well, how does any company's values get actualized, right?
 And I think that a lot of that comes down to
 that you need people who are here
 because they really believe in that mission,
 and they believe in the charter,
 and that they are willing to take actions
 that maybe are worse for them,
 but are better for the charter.
 And that's something that's really baked into the culture.
 And honestly, I think it's,
 I think that that's one of the things
 that we really have to work to preserve as time goes on.
 And that's a really important part
 of how we think about hiring people
 and bringing people into OpenAI.
 So there's people here,
 there's people here who could speak up and say,
 like, hold on a second,
 this is totally against what we stand for, culture-wise.
 Yeah, yeah, for sure.
 I mean, I think that we actually have,
 I think that's like a pretty important part
 of how we operate and how we have,
 even again, with designing the charter
 and designing OpenAILP in the first place,
 that there has been a lot of conversation
 with employees here,
 and a lot of times where employees said, wait a second,
 this seems like it's going in the wrong direction,
 and let's talk about it.
 And so I think one thing that's, I think, a really,
 and here's actually one thing that I think is very unique
 about us as a small company,
 is that if you're at a massive tech giant,
 it's a little bit hard for someone who's a line employee
 to go and talk to the CEO and say,
 I think that we're doing this wrong.
 And you look at companies like Google
 that have had some collective action from employees
 to make ethical change around things like Maven.
 And so maybe there are mechanisms
 that other companies that work,
 but here, super easy for anyone to pull me aside,
 to pull Sam aside, to pull Ilya aside,
 and people do it all the time.
 One of the interesting things in the charter
 is this idea that it'd be great
 if you could try to describe or untangle
 switching from competition to collaboration
 and late stage AGI development.
 It was really interesting,
 this dance between competition and collaboration.
 How do you think about that?
 Yeah, assuming that you can actually do
 the technical side of AGI development,
 I think there's going to be two key problems
 with figuring out how do you actually deploy it,
 make it go well.
 The first one of these is the run up
 to building the first AGI.
 You look at how self-driving cars are being developed,
 and it's a competitive race.
 And the thing that always happens in competitive race
 is that you have huge amounts of pressure
 to get rid of safety.
 And so that's one thing we're very concerned about, right?
 Is that people, multiple teams figuring out
 we can actually get there,
 but if we took the slower path
 that is more guaranteed to be safe, we will lose.
 And so we're going to take the fast path.
 And so the more that we can both ourselves
 be in a position where we don't generate
 that competitive race, where we say
 if the race is being run and that someone else
 is further ahead than we are,
 we're not going to try to leapfrog,
 we're going to actually work with them, right?
 We will help them succeed
 as long as what they're trying to do
 is to fulfill our mission, then we're good.
 We don't have to build AGI ourselves.
 And I think that's a really important commitment from us,
 but it can't just be unilateral, right?
 I think that it's really important
 that other players who are serious about building AGI
 make similar commitments, right?
 And I think that, you know, again,
 to the extent that everyone believes
 that AGI should be something to benefit everyone,
 then it actually really shouldn't matter
 which company builds it.
 And we should all be concerned about the case
 where we just race so hard to get there
 that something goes wrong.
 So what role do you think government,
 our favorite entity, has in setting policy and rules
 about this domain, from research to the development
 to early stage to late stage AI and AGI development?
 So I think that, first of all,
 it's really important that government's in there, right?
 In some way, shape, or form.
 You know, at the end of the day,
 we're talking about building technology
 that will shape how the world operates,
 and that there needs to be government
 as part of that answer.
 And so that's why we've done
 a number of different congressional testimonies.
 We interact with a number of different lawmakers,
 and that, you know, right now,
 a lot of our message to them
 is that it's not the time for regulation,
 it is the time for measurement, right?
 That our main policy recommendation is that people,
 and, you know, the government does this all the time
 with bodies like NIST,
 spend time trying to figure out
 just where the technology is, how fast it's moving,
 and can really become literate and up to speed
 with respect to what to expect.
 So I think that today, the answer really is about measurement,
 and I think that there will be a time and place
 where that will change.
 And I think it's a little bit hard to predict exactly
 what exactly that trajectory should look like.
 So there will be a point at which regulation,
 federal and the United States, the government steps in
 and helps be the, I don't want to say the adult in the room,
 to make sure that there is strict rules,
 maybe conservative rules that nobody can cross.
 Well, I think there's kind of maybe two angles to it.
 So today, with narrow AI applications,
 that I think there are already existing bodies
 that are responsible and should be responsible
 for regulation.
 You think about, for example, with self-driving cars,
 that you want the National Highway,
 exactly to be regulating that.
 That makes sense, right?
 That basically what we're saying is that we're going to have
 these technological systems that are going to be
 performing applications that humans already do.
 Great, we already have ways of thinking about standards
 and safety for those.
 So I think actually empowering those regulators today
 is also pretty important.
 And then I think for AGI, that there's going to be a point
 where we'll have better answers.
 And I think that maybe a similar approach
 of first measurement and start thinking about
 what the rules should be.
 I think it's really important that we don't prematurely
 squash progress.
 I think it's very easy to smother a budding field.
 And I think that's something to really avoid.
 But I don't think it's the right way of doing it
 is to say, let's just try to blaze ahead
 and not involve all these other stakeholders.
 So you've recently released a paper on GPT-2
 language modeling, but did not release the full model
 because you had concerns about the possible negative effects
 of the availability of such model.
 It's outside of just that decision,
 it's super interesting because of the discussion
 at a societal level, the discourse it creates.
 So it's fascinating in that aspect.
 But if you think at the specifics here at first,
 what are some negative effects that you envisioned?
 And of course, what are some of the positive effects?
 Yeah, so again, I think to zoom out,
 the way that we thought about GPT-2
 is that with language modeling,
 we are clearly on a trajectory right now
 where we scale up our models
 and we get qualitatively better performance.
 GPT-2 itself was actually just a scale up of a model
 that we've released in the previous June, right?
 And we just ran it at much larger scale
 and we got these results where suddenly starting
 to write coherent prose,
 which was not something we'd seen previously.
 And what are we doing now?
 Well, we're gonna scale up GPT-2 by 10x, by 100x,
 by 1000x, and we don't know what we're gonna get.
 And so it's very clear that the model
 that we released last June,
 I think it's kind of like, it's a good academic toy.
 It's not something that we think is something
 that can really have negative applications
 or to the extent that it can,
 that the positive of people being able to play with it
 is far outweighs the possible harms.
 You fast forward to not GPT-2, but GPT-20,
 and you think about what that's gonna be like.
 And I think that the capabilities
 are going to be substantive.
 And so there needs to be a point in between the two
 where you say, this is something
 where we are drawing the line
 and that we need to start thinking about the safety aspects.
 And I think for GPT-2, we could have gone either way.
 And in fact, when we had conversations internally,
 that we had a bunch of pros and cons,
 and it wasn't clear which one outweighed the other.
 And I think that when we announced that,
 hey, we decide not to release this model,
 then there was a bunch of conversation
 where various people said,
 it's so obvious that you should have just released it.
 There are other people said,
 it's so obvious you should not have released it.
 And I think that that almost definitionally means
 that holding it back was the correct decision.
 If it's not obvious whether something is beneficial or not,
 you should probably default to caution.
 And so I think that the overall landscape
 for how we think about it
 is that this decision could have gone either way.
 There are great arguments in both directions,
 but for future models down the road
 and possibly sooner than you'd expect
 because scaling these things up
 doesn't actually take that long,
 those ones you're definitely not going to want
 to release into the wild.
 And so I think that we almost view this as a test case
 and to see, can we even design,
 how do you have a society,
 how do you have a system
 that goes from having no concept of responsible disclosure
 where the mere idea of not releasing something
 for safety reasons is unfamiliar
 to a world where you say, okay, we have a powerful model,
 let's at least think about it,
 let's go through some process.
 And you think about the security community,
 it took them a long time
 to design responsible disclosure.
 You think about this question of,
 well, I have a security exploit,
 I send it to the company,
 the company is like tries to prosecute me
 or just ignores it, what do I do?
 And so the alternatives of,
 oh, I just always publish your exploits,
 that doesn't seem good either.
 And so it really took a long time
 and it was bigger than any individual,
 it's really about building a whole community
 that believe that, okay, we'll have this process
 where you send it to the company,
 if they don't act in a certain time,
 then you can go public and you're not a bad person,
 you've done the right thing.
 And I think that in AI, part of the response at GPT-2
 just proves that we don't have any concept of this.
 So that's the high level picture.
 And so I think that,
 I think this was a really important move to make
 and we could have maybe delayed it for GPT-3,
 but I'm really glad we did it for GPT-2.
 And so now you look at GPT-2 itself
 and you think about the substance of, okay,
 what are potential negative applications?
 So you have this model that's been trained on the internet,
 which it's also going to be a bunch of very biased data,
 a bunch of very offensive content in there.
 And you can ask it to generate content for you
 on basically any topic, right?
 You just give it a prompt and it'll just start writing
 and it writes content like you see on the internet,
 even down to like saying advertisement
 in the middle of some of its generations.
 And you think about the possibilities
 for generating fake news or abusive content.
 And it's interesting seeing what people have done with,
 we released a smaller version of GPT-2
 and the people have done things like try to generate,
 take my own Facebook message history
 and generate more Facebook messages like me
 and people generating fake politician content
 or there's a bunch of things there
 where you at least have to think,
 is this going to be good for the world?
 There's the flip side,
 which is I think that there's a lot of awesome applications
 that we really want to see,
 like creative applications in terms of,
 if you have sci-fi authors that can work with this tool
 and come with cool ideas,
 like that seems awesome if we can write better sci-fi
 through the use of these tools.
 And we've actually had a bunch of people write into us
 asking, hey, can we use it for,
 a variety of different creative applications?
 So the positive are actually pretty easy to imagine.
 The usual NLP applications are really interesting,
 but let's go there.
 It's kind of interesting to think about a world where,
 look at Twitter, where not just fake news,
 but smarter and smarter bots being able to spread
 in an interesting complex networking way information
 that just floods out us regular human beings
 with our original thoughts.
 So what are your views of this world with GPT-20, right?
 How do we think about it?
 Again, it's like one of those things about in the 50s
 trying to describe the internet or the smartphone.
 What do you think about that world,
 the nature of information?
 One possibility is that we'll always try to design systems
 that identify robot versus human,
 and we'll do so successfully,
 and so we'll authenticate that we're still human.
 And the other world is that we just accept the fact
 that we're swimming in a sea of fake news
 and just learn to swim there.
 Well, have you ever seen the,
 there's a popular meme of a robot with a physical arm
 and pen clicking the I'm not a robot button?
 Yeah.
 I think that the truth is that really trying
 to distinguish between robot and human is a losing battle.
 Ultimately, you think it's a losing battle.
 I think it's a losing battle ultimately, right?
 I think that that is that in terms of the content,
 in terms of the actions that you can take,
 I mean, think about how captures have gone, right?
 The captures used to be a very nice, simple,
 you just have this image, all of our OCR is terrible,
 you put a couple of artifacts in it,
 humans are gonna be able to tell what it is,
 an AI system wouldn't be able to.
 Today, I can barely do captures.
 And I think that this is just kind of where we're going.
 I think captures were a moment in time thing,
 and as AI systems become more powerful,
 that there being human capabilities that can be measured
 in a very easy automated way
 that AIs will not be capable of,
 I think that's just like,
 it's just an increasingly hard technical battle.
 But it's not that all hope is lost, right?
 You think about how do we already authenticate ourselves,
 right?
 We have systems, we have social security numbers,
 if you're in the US or you have ways
 of identifying individual people
 and having real world identity tied to digital identity
 seems like a step towards authenticating the source
 of content rather than the content itself.
 Now, there are problems with that.
 How can you have privacy and anonymity
 in a world where the only content you can really trust is,
 or the only way you can trust content
 is by looking at where it comes from.
 And so I think that building out good reputation networks,
 maybe one possible solution.
 But yeah, I think that this question is not an obvious one.
 And I think that we, maybe sooner than we think
 we'll be in a world where today,
 I often will read a tweet and be like,
 do I feel like a real human wrote this?
 Or do I feel like this is like genuine?
 Do I feel like I can kind of judge the content a little bit?
 And I think in the future, it just won't be the case.
 You will get, for example, the FCC comments on net neutrality.
 It came out later that millions of those
 were auto-generated and that the researchers
 were able to do various statistical techniques to do that.
 What do you do in a world where
 those statistical techniques don't exist?
 It's just impossible to tell the difference
 between humans and AIs.
 And in fact, the most persuasive arguments
 are written by AI.
 All that stuff, it's not sci-fi anymore.
 You look at GPT-2 making a great argument
 for why recycling is bad for the world.
 You gotta read that and be like, huh, you're right.
 We are addressing those symptoms.
 Yeah, that's quite interesting.
 I mean, ultimately it boils down to the physical world
 being the last frontier of proving,
 so you said like basically networks of people,
 humans vouching for humans in the physical world.
 And somehow the authentication ends there.
 I mean, if I had to ask you,
 I mean, you're way too eloquent for a human.
 So if I had to ask you to authenticate,
 like prove how do I know you're not a robot
 and how do you know I'm not a robot?
 I think that's so far where in this space,
 this conversation we just had,
 the physical movements we did is the biggest gap
 between us and AI systems is the physical manipulation.
 So maybe that's the last frontier.
 Well, here's another question is why is solving
 this problem important?
 What aspects are really important to us?
 I think that probably where we'll end up
 is we'll hone in on what do we really want
 out of knowing if we're talking to a human.
 And I think that again, this comes down to identity.
 And so I think that the internet of the future,
 I expect to be one that will have lots of agents out there
 that will interact with you.
 But I think that the question of is this real flesh
 and blood human or is this an automated system
 may actually just be less important.
 Let's actually go there.
 It's GPT-2 is impressive and let's look at GPT-20.
 Why is it so bad that all my friends are GPT-20?
 Why is it so important on the internet,
 do you think, to interact with only human beings?
 Why can't we live in a world where ideas can come
 from models trained on human data?
 Yeah, I think this is actually a really interesting question.
 This comes back to the how do you even picture a world
 with some new technology?
 And I think that one thing that I think is important
 is let's say honesty.
 And I think that if you have almost in the Turing test style
 sense of technology, you have AIs that are pretending
 to be humans and deceiving you.
 I think that feels like a bad thing.
 I think that it's really important that we feel
 like we're in control of our environment,
 that we understand who we're interacting with
 and if it's an AI or a human,
 that that's not something that we're being deceived about.
 But I think that the flip side of can I have as meaningful
 of an interaction with an AI as I can with a human?
 Well, I actually think here you can turn to Sci-Fi
 and her I think is a great example
 of asking this very question.
 One thing I really love about her is it really starts out
 almost by asking how meaningful
 are human virtual relationships, right?
 And then you have a human who has a relationship with an AI
 and that you really start to be drawn into that, right?
 And that all of your emotional buttons get triggered
 in the same way as if there was a real human
 that was on the other side of that phone.
 And so I think that this is one way of thinking about it
 is that I think that we can have meaningful interactions
 and that if there's a funny joke,
 some sense it doesn't really matter
 if it was written by a human or an AI,
 but what you don't want in a way
 where I think we should really draw hard lines is deception.
 And I think that as long as we're in a world where,
 why do we build AI systems at all, right?
 The reason we want to build them is to enhance human lives,
 to make humans be able to do more things,
 to have humans feel more fulfilled.
 And if we can build AI systems that do that,
 sign me up.
 So the process of language modeling,
 how far do you think it take us?
 Let's look at movie Her.
 Do you think dialogue, natural language conversation
 is formulated by the Turing test, for example,
 do you think that process could be achieved
 through this kind of unsupervised language modeling?
 So I think the Turing test in its real form
 isn't just about language, right?
 It's really about reasoning too, right?
 To really pass the Turing test,
 I should be able to teach calculus
 to whoever's on the other side
 and have it really understand calculus
 and be able to go and solve new calculus problems.
 And so I think that to really solve the Turing test,
 we need more than what we're seeing with language models.
 We need some way of plugging in reasoning.
 Now, how different will that be from what we already do?
 That's an open question, right?
 It might be that we need some sequence
 of totally radical new ideas,
 or it might be that we just need to kind of shape
 our existing systems in a slightly different way.
 But I think that in terms of how far language modeling
 will go, it's already gone way further
 than many people would have expected, right?
 I think that things like,
 and I think there's a lot of really interesting angles
 to poke in terms of how much does GPT-2
 understand physical world?
 Like, you know, you read a little bit
 about fire underwater in GPT-2,
 so it's like, okay, maybe it doesn't quite understand
 what these things are.
 But at the same time, I think that you also see
 various things like smoke coming from flame
 and a bunch of these things that GPT-2, it has no body,
 it has no physical experience.
 It's just statically read data.
 And I think that the answer is like, we don't know yet.
 And these questions, though,
 we're starting to be able to actually ask them
 to physical systems, to real systems that exist,
 and that's very exciting.
 Do you think, what's your intuition?
 Do you think if you just scale language modeling,
 like significantly scale, that reasoning can emerge
 from the same exact mechanisms?
 I think it's unlikely that if we just scale GPT-2
 that we'll have reasoning in the full-fledged way.
 And I think that there's like,
 you know, the type signature's a little bit wrong, right?
 That like, there's something we do with,
 that we call thinking, right?
 Where we spend a lot of compute,
 like a variable amount of compute,
 to get to better answers, right?
 I think a little bit harder, I get a better answer.
 And that that kind of type signature
 isn't quite encoded in a GPT, right?
 GPT will kind of like, it's spent a long time,
 and it's like evolutionary history,
 baking in all this information,
 getting very, very good at this predictive process.
 And then at runtime, I just kind of do one forward pass,
 and I'm able to generate stuff.
 And so, you know, there might be small tweaks
 to what we do in order to get the type signature right.
 For example, well, you know,
 it's not really one forward pass, right?
 You know, you generate symbol by symbol,
 and so maybe you generate like a whole sequence of thoughts,
 and you only keep like the last bit or something.
 But I think that at the very least,
 I would expect you have to make changes like that.
 Yeah, just exactly how we, you said, think,
 is the process of generating thought by thought
 in the same kind of way, like you said, keep the last bit,
 the thing that we converge towards.
 Yep.
 And I think there's another piece, which is interesting,
 which is this out of distribution generalization, right?
 That like thinking somehow lets us do that, right?
 That we haven't experienced a thing,
 and yet somehow we just kind of keep refining
 our mental model of it.
 This is again, something that feels tied
 to whatever reasoning is.
 And maybe it's a small tweak to what we do,
 maybe it's many ideas and we'll take as many decades.
 Yeah, so the assumption there,
 generalization out of distribution
 is that it's possible to create new ideas.
 It's possible that nobody's ever created any new ideas,
 and then with scaling GPT-2 to GPT-20,
 you would essentially generalize to all possible thoughts
 that us humans can have.
 Just to play devil's advocate.
 I mean, how many new story ideas
 have we come up with since Shakespeare, right?
 Yeah, exactly.
 It's just all different forms of love and drama and so on.
 Okay, not sure if you read Biddle Lesson,
 a recent blog post by Ray Sutton.
 Yep, I have.
 He basically says something that echoes some of the ideas
 that you've been talking about,
 which is he says the biggest lesson that can be read
 from so many years of AI research
 is that general methods that leverage computation
 are ultimately going to ultimately win out.
 Do you agree with this?
 So basically, open AI in general,
 but the ideas you're exploring,
 well, coming up with methods, whether it's GPT-2 modeling
 or whether it's open AI-5 playing Dota,
 or a general method is better than a more fine-tuned,
 expert-tuned method.
 Yeah, so I think that, well, one thing that I think
 was really interesting about the reaction to that blog post
 was that a lot of people have read this as saying that
 compute is all that matters.
 And that's a very threatening idea, right?
 And I don't think it's a true idea either, right?
 It's very clear that we have algorithmic ideas
 that have been very important for making progress
 and to really build AGI,
 you wanna push as far as you can on the computational scale,
 and you wanna push as far as you can on human ingenuity.
 And so I think you need both.
 But I think the way that you phrase the question
 is actually very good, right?
 That it's really about what kind of ideas
 should we be striving for?
 And absolutely, if you can find a scalable idea,
 you pour more data into it, it gets better.
 Like, that's the real holy grail.
 And so I think that the answer to the question,
 I think, is yes.
 That that's really how we think about it,
 and that part of why we're excited
 about the power of deep learning,
 the potential for building AGI,
 is because we look at the systems that exist
 in the most successful AI systems,
 and we realize that you scale those up,
 they're gonna work better.
 And I think that that scalability
 is something that really gives us hope
 for being able to build transformative systems.
 So I'll tell you, this is partially an emotional,
 you know, a thing that,
 a response that people often have is
 compute is so important for state of the art performance.
 You know, individual developers,
 maybe a 13 year old sitting somewhere in Kansas
 or something like that, you know,
 they're sitting, they might not even have a GPU,
 or may have a single GPU, a 1080 or something like that.
 And there's this feeling like,
 well, how can I possibly compete
 or contribute to this world of AI if scale is so important?
 So if you can comment on that,
 and in general, do you think we need to also
 in the future focus on democratizing
 compute resources more,
 or as much as we democratize the algorithms?
 Well, so the way that I think about it is that
 there's this space of possible progress, right?
 There's a space of ideas and sort of systems
 that will work, that will move us forward.
 And there's a portion of that space,
 and to some extent,
 an increasingly significant portion of that space
 that does just require massive compute resources.
 And for that, that I think that
 the answer is kind of clear,
 and that part of why we have the structure that we do
 is because we think it's really important
 to be pushing the scale and to be building
 these large clusters and systems.
 But there's another portion of the space
 that isn't about the large scale compute,
 that are these ideas that,
 and again, I think that for the ideas
 to really be impactful and really shine,
 that they should be ideas that if you scale them up,
 would work way better than they do at small scale.
 But that you can discover them
 without massive computational resources.
 And if you look at the history of recent developments,
 you think about things like the GAN or the VAE,
 that these are ones that I think you could come up with them
 without having, and in practice,
 people did come up with them
 without having massive, massive computational resources.
 Right, I just talked to Ian Goodfellow,
 but the thing is,
 the initial GAN produced pretty terrible results, right?
 So only because it was in a very specific,
 only because they're smart enough to know
 that this is quite surprising
 and generate anything that they know.
 Do you see a world, or is that too optimistic and dreamer-like
 to imagine that the compute resources
 are something that's owned by governments
 and provided as a utility?
 Actually, to some extent, this question reminds me
 of a blog post from one of my former professors at Harvard,
 this guy, Matt Welsh, who was a systems professor.
 I remember sitting in his tenure talk, right,
 and he had literally just gotten tenure.
 He went to Google for the summer
 and then decided he wasn't going back to academia, right?
 And that kind of in his blog post,
 he makes this point that, look, as a systems researcher,
 that I come up with these cool system ideas, right,
 and I kind of build a little proof of concept,
 and the best thing I can hope for
 is that the people at Google or Yahoo,
 which was around at the time,
 will implement it and actually make it work at scale, right?
 That's like the dream for me, right?
 I built the little thing
 and they turned it into the big thing that's actually working.
 And for him, he said, I'm done with that.
 I want to be the person who's actually doing building
 and deploying, and I think that there's
 a similar dichotomy here, right?
 I think that there are people
 who really actually find value,
 and I think it is a valuable thing to do,
 to be the person who produces those ideas, right,
 who builds the proof of concept.
 And yeah, you don't get to generate
 the coolest possible GAN images,
 but you invented the GAN, right?
 And so there's a real trade-off there,
 and I think that's a very personal choice,
 but I think there's value in both sides.
 So do you think creating AGI,
 something or some new models,
 we would see echoes of the brilliance,
 even at the prototype level?
 So you would be able to develop those ideas
 without scale, the initial seeds?
 So take a look at, you know,
 I always like to look at examples that exist, right?
 Look at real precedent.
 And so take a look at the June 2018 model
 that we released, that we scaled up to turn into GPT-2.
 And you can see that at small scale,
 it set some records, right?
 This was the original GPT.
 We actually had some cool generations.
 They weren't nearly as amazing
 and really stunning as the GPT-2 ones,
 but it was promising, it was interesting.
 And so I think it is the case
 that with a lot of these ideas,
 that you see promise at small scale.
 But there is an asterisk here, a very big asterisk,
 which is sometimes we see behaviors that emerge
 that are qualitatively different
 from anything we saw at small scale.
 And that the original inventor of whatever algorithm
 looks at and says, I didn't think it could do that.
 This is what we saw in Dota, right?
 So PPO was created by John Schulman,
 who's a researcher here.
 And with Dota, we basically just ran PPO
 at massive, massive scale.
 And there's some tweaks in order to make it work,
 but fundamentally it's PPO at the core.
 And we were able to get this long-term planning,
 these behaviors to really play out on a time scale
 that we just thought was not possible.
 And John looked at that and was like,
 I didn't think it could do that.
 That's what happens when you're at three orders
 of magnitude more scale than you tested at.
 Yeah, but it still has the same flavors of,
 at least echoes of the expected billions.
 Although I suspect with GPT, it's scaled more and more.
 You might get surprising things.
 So yeah, you're right.
 It's interesting.
 It's difficult to see how far an idea will go
 when it's scaled.
 It's an open question.
 Well, so to that point with Dota and PPO,
 I mean, here's a very concrete one, right?
 It's actually, one thing that's very surprising about Dota
 that I think people don't really pay that much attention to
 is the degree of generalization out of distribution
 that happens, right?
 That you have this AI that's trained against other bots
 for its entirety, the entirety of its existence.
 Sorry to take a step back.
 Can you talk through a story of Dota,
 a story of leading up to opening AI-5 and that past,
 and what was the process of self-play
 and so on of training on that?
 Yeah, yeah, yeah.
 So with Dota. What is Dota?
 Yeah, Dota is a complex video game.
 And we started training, we started trying to solve Dota
 because we felt like this was a step towards the real world
 relative to other games like Chess or Go, right?
 Those very cerebral games where you just kind of have
 this board of very discreet moves.
 Dota starts to be much more continuous time
 that you have this huge variety of different actions,
 that you have a 45 minute game with all these different units
 and it's got a lot of messiness to it
 that really hasn't been captured by previous games.
 And famously, all of the hard-coded bots for Dota
 were terrible, right?
 Just impossible to write anything good for it
 because it's so complex.
 And so this seemed like a really good place
 to push what's the state of the art in reinforcement learning.
 And so we started by focusing on the one versus one version
 of the game and we're able to solve that.
 We're able to beat the world champions and the learning,
 the skill curve was this crazy exponential, right?
 And it was like constantly we were just scaling up
 that we were fixing bugs and that you look at the skill curve
 and it was really a very, very smooth one.
 This is actually really interesting to see how that
 human iteration loop yielded very steady exponential progress.
 And to one side note, first of all,
 it's an exceptionally popular video game.
 The side effect is that there's a lot of incredible
 human experts at that video game.
 So the benchmark that you're trying to reach is very high.
 And the other, can you talk about the approach
 that was used initially and throughout training
 these agents to play this game?
 Yep, and so the approach that we used is self-play.
 And so you have two agents that don't know anything.
 They battle each other.
 They discover something a little bit good
 and now they both know it.
 And they just get better and better and better
 without bound.
 And that's a really powerful idea, right?
 That we then went from the one versus one version of the game
 and scaled up to five versus five, right?
 So you think about kind of like with basketball
 where you have this like team sport
 and you need to do all this coordination
 and we were able to push the same idea,
 the same self-play to really get to the professional level
 at the full five versus five version of the game.
 And the things I think are really interesting here
 is that these agents, in some ways,
 they're almost like an insect-like intelligence, right?
 Where they have a lot in common
 with how an insect is trained, right?
 Insect kind of lives in this environment
 for a very long time, or the ancestors of this insect
 have been around for a long time and had a lot of experience
 that gets baked into this agent.
 And it's not really smart in the sense of a human, right?
 It's not able to go and learn calculus,
 but it's able to navigate its environment extremely well.
 It's able to handle unexpected things in the environment
 that it's never seen before pretty well.
 And we see the same sort of thing with our Dota bots,
 right, that they're able to, within this game,
 they're able to play against humans,
 which is something that never existed
 in its evolutionary environment,
 totally different play styles from humans versus the bots.
 And yet it's able to handle it extremely well.
 And that's something that I think was very surprising to us,
 was something that doesn't really emerge
 from what we've seen with PPO at smaller scale, right?
 And the kind of scale we're running this stuff at was,
 I could say like 100,000 CPU cores,
 running with like hundreds of GPUs,
 it was probably about something like hundreds of years
 of experience going into this bot every single real day.
 And so that scale is massive,
 and we start to see very different kinds of behaviors
 out of the algorithms that we all know and love.
 And Dota, you mentioned, beat the world expert 1v1.
 And then you weren't able to win 5v5 this year
 at the best players in the world.
 So what's the comeback story?
 First of all, talk through that,
 that was an exceptionally exciting event.
 And what's the following months in this year look like?
 Yeah, yeah, so well, one thing that's interesting
 is that we lose all the time.
 Because we play-
 Who's we here?
 The Dota team at OpenAI,
 we play the bot against better players
 than our system all the time,
 or at least we used to, right?
 Like, the first time we lost publicly
 was we went up on stage at the International,
 and we played against some of the best teams in the world,
 and we ended up losing both games,
 but we gave them a run for their money, right?
 That both games were kind of 30 minutes, 25 minutes,
 and they went back and forth, back and forth,
 back and forth.
 And so I think that really shows
 that we're at the professional level,
 and that kind of looking at those games,
 we think that the coin could have gone a different direction,
 and we could have had some wins,
 and so that was actually very encouraging for us.
 And you know, it's interesting
 because the International was at a fixed time, right?
 So we knew exactly what day we were going to be playing,
 and we pushed as far as we could, as fast as we could.
 Two weeks later, we had a bot that had an 80% win rate
 versus the one that played at TI.
 So the march of progress, you know,
 you should think of as a snapshot
 rather than as an end state.
 And so in fact, we'll be announcing our finals pretty soon.
 I actually think that we'll announce our final match
 prior to this podcast being released.
 So there should be, we'll be playing
 against the world champions.
 And you know, for us, it's really less about,
 like the way that we think about what's upcoming
 is the final milestone,
 the final competitive milestone for the project, right?
 That our goal in all of this
 isn't really about beating humans at Dota.
 Our goal is to push the state of the art
 in reinforcement learning, and we've done that, right?
 And we've actually learned a lot from our system
 and that we have, you know,
 I think a lot of exciting next steps that we wanna take.
 And so, you know, kind of as a final showcase
 of what we built, we're going to do this match.
 But for us, it's not really the success or failure to see,
 you know, do we have the coin flip go
 in our direction or against.
 Where do you see the field of deep learning
 heading in the next few years?
 Where do you see the work and reinforcement learning
 perhaps heading, and more specifically with OpenAI,
 all the exciting projects that you're working on,
 what does 2019 hold for you?
 Massive scale.
 Scale.
 I will put an atris on that and just say, you know,
 I think that it's about ideas plus scale.
 You need both.
 So that's a really good point.
 So the question in terms of ideas,
 you have a lot of projects that are exploring
 different areas of intelligence.
 And the question is, when you think of scale,
 do you think about growing the scale
 of those individual projects,
 or do you think about adding new projects?
 And sorry, if you were thinking about adding new projects,
 or if you look at the past,
 what's the process of coming up with new projects
 and new ideas?
 Yep.
 So we really have a life cycle of project here.
 So we start with a few people
 just working on a small scale idea,
 and language is actually a very good example of this.
 There was really, you know, one person here
 who was pushing on language for a long time.
 I mean, then you get signs of life, right?
 And so this is like, let's say, you know,
 with the original GPT,
 we had something that was interesting.
 And we said, okay, it's time to scale this, right?
 It's time to put more people on it,
 put more computational resources behind it.
 And then we just kind of keep pushing and keep pushing.
 And the end state is something that looks like
 Dota or robotics,
 where you have a large team of, you know,
 10 or 15 people that are running things at very large scale,
 and that you're able to really have material engineering
 and, you know, sort of machine learning science
 coming together to make systems that work
 and get material results
 that just would have been impossible otherwise.
 So we do that whole life cycle.
 We've done it a number of times, you know,
 typically end to end,
 it's probably two years or so to do it.
 I know the organization has been around for three years,
 so maybe we'll find that we also have
 longer life cycle projects,
 but, you know, we will work up to those.
 We have, so one team that we were actually just starting,
 Ilya and I are kicking off a new team
 called the Reasoning Team,
 and that this is to really try to tackle
 how do you get neural networks to reason.
 And we think that this will be a long term project
 and it's one that we're very excited about.
 In terms of reasoning, super exciting topic,
 what kind of benchmarks,
 what kind of tests of reasoning do you envision?
 What would, if you set back with whatever drink
 and you would be impressed that this system
 is able to do something, what would that look like?
 Theorem proving.
 Theorem proving.
 So some kind of logic and especially mathematical logic.
 I think so, right?
 And I think that there's kind of other problems
 that are dual to theorem proving in particular.
 You know, you think about programming,
 you think about even like security analysis of code,
 that these all kind of capture
 the same sorts of core reasoning
 and being able to do some out of distribution generalization.
 It would be quite exciting if OpenAI reasoning team
 was able to prove that P equals NP.
 That would be very nice.
 It would be very, very, very exciting, especially
 if it turns out that P equals NP,
 that'll be interesting too.
 Hey, but it would be ironic and humorous.
 So what problem stands out to you
 as the most exciting and challenging impact
 for the work for us as a community in general
 and for OpenAI this year?
 You mentioned reasoning.
 I think that's a heck of a problem.
 Yeah, so I think reasoning is an important one.
 I think it's gonna be hard to get good results in 2019.
 You know, again, just like we think about the life cycle,
 takes time.
 I think for 2019, language modeling seems to be
 kind of on that ramp, right?
 It's at the point that we have a technique that works.
 We wanna scale 100X, 1000X, see what happens.
 Awesome.
 Do you think we're living in a simulation?
 I think it's hard to have a real opinion about it.
 You know, it's actually interesting.
 I separate out things that I think can have like,
 you know, yield materially different predictions
 about the world from ones that are just kind of,
 you know, fun to speculate about.
 And I kind of view simulation as more like,
 is there a flying teapot between Mars and Jupiter?
 Like, maybe, but it's a little bit hard to know
 what that would mean for my life.
 So there is something actionable.
 So some of the best work OpenAI has done
 is in the field of reinforcement learning.
 And some of the success of reinforcement learning
 come from being able to simulate the problem
 you're trying to solve.
 So do you have a hope for reinforcement,
 for the future of reinforcement learning
 and for the future of simulation?
 Like whether we're talking about autonomous vehicles
 or any kind of system, do you see that scaling
 to be able to simulate systems and hence,
 be able to create a simulator that echoes our real world
 and proving once and for all,
 even though you're denying it,
 that we're living in a simulation?
 I feel like there's two separate questions, right?
 So, you know, kind of at the core there of like,
 can we use simulation for self-driving cars?
 Take a look at our robotic system, Dactyl, right?
 That was trained in simulation using the Dota system,
 in fact, and it transfers to a physical robot.
 And I think everyone looks at our Dota system,
 they're like, okay, it's just a game.
 How are you ever gonna escape to the real world?
 And the answer is, well, we did it with the physical robot
 that no one could program.
 And so I think the answer is simulation goes a lot further
 than you think if you apply the right techniques to it.
 Now there's a question of, you know,
 are the beings in that simulation gonna wake up
 and have consciousness?
 I think that one seems a lot harder to, again, reason about.
 I think that, you know, you really should think about like,
 where exactly does human consciousness come from
 in our own self-awareness?
 And, you know, is it just that like,
 once you have like a complicated enough neural net,
 do you have to worry about the agents feeling pain?
 And I think there's like interesting speculation to do there,
 but, you know, again, I think it's a little bit hard
 to know for sure.
 Well, let me just keep with the speculation.
 Do you think to create intelligence, general intelligence,
 you need one, consciousness, and two, a body?
 Do you think any of those elements are needed,
 or is intelligence something that's orthogonal to those?
 I'll stick to the kind of like,
 the non-grand answer first, right?
 So the non-grand answer is just to look at, you know,
 what are we already making work?
 You'll get GPT-2, a lot of people would have said
 that to even get these kinds of results,
 you need real-world experience, you need a body,
 you need grounding.
 How are you supposed to reason about any of these things?
 How are you supposed to like even kind of know
 about smoke and fire and those things
 if you've never experienced them?
 And GPT-2 shows that you can actually go way further
 than that kind of reasoning would predict.
 So I think that in terms of, do we need consciousness?
 Do we need a body?
 It seems the answer is probably not, right?
 That we could probably just continue to push
 kind of the systems we have.
 They already feel general.
 They're not as competent or as general
 or able to learn as quickly as an AGI would,
 but you know, they're at least like kind of proto-AGI
 in some way and they don't need any of those things.
 Now, let's move to the grand answer,
 which is, you know, if our neural nets conscious already,
 would we ever know, how can we tell, right?
 And you know, here's where the speculation starts
 to become, you know, at least interesting or fun
 and maybe a little bit disturbing,
 depending on where you take it.
 But it certainly seems that when we think about animals,
 that there's some continuum of consciousness.
 You know, my cat, I think is conscious in some way, right?
 You know, not as conscious as a human.
 And you could imagine that you could build
 a little consciousness meter, right?
 You point at a cat, it gives you a little reading.
 Point at a human, it gives you much bigger reading.
 Well, what would happen if you pointed one of those
 at a donor neural net?
 And if you're training in this massive simulation,
 do the neural nets feel pain?
 You know, it becomes pretty hard to know
 that the answer is no.
 And it becomes pretty hard to really think about
 what that would mean if the answer were yes.
 And it's very possible, you know, for example,
 you could imagine that maybe the reason
 that humans have consciousness is because
 it's a convenient computational shortcut, right?
 If you think about it, if you have a being
 that wants to avoid pain, which seems pretty important
 to survive in this environment and wants to like,
 you know, eat food, then that maybe the best way
 of doing it is to have a being that's conscious, right?
 That, you know, in order to succeed in the environment,
 you need to have those properties,
 and how are you supposed to implement them?
 And maybe this consciousness's way of doing that.
 If that's true, then actually maybe we should expect
 that really competent reinforcement learning agents
 will also have consciousness.
 But you know, it's a big if, and I think there are
 a lot of other arguments they can make in other directions.
 I think that's a really interesting idea
 that even GPT-2 has some degree of consciousness.
 That's something, it's actually not as crazy to think about,
 it's useful to think about as we think about
 what it means to create intelligence of a dog,
 intelligence of a cat, and the intelligence of a human.
 So last question, do you think we will ever fall in love,
 like in the movie Her, with an artificial intelligence system
 or an artificial intelligence system
 falling in love with a human?
 I hope so. If there's any better way to end it,
 is on love, so Greg, thanks so much for talking today.
 Thank you for having me.