Wojciech Zaremba: OpenAI Codex, GPT-3, Robotics, and the Future of AI | Lex Fridman Podcast #215
agi ai ai podcast artificial intelligence artificial intelligence podcast codex consciousness copilot deep learning gpt ilya sutskever intelligence lex ai lex fridman lex jre lex mit lex podcast machine learning meditation mit ai openai productivity sam altman sleep wojciech zaremba
Wojciech Zaremba is a co-founder of OpenAI. Please support this podcast by checking out our sponsors:
- Paperspace: https://gradient.run/lex to get $15 credit
- Indeed: https://indeed.com/lex to get $75 credit
- Blinkist: https://blinkist.com/lex and use code LEX to get 25% off premium
- Grammarly: https://grammarly.com/lex to get 20% off premium
- Eight Sleep: https://www.eightsleep.com/lex and use code LEX to get special savings

EPISODE LINKS:
Wojciech's Twitter: https://twitter.com/woj_zaremba
Wojciech's Website: https://wojzaremba.com/
OpenAI's Website: https://openai.com/

PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
Full episodes playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
1:18 - The Fermi paradox
8:20 - Systems of government
10:57 - Life, intelligence, and consciousness
18:10 - GPT language model
20:23 - Engineering consciousness
24:40 - Is there an algorithm for intelligence?
32:02 - Neural networks and deep learning
44:53 - Human reward functions
49:47 - Love is part of the human condition
52:14 - Expanding our circle of empathy
56:19 - Psychedelics and meditation
1:07:46 - Ilya Sutskever
1:15:03 - How does GPT work?
1:24:56 - AI safety
1:31:42 - OpenAI Codex
1:45:15 - Robotics
1:54:32 - Developing self driving cars and robots 
2:05:23 - What is the benchmark for intelligence?
2:08:25 - Will we spend more time in virtual reality?
2:10:39 - AI Friendships
2:20:09 - Sleep
2:22:43 - Generating good ideas
2:29:08 - Advice for young people
2:33:52 - Getting started with machine learning
2:37:05 - What is beauty?
2:40:56 - Death
2:47:44 - Meaning of life

SOCIAL:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Reddit: https://reddit.com/r/lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman
 The following is a conversation with Wojciech Zaremba, co-founder of OpenAI, which is one of the top organizations in the world doing artificial intelligence research and development. Wojciech is the head of language and cogeneration teams, building and doing research on GitHub Copilot, OpenAI Codex, and GPT-3 and who knows, four, five, six, N, and N plus one. And he also previously led OpenAI's robotic efforts. These are incredibly exciting projects to me that deeply challenge and expand our understanding of the structure and nature of intelligence. The 21st century, I think, may very well be remembered for a handful of revolutionary AI systems and their implementations. GPT, Codex, and applications of language models and transformers in general to the language and visual domains may very well be at the core of these AI systems. To support this podcast, please check out our sponsors. They're listed in the description. This is the Lex Friedman podcast, and here is my conversation with Wojciech Zaremba. You mentioned that Sam Altman asked about the Fermi paradox, and the people at OpenAI had really sophisticated, interesting answers. So that's when you knew this is the right team to be working with. So let me ask you about the Fermi paradox, about aliens. Why have we not found overwhelming evidence for aliens visiting Earth? I don't have a conviction in the answer, but rather kind of probabilistic perspective on what might be a, let's say, possible answer. It's also interesting that the question itself even can touch on your typical question of what's the meaning of life? Because if you assume that we don't see aliens because they destroy themselves, that kind of upweighs the focus on making sure that we won't destroy ourselves. But at the moment, the place where I am, actually with my belief, and these things also change over the time, is I think that we might be alone in the universe, which actually makes life more, or let's say consciousness, life more kind of valuable. And that means that we should more appreciate it. Have we always been alone? So what's your intuition about our galaxy, our universe? Is it just sprinkled with graveyards of intelligent civilizations, or is life, intelligent life, truly unique? At the moment, my belief that it is unique, but I would say I could also, there was some footage released with UFO objects, which makes me actually doubt my own belief. Yeah, I can tell you one crazy answer that I have heard. So apparently when you look actually at the limits of computation, you can compute more if the temperature of the universe would drop down. So one of the things that aliens might want to do if they are truly optimizing to maximize amount of compute, which maybe can lead to more, let's say simulations or so, it's instead of wasting current entropy of the universe, because we by living, we are actually somewhat wasting entropy, then you can wait for the universe to cool down such that you have more computation. That's kind of a funny answer. I'm not sure if I believe in it, but that would be one of the reasons why you don't see aliens. It's also possible to some people say that maybe there is not that much point in actually going to other galaxies if you can go inwards. So there is no limits of what could be an experience if we could connect machines to our brains, while there are still some limits if we want to explore the universe. Yeah, there could be a lot of ways to go inwards too. Once you figure out some aspect of physics we haven't figured out yet, maybe you can travel to different dimensions. I mean, travel in three-dimensional space may not be the most fun kind of travel. There may be just a huge amount of different ways to travel, and it doesn't require a spaceship going slowly in 3D space to space time. It also feels like one of the problems is that speed of light is low and the universe is vast. And it seems that actually most likely if we want to travel very far, then we would instead of actually sending spaceships with humans that weight a lot, we would send something similar to what Yuri Miller is working on. These are like a huge sail, which is at first powered, or there is a shot of laser from an air and it can propel it to quarter of speed of light. And sail itself contains a few grams of equipment. And that might be the way to actually transport matter through universe. But then when you think what would it mean for humans, it means that we would need to actually put their 3D printer and 3D print a human on other planet. I don't know, play them YouTube or let's say, or like a 3D print like huge human right away or maybe a womb or so. With our current techniques of archeology, if a civilization was born and died long enough ago on Earth, we wouldn't be able to tell. And so that makes me really sad. And so I think about Earth in that same way, how can we leave some remnants if we do destroy ourselves? How can we leave remnants for aliens in the future to discover? Like here's some nice stuff we've done. Like Wikipedia and YouTube, do we have it like in a satellite orbiting Earth with a hard drive? Like how do we say, how do we back up human civilization for the good parts or all of it is good parts so that it can be preserved longer than our bodies can? It's a difficult question. It also requires the difficult acceptance of the fact that we may die. And if we die, we may die suddenly as a civilization. So let's see, I think it kind of depends on the cataclysm. We have observed in other parts of the universe that birds of gamma rays, these are high energy rays of light that actually can apparently kill entire galaxy. So there might be actually nothing even to, nothing to protect us from it. I'm also, and I'm looking actually at the past civilizations. So it's like Aztecs or so, they disappear from the surface of the Earth and one can ask why is it the case? And the way I'm thinking about it is, you know, that definitely they had some problem that they couldn't solve. And maybe there was a flood and all of a sudden they couldn't drink, there was no potable water and they all died. And I think that so far, the best solution to such a problems is, I guess, technology. So, I mean, if they would know that you can just boil water and then drink it after, then that would save their civilization. And even now when we look actually at the current pandemic, it seems that once again, actually science comes to rescue and somehow science increases size of the action space. And I think that's a good thing. Yeah, but nature has a vastly larger action space. But still it might be a good thing for us to keep on increasing action space. Well, okay. Looking at past civilizations, yes. But looking at the destruction of human civilization, perhaps expanding the action space will add actions that are easily acted upon, easily executed, and as a result, destroy us. So let's see. I was pondering why actually even we have negative impact on the globe, because if you ask every single individual, they would like to have clean air, they would like healthy planet, but somehow it actually is not the case that as a collective, we are not going in this direction. I think that there exists very powerful system to describe what we value. That's capitalism. It assigns actually monetary values to various activities. At the moment, the problem in the current system is that there are some things which we value. There is no cost assigned to it. So even though we value clean air, or maybe we also value lack of destruction on the internet or so, at the moment, these quantities, companies, corporations can pollute them for free. So in some sense, I wished, or like that's I guess purpose of politics, to align the incentive systems. And we are kind of maybe even moving in this direction. The first issue is even to be able to measure the things that we value, then we can actually assign the monetary value to them. Yeah, and that's so it's getting the data and also probably through technology, enabling people to vote and to move money around in a way that is aligned with their values. And that's very much a technology question. So like having one president and Congress and voting that happens every four years or something like that, that's a very outdated idea. There could be some technological improvements to that kind of idea. So I'm thinking from time to time about these topics, but it also feels to me that it's a little bit like, it's hard for me to actually make correct predictions what is the appropriate thing to do. I extremely trust Sam Altman, our CEO on these topics here. Okay, I'm more on the side of being, I guess, naive hippie. That's your life philosophy. Well, I think self-doubt and I think hippie implies optimism. Those two things are pretty good way to operate. I mean, still it is hard for me to actually understand how the politics works or exactly how the things would play out. And Sam is really excellent with it. What do you think is rarest in the universe? You said we might be alone. What's hardest to build is another engineering way to ask that, life, intelligence or consciousness. So like you said that we might be alone, which is the thing that's hardest to get to? Is it just the origin of life? Is it the origin of intelligence? Is it the origin of consciousness? So let me at first explain you my kind of mental model, what I think is needed for life to appear. So I imagined that at some point there was this primordial soup of amino acids and maybe some proteins in the ocean. And some proteins were turning into some other proteins through reaction. And you can almost think about this cycle of what turns into what. There is a graph essentially describing which substance turns into some other substance. And essentially life means that all of a sudden in the graph has been created a cycle such that the same thing keeps on happening over and over again. That's what is needed for life to happen. And in some sense you can think almost that you have this gigantic graph and it needs a sufficient number of edges for the cycle to appear. Then from perspective of intelligence and consciousness, my current intuition is that they might be quite intertwined. First of all, it might not be that it's like a binary thing that you have intelligence or consciousness. It seems to be more a continuous component. Let's see, if we look for instance on the event networks recognizing images, people are able to show that the activations of these networks correlate very strongly with activations in visual cortex of some monkeys. The same seems to be true about language models. Also, if you for instance look, if you train agent in 3D world, at first it barely recognizes what is going on. Over time it recognizes foreground from background. Over time it knows where there is a foot and it just follows it. Over time it actually starts having a 3D perception. It is possible for instance to look inside of the head of an agent and ask what would it see if it looks to the right. The crazy thing is initially when the agents are barely trained these predictions are pretty bad. Over time they become better and better. You can still see that if you ask what happens when the head is turned by 360° for some time they think that the different thing appears and then at some stage they understand actually that the same thing is supposed to appear. So they get an understanding of 3D structure. It's also very likely that they have inside some level of symbolic reasoning, like particularly symbols for other agents. So when you look at DOTA agents, they collaborate together and they have some anticipation of if they would win battle. They have some expectations with respect to other agents. I might be too much anthropomorphizing how things look for me. But the fact that they have a symbol for other agents makes me believe that at some stage as they are optimizing for skills they would have also a symbol to describe themselves. This is a very useful symbol to have. This particularity I would call it self-consciousness or self-awareness. Still it might be different from the consciousness. I guess the way how I'm understanding the word consciousness, let's say the experience of drinking a coffee or let's say experience of being a butt, that's the meaning of the word consciousness. It doesn't mean to be awake. It might be also somewhat related to memory and recurrent connections. So it's kind of like if you look at anesthetic drugs, they essentially disturb brainwaves such that maybe memory is not formed. So there's a lessening of consciousness when you do that. Correct. And so that's the one way to intuit what is consciousness. There's also kind of another element here. It could be that it's this kind of self-awareness module that you described. Plus the actual subjective experience is a storytelling module that tells us a story about what we're experiencing. The crazy thing. So let's say, I mean, in meditation they teach people not to speak story inside of the head. There is also some fraction of population who doesn't have actually narrator. I know people who don't have a narrator and they have to use external people in order to kind of solve tasks that require internal narrator. So it seems that it's possible to have that experience without the talk. What are we talking about when we talk about the internal narrator? Is that the voice when you're reading a book? Yeah, I thought that that's what you are referring to. Well, I was referring more on not an actual voice. I meant like there's some kind of subjective experience feels like it's fundamentally about storytelling to ourselves. It feels like the feeling is a story that is much simpler abstraction than the raw sensory information. So there feels like it's a very high level abstraction that is useful for me to feel like entity in this world. Most useful aspect of it is that because I'm conscious, I think there's an intricate connection to me not wanting to die. So like it's a useful hack to really prioritize not dying. Those seem to be somewhat connected. So I'm telling the story of like it's rich. Feels like something to be me and the fact that me exists in this world. I want to preserve me. So that makes it a useful agent hack. So I will just refer maybe to that first part, as you said, about that kind of story of describing who you are. I was thinking about it even. Obviously, I like thinking about consciousness. I like thinking about AI as well. And I'm trying to see analogies of these things in AI. What would it correspond to? So OpenAI trained a model called GPT, which can generate pretty amusing text on arbitrary topic. And one way to control GPT is by putting into prefix at the beginning of the text some information. What would be the story about? You can have even chat with GPT by saying that the chat is with Lex or Elon Musk or so. And GPT would just pretend to be you or Elon Musk or so. And it almost feels that this story that we give ourselves to describe our life, it's almost like things that you put into context of GPT. Yeah, but the context we provide to GPT is multimodal. So GPT itself is multimodal. GPT itself hasn't learned actually from experience of single human, but from the experience of humanity. It's a chameleon. You can turn it into anything. And in some sense, by providing context, it behaves as the thing that you wanted it to be. It's interesting that people have stories of who they are. And as I said, these stories, they help them to operate in the world. But it's also interesting, I guess, various people find it out through meditation or so that there might be some patterns that you have learned when you were a kid that actually are not serving you anymore. And you also might be thinking that that's who you are. And that's actually just a story. Yes, it's a useful hack, but sometimes it gets us into trouble. It's a local optima. It's a local optima. You wrote that Stephen Hawking, he tweeted, Stephen Hawking asked what breathes fire into equations, which meant what makes given mathematical equations realize the physics of a universe. Similarly, I wonder what breathes fire into computation. What makes given computation conscious? OK, so how do we engineer consciousness? How do you breathe fire and magic into the machine? So it seems clear to me that not every computation is conscious. I mean, you can, let's say, just keep on multiplying one matrix over and over again and maybe gigantic matrix. You can put a lot of computation. I don't think it would be conscious. So in some sense, the question is, what are the computations which could be conscious? I mean, so one assumption is that it has to do purely with computation, that you can abstract away matter. Other possibilities that it's very important was the realization of computation that it has to do with some force fields or so. And they bring consciousness. At the moment, my intuition is that it can be fully abstracted away. So in case of computation, you can ask yourself, what are the mathematical objects or so that could bring such properties? So for instance, if we think about the models, AI models, what they truly try to do, or like models like GPT, is they try to predict next word or so. And this turns out to be equivalent to compressing text. And because in some sense, compression means that you learn the model of reality and you have just to remember where are your mistakes. The better you are in predicting. And in some sense, when we look at our experience, also when you look, for instance, at the car driving, you know in which direction it will go. You are good in prediction. And it might be the case that the consciousness is intertwined with compression. It might be also the case that self-consciousness has to do with compressor trying to compress itself. So I was just wondering what are the objects in mathematics or computer science, which are mysterious, that could have to do with consciousness. And then I thought, you see in mathematics, there is something called Gadel theorem, which means if you have sufficiently complicated mathematical system, it is possible to point the mathematical system back on itself. In computer science, there is something called halting problem. It's somewhat similar construction. So I thought that if we believe that under assumption that consciousness has to do with compression, then you could imagine that as you keep on compressing things, then at some point it actually makes sense for the compressor to compress itself. Metacompression. Consciousness is metacompression. That's an idea. And in some sense, you know, the crazy... Thank you. But do you think if we think of a touring machine, a universal touring machine, can that achieve consciousness? So is there something beyond our traditional definition of computation that's required? So it's a specific computation. And I said this computation has to do with compression. And the compression itself, maybe other way of putting it is like you are internally creating the model of reality. In order that you try inside to simplify reality in order to predict what's going to happen. And that also feels somewhat similar to how I think actually about my own conscious experience. So clearly I don't have access to reality. The only access to reality is through, you know, cable going to my brain. And my brain is creating a simulation of reality and I have access to the simulation of reality. Are you by any chance aware of the Hutter Prize, Marcus Hutter? He made this prize for compression of Wikipedia pages. And there's a few qualities to it. One, I think, has to be perfect compression, which makes... I think that little quirk makes it much less applicable to the general task of intelligence. Because it feels like intelligence is always going to be messy. Like perfect compression feels like it's not the right goal, but it's nevertheless a very interesting goal. So for him, intelligence equals compression. And so the smaller you make the file, given a large Wikipedia page, the more intelligent the system has to be. Yeah, that makes sense. So you can make perfect compression if you store errors. And I think that actually what he meant is you have algorithm plus errors. By the way, Hutter is a... he was a PhD advisor of Chen Leck, who is a DeepMind co-founder. Yeah, yeah. So there's an interesting... and now he's a DeepMind. There's an interesting network of people. He's one of the people that I think seriously took on the task of what would an AGI system look like? I think for the longest time, the question of AGI was not taken seriously or rather rigorously. And he did just that. Like mathematically speaking, what would the model look like? If you remove the constraints of it having to be... having to have a reasonable amount of memory, reasonable amount of running time complexity, computation time, what would it look like? And essentially, it's a half math, half philosophical discussion of how would a reinforcement learning type of framework look like for an AGI? So he developed the framework even to describe what's optimal with respect to reinforcement learning. Like there is a theoretical framework, which is, as you said, under assumption, there is infinite amount of memory and compute. There was actually one person before his name is Solomonov. Hutter extended Solomonov work to reinforcement learning, but there exists a theoretical algorithm, which is optimal algorithm to build intelligence. And I can actually explain you the algorithm. Yes, let's go. Let's go. Let's go. So the task itself, you can... Can I just pause how absurd it is for brain in a skull trying to explain the algorithm for intelligence? Just go ahead. It is pretty crazy. It is pretty crazy that, you know, the brain itself is actually so small and it can't ponder. How to design algorithms that optimally solve the problem of intelligence. OK. So what's the algorithm? So let's see. So first of all, the task itself is described as you have infinite sequence of zeros and ones. OK, you read n bits and you are about to predict n plus 1 bit. So that's the task. And you could imagine that every task could be casted as such a task. So if, for instance, you have images and labels, you can just turn every image into a sequence of zeros and ones, then label, you concatenate labels. And that's actually... You could start by having training data first and then afterwards you have test data. So theoretically any problem could be casted as a problem of predicting zeros and ones on this infinite tape. So let's say you read already n bits and you want to predict n plus 1 bit. And I will ask you to write every possible program that generates these n bits. OK, so and you can have... You choose programming language. It can be Python or C++. And the difference between programming languages might be there is a difference by constant. Asymptotically, your predictions will be equivalent. So you read n bits, you enumerate all the programs that produce these n bits in their output. And then in order to predict n plus 1 bit, you actually weight the programs according to their length. There is some specific formula how you weight them. And then the n plus 1 bit prediction is the prediction from each of these programs according to that weight. Like statistically. So the smaller the program, the more likely you are to pick its output. So that algorithm is grounded in the hope or the intuition that the simple answer is the right one. It is a formalization of it. It also means like if you would ask the question after how many years would sun explode. You can say it's more likely the answer is 2 to some power because it's a shorter program. Yeah. I don't have a good intuition about how different the space of short programs are from the space of large programs. Like what is the universe where short programs run things? So as I said, the things have to agree with n bits. So even if you have you need to start. If you have a very short program and they're like a still some has if it's not perfect with prediction of n bits, you have to start errors. What are the errors? And that gives you the full program that agrees on n bits. Oh, so you don't agree perfectly with the n bits and you store. That's like a longer, longer programs, like longer program, because it contains these extra bits of errors. That's fascinating. What's what's your intuition about the programs that are able to do cool stuff like intelligence and consciousness? Are they perfectly like is it is there if then statements in them? So like is there a lot of exceptions that they're storing? So you could imagine if there would be tremendous amount of statements, then they wouldn't be that short. In case of neural networks, you could imagine that what happens is they when you start with an uninitialized neural network, it stores internally many possibilities how the how the problem can be solved. And HDD is kind of magnifying some some some paths which are slightly similar to the correct answers. So it's kind of magnifying correct programs and in some sense HDD is the search algorithm in the program space. And the program space is represented by, you know, kind of the wiring inside of the neural network. And there's like an insane number of ways how the features can be computed. Let me ask you the high level basic question that's not so basic. What is deep learning? Is there a way you'd like to think of it that is different than like a generic? Textbook definition? The thing that I hinted just a second ago is maybe that closest to how I'm thinking these days about deep learning. So now the statement is neural networks can represent some programs. Seems that various modules that we are actually adding up to are like, you know, we we want networks to be deep because we we want multiple steps of the computation. And and deep learning provides the way to represent space of programs, which is searchable and it's searchable with stochastic gradient descent. So we have an algorithm to search over a humongous number of programs and gradient descent kind of bubbles up the things that tend to give correct answers. So a neural network with a with fixed weights that's optimized. Do you think of that as a single program? So there is a work by Christopher Ola, where he so he works on interpretability of neural networks and he was able to to identify inside of the neural network, for instance, a detector of a wheel for a car or the detector of a mask for a car. And then he was able to separate them out and assemble them together using a simple program for the detector for a car detector. That's like if you think of traditionally defined programs, that's like a function within a program that this particular neural network was able to find. And you can tear that out just like you can copy and paste from Stack Overflow that. So any program is a composition of smaller programs. Yeah, I mean, the nice thing about the neural networks is that it allows the things to be more fuzzy than in case of programs. In case of programs, you have this like a branching this way or that way. And the neural networks, they have an easier way to to be somewhere in between or to share things. What to use the most beautiful or surprising idea in deep learning and the utilization of these neural networks, which, by the way, for people who are not familiar, neural networks is a bunch of what would you say? It's inspired by the human brain. There's neurons. There's connection between those neurons. There's inputs and there's outputs and there's millions or billions of those neurons. And the learning happens by adjusting the weights on the edges that connect these neurons. Thank you for giving definition that I supposed to do it. But I guess you have enough empathy to listeners to actually know that that might be useful. No, that's like some asking Plato of like, what is the meaning of life? He's not going to answer. You're being philosophical and deep and quite profound talking about the space of programs, which is just very interesting. But also people are just not familiar with the hell we're talking about when we talk about deep learning. Anyway, sorry. What is the most beautiful or surprising idea to you in in in all the time you've worked at deep learning? And you worked on a lot of fascinating projects, applications of neural networks. It doesn't have to be big and profound. It can be a cool trick. Yeah, I mean, I'm thinking about the trick, but like it's still amusing to me that it works at all. Yeah. That let's say that the extremely simple algorithm stochastic gradient descent, which is something that I would be able to derive on the piece of paper to high school student when put at the scale of thousands of machines actually can create the behaviors which we called kind of human like behaviors. So in general, any application is stochastic gradient descent to neural networks is is amazing to you. So that or is there a particular application in natural language reinforcement learning? And also, what do you attribute that success to? Is it just scale? What profound insight can we take from the fact that the thing works for gigantic sets of variables? I mean, the interesting thing is this algorithms, they were invented decades ago and people actually gave up on the idea. Yeah. And, you know, back then they thought that we need profoundly different algorithms and they spent a lot of cycles on very different algorithms. And I believe that, you know, we have seen that various innovations that say like transformer or dropout or so they can, you know, pass the help. But it's also remarkable to me that this algorithm from 60s or so or I mean, you can even say that the gradient descent was invented by Leibniz in, I guess, 18th century or so. That actually is the core of learning. In the past, people are it's almost like out of the maybe an ego. People are saying that it cannot be the case that such a simple algorithm is there, you know, could solve complicated problems. So they were in search for the other algorithms. And as I'm saying, like, I believe that actually we are in the game where there is there are actually, frankly, three levers. There is compute, there are algorithms and there is data. And if we want to build intelligent systems, we have to pull all three levers and they are actually multiplicative. And it's also interesting. So yes, is it only compute? People internally, they did the studies to determine how much gains they were coming from different levers. And so far we have seen that more gains came from compute than algorithms. But also we are in the world that in case of compute, there is a kind of, you know, exponential increase in funding. And at some point, it's impossible to invest more. It's impossible to, you know, invest $10 trillion. We are speaking about the, let's say, all taxes in the US. But you're talking about money. There could be innovation in the compute. That's that's true as well. So, I mean, there are like a few pieces. So one piece is human brain is an incredible supercomputer. And they're like it has 100 trillion parameters. Or like if you try to count various quantities in the brain, they're like a neuron synapses, that small number of neurons. There is a lot of synapses. It's unclear even how to map synapses to parameters of neural networks. But it's clear that there are many more. So it might be the case that our networks are still somewhat small. It also might be the case that they are more efficient than brain or less efficient by some by some huge factor. I also believe that there will be like, you know, at the moment we are at the stage that these neural networks, they require 1000x or like a huge factor of more data than humans do. And it will be a matter of there will be algorithms that vastly decrease sample complexity. I believe so. But the place where we are heading today is there are domains which contains million X more data. And even though computers might be 1000 times slower than humans in learning, that's not the problem. For instance, I believe that it should be possible to create superhuman therapies by and there are even simple steps of doing it. And, you know, that the core reason is there is just machine will be able to read way more transcripts of therapies and then it should be able to speak simultaneously with many more people. And it should be possible to optimize it all in parallel. Well, there's now you're touching on something I deeply care about and think is way harder than we imagine. What's the goal of a therapist? What's the goal of therapists? So, OK, so one goal. Now, this is terrifying to me, but there's a lot of people that contemplate suicide, suffer from depression, and they could significantly be helped with therapy. And the idea that an AI algorithm might be in charge of that, it's like a life and death task. It's the stakes are high. So one goal for a therapist, whether human or AI, is to prevent suicide ideation to prevent suicide. How do you achieve that? So let's see. So to be clear, I don't think that the current models are good enough for such a task because it requires insane amount of understanding and patting. And the models are far from this place. But it's. But do you think that understanding empathy, that signal is in the data? I think there is some signal in the data. Yes. I mean, there are plenty of transcripts of conversations and it is possible to it is possible from it to understand personalities. It is possible from it to understand if conversation is friendly, amicable, antagonistic. It is, I believe that, you know, given the fact that the models that we train now, they can. They can have they are chameleons that they can have any personality. They might turn out to be better in understanding personality of other people than anyone else. And they are pathetic to be empathetic. Yeah. Interesting. But I wonder if there's some level of multiple modalities required to be able to be empathetic of the human experience, whether language is not enough to understand death, to understand fear, to understand childhood trauma, to understand wit and humor required when you're dancing with a person who might be depressed or suffering, both humor and hope and love and all those kinds of things. So there's another underlying question, which is self supervised versus supervised. So can you get that from the data by just reading a huge number of transcripts? I actually so I think that reading huge number of transcripts is a step one. It's like the same way as you cannot learn to dance if just from YouTube by watching it. You have to actually try it out yourself. And so I think that here that's a similar situation. I also wouldn't deploy the system in the high stakes situations right away, but kind of see gradually where it goes. And obviously, initially, it would have to go hand in hand with humans. But at the moment, we are in the situation that actually there is many more people who actually would like to have a therapy or speak with someone than there are therapies out there. I can you know, I was so fundamentally, I was thinking, what are the things that can vastly increase people well-being? Therapy is one of them. Being meditation is other one. I guess maybe human connection is the third one. And I guess pharmacologically, it's also possible. Maybe direct brain stimulation or something like that. But these are pretty much options out there. Then let's say the way I'm thinking about the AGI endeavor is by default, that's an endeavor to increase the amount of wealth. And I believe that we can vastly increase the amount of wealth for everyone. And simultaneously, so I mean, there are like two endeavors that make sense to me. One is like essentially increase the amount of wealth. And the second one is increase overall human well-being. And those are coupled together. I would say these are different topics. One can help another. And you know, therapist is a funny word because I see friendship and love as therapy. I mean, so therapist broadly defined is just friendship as a friend. So like therapist has a very kind of clinical sense to it. But what is human connection? You're like not to get all Camus and Dostoevsky on you, but you know, life is suffering and we draw. We seek connection with other humans as we desperately try to make sense of this world in a deep, overwhelming loneliness that we feel inside. So I think connection has to do with understanding. And I think that almost like a lack of understanding causes suffering. If you speak with someone and you feel ignored, that actually causes pain. If you are feeling deeply understood that actually they might not even tell you what to do in life, but like a pure understanding. Or just being heard. Understanding is a kind of, it's a lot, you know, just being heard. Feel like you're being heard. Like somehow that's alleviation temporarily of the loneliness. That if somebody knows you're here with their body language, with the way they are, with the way they look at you, with the way they talk, you feel less alone for a brief moment. Yeah, very much agree. So I thought in the past about somewhat similar question to yours, which is what is love rather what is connection? And obviously I think about these things from AI perspective. What would it mean? So I said that intelligence has to do with some compression, which is more or less like you can say almost understanding of what is going around. It seems to me that other aspect is there seem to be reward functions and you can have a reward for food, for maybe human connection, for let's say warmth, sex, and so on. And it turns out that the various people might be optimizing slightly different reward functions. They essentially might care about different things. And in case of love, at least the love between two people, you can say that the boundary between people dissolves to such extent that they end up optimizing each other reward functions. Yeah, that's interesting. Celebrate the success of each other. Yeah, in some sense, I would say love means helping others to optimize their reward functions, not your reward functions, not the things that you think are important, but the things that the person cares about. You try to help them to optimize it. So love is, if you think of two reward functions, you just it's a condition. You combine them together. Yeah, pretty much. Maybe like with a weight and it depends like the dynamic of the relationship. Yeah, I mean, you could imagine that if you are fully optimizing someone's reward function without yours, then maybe you are creating codependency or something like that. I'm not sure what's the appropriate weight. But the interesting thing is I even think that the individual person, we ourselves, we are actually less of a unified insight. So, for instance, if you look at the donut on the one level, you might think, oh, this looks tasty. I would like to eat it. On the other level, you might tell yourself, I shouldn't be doing it because I want to gain muscles. So and you know, you might do it regardless, kind of against yourself. So it seems that even within ourselves, they're almost like a kind of intertwined personas. And I believe that the self love means that the love between all these personas, which also means being able to love yourself when you are angry or stressed or so. Combining all those reward functions of the different selves you have. Yeah, and accepting that they are there. Like, you know, often people, they have a negative self-talk or they say, I don't like when I'm angry. And like I tried to imagine, tried to imagine if there would be like a small baby Lex, like a five years old, angry. And then they are like, you shouldn't be angry. Like, stop being angry. Yeah. But like instead, actually, you want the Lex to come over, give him a hug and is like, I say, it's fine. OK, you can be angry as long as you want. And then he would stop. Or maybe not. Or maybe not. But you cannot expect it even. Yeah. But still, that doesn't explain the why of love. Like, why is love part of the human condition? Why is it useful to combine the reward functions? It seems like that doesn't I mean, I don't think reinforcement learning frameworks can give us answers to why even even the Hutter framework has an objective function that's static. So we came to existence as a consequence of evolutionary process. And in some sense, the purpose of evolution is survival. And then this complicated optimization objective baked into us, let's say compression, which might help us operate in the real world. And it baked into us various reward functions. Yeah. Then to be clear, at the moment, we are operating in the regime, which is somewhat out of distribution where the even evolution optimized us. It's almost like love is a consequence of cooperation that we've discovered is useful. Correct. In some way, it's even the case if you. I just love the idea that love is like the out of distribution. Or it's not out of distribution. It's like as you said, it evolved for cooperation. Yes. And I believe that the cop like in some sense, cooperation ends up helping each of us individually. So it makes sense evolutionary. And there is in some sense, and you know, love means there is this dissolution of boundaries that you have a shared reward function. And we evolved to actually identify ourselves with larger groups. So we can identify ourselves with a family. We can identify ourselves with a country to such extent that people are willing to give away their life for country. So there is we are wired, actually, even for love. And at the moment, I guess the maybe it would be somewhat more beneficial if you will, if we would identify ourselves with all the humanity as a whole. So you can clearly see when people travel around the world, when they run into person from the same country, they say, oh, which city are and all this? Like all the sudden they find all these similarities. They they they find some they befriended those folks earlier than others. So there is like a sense, some sense of the belonging. And I would say I think it would be overall good thing to the world for people to move towards. I think it's even called open individualism, move toward the mindset of a larger and larger groups. So the challenge there, that's a beautiful vision. And I share it to expand that circle of empathy, that circle of love towards the entirety of humanity. But then you start to ask, well, where do you draw the line? Because why not expand it to other conscious beings? And then finally, for our discussion, something I think about is why not expand it to AI systems? Like we we start respecting each other when the person, the entity on the other side has the capacity to suffer, because then we develop a capacity to sort of empathize. And so I could see AI systems that are interacting with humans more and more having conscious like displays. So like they display consciousness through language and through other means. And so then the question is like, well, is that consciousness? Because they're acting conscious. And so, you know, the reason we don't like torturing animals is because they look like they're suffering when they're tortured. And if AI looks like it's suffering when it's tortured, how is that not requiring of the same kind of empathy from us and respect and rights that animals do and other humans do? I think it requires empathy as well. I mean, I would like, I guess, us or humanity or so make a progress in understanding what consciousness is, because I don't want just to be speaking about that philosophy, but rather actually make a scientific. To have a, like, you know, there was a time that people thought that there is a force of life and the things that have this force, they are alive. And I think that there is actually a path to understand exactly what consciousness is. And in some sense, it might require essentially putting probes inside of a human brain. What Neuralink does. So the goal there, I mean, there's several things with consciousness that make it a real discipline, which is one is rigorous measurement of consciousness. And then the other is the engineering of consciousness, which may or may not be related. You could also run into trouble, like, for example, in the United States, the Department of Department of Transportation and a lot of different places put a value on human life. I think DOT's values nine million dollars per person. Sort of in that same way, you can get into trouble if you put a number on how conscious a being is, because then you can start making policy. If a cow is point one or like 10 percent as conscious as a human, then you can start making calculations and might get you into trouble. But then again, that might be a very good way to do it. I would like to move to that place that actually we have scientific understanding what consciousness is. And then we'll be able to actually assign value. And I believe that there is even the path for the experimentation in it. So, you know, we said that, you know, you could put the probes inside of the brain. There is actually a few other things that you could do with devices like Neuralink. So you could imagine that the way even to measure if a system is conscious is by literally just plugging into the brain. I mean, that seems that's kind of easy. But the plugging into the brain and asking person if they feel that their consciousness expanded. And this direction, of course, has some issues. You can say, you know, if someone takes a psychedelic drug, they might feel that their consciousness expanded, even though that drug itself is not conscious. Right. So like you can't fully trust the self-report of a person saying their their consciousness is expanded or not. Let me ask you a little bit about psychedelics. There's been a lot of excellent research on different psychedelics, psilocybin, MDMA, even DMT, drugs in general, marijuana, too. What do you think psychedelics do to the human mind? It seems they take the human mind to some interesting places. Is that just a little hack, a visual hack, or is there some profound expansion of the mind? So let's see. I don't believe in magic. I believe in I believe in in science, in in causality. Still, let's say. And then, as I said, like, I think that the brain. That our subjective experience of reality is a. We live in the simulation run by our brain and the simulation that our brain runs, they can be very pleasant or very hellish. Drugs, they are changing some hyperparameters of the simulation. It is possible thanks to change of these hyperparameters to actually look back on your experience and even see that the given things that we took for granted, they are changeable. So they allow to have an amazing perspective. There is also, for instance, the fact that after DMT, people can see the full movie inside of their head gives me further belief. That brain can generate that full movie that the brain is actually. Learning the model of reality to such extent that it tries to predict what's going to happen next. Yeah, very high resolution so it can replay realities at extremely high resolution. Yeah, it's also kind of interesting to me that somehow there seems to be some similarity between. These drugs and meditation itself, and I actually started even these days to think about meditation as a psychedelic. Do you practice meditation? I practice meditation. I mean, I once few times on the retreats and it feels after like after second or third day of meditation. There is a there is almost like a sense of, you know, tripping. What is the meditation retreat entail? So you wake up early in the morning and you meditate for extended period of time. And alone. Yeah, so it's optimized even though there are other people, it's optimized for isolation. So you don't speak with anyone. You don't actually look into other people eyes. And you know you sit on the chair. Say the personal meditation tells you to focus on the breath. So you try to put all the all attention into breathing and breathing in and breathing out. And the crazy thing is that as you focus attention like that, after some time. There's some starts coming back like some memories that you're completely forgotten. It almost feels like that you have a mailbox and then you know you are just like archiving email one by one. And at some point, at some point there is like an amazing feeling of getting to mailbox zero, zero emails. And it's very pleasant. It's kind of it's it's it's crazy to me that that once. You resolve these inner store stories or like inner traumas. Then once there is nothing left, the default state of human mind is extremely peaceful and happy. Extreme like some sense it it feels that the. It feels at least to me in that way how when I was a child that I can look at any object and it's very beautiful. I have a lot of curiosity about the simple things. And that's where the usual meditation takes me. Are you what are you experiencing? Are you just taking in simple sensory information and are just enjoying the rawness of that sensory information? So there's no there's no memories or all that kind of stuff. You're just enjoying being. Yeah, pretty much. I mean, still there is a that it's it's thoughts are slowing down. Sometimes they pop up, but it's also somehow the extended meditation takes you to the space that they are way more friendly. You're way more positive. There is also this this thing that we've extracted. It almost feels that the. It almost feels that the we are constantly getting a little bit of a reward function and we are just spreading this reward function on various activities. But if you stay still for extended period of time, it kind of accumulates, accumulates, accumulates and. There is a there is a sense there is a sense that some point it passes some threshold and it feels as. Drop is falling into kind of ocean of love and bliss. And that's like a this is like a very pleasant and that's I'm saying like. That corresponds to the subjective experience. Some people, I guess, in spiritual community, they describe it that that's the reality. And I would say I believe that there are like all sorts of subjective experience that one can have. And I believe that, for instance, meditation might take you to the subjective experiences, which are very pleasant, collaborative. And I would like a word to move toward a more collaborative place. Yeah, I would say that's very pleasant and I enjoy doing stuff like that. I wonder how that maps to your mathematical model of love with the reward function combining a bunch of things. It seems like our life then is we're just we have this reward function and we're accumulating a bunch of stuff in it with weights. It's like like multi objective. And what meditation is, is you just remove them, remove them until the weight on one or just a few is very high. And that's where the pleasure comes from. Yeah. So something similar, how I'm thinking about it. So I told you that there is this like that there is a story of who you are. And I think almost about it as a, you know, text prepended to GPT. Yeah. And some people refer to it as ego. OK, it's like a story. Who, who, who you are. OK, so ego is the prompt for GPT3. Yeah, yes, yes. And that's description of you. And then with meditation, you can get to the point that actually you experience things without the prompt and you experience things like as they are. You are not biased or the description how they supposed to be. That's very pleasant. And then with respect to the reward function, it's possible to get to the point that there is the solution of self. And therefore you can say that the you're having your or like a your brain attempts to simulate the reward function of everyone else or like everything that's that there is this like a love, which feels like a oneness with everything. And that's also, you know, very beautiful, very pleasant. At some point you might have a lot of altruistic thoughts during that moment. And then the self always comes back. How would you recommend if somebody is interested in meditation, like a big thing to take on as a project? Would you recommend a meditation retreat? How many days? What kind of thing would you recommend? I think that actually retreat is the way to go. It almost feels that the say meditation is a psychedelic, but when you take it in the small dose, you might barely feel it. Once you get the high dose, actually you're going to feel it. So even cold turkey, if you haven't really seriously meditated for prolonged period of time, just go to a retreat. Yeah, how many days? How many days? Start the weekend one weekend. So like two, three days. And it's interesting that first or second day, it's hard. And at some point it becomes easy. There's a lot of seconds in a day. How hard is the meditation retreat? Just sitting there in a chair. So the thing is actually, it literally just depends on your own framing. Like if you are in the mindset that you are waiting for it to be over or you are waiting for Nirvana to happen, it will be very unpleasant. And in some sense, even the difficulty, it's not even in the lack of being able to speak with others. Like you're sitting there, your legs will hurt from sitting. In terms of like the practical things, do you experience kind of discomfort, like physical discomfort of just sitting, like your your butt being numb, your legs being sore, all that kind of stuff? Yes, you experience it. And then they teach you to observe it rather than it's like the crazy thing is you at first might have a feeling toward trying to escape it. And that becomes very apparent that that's extremely unpleasant. And then you just just observe it. And at some point it just becomes it just is. It's like I remember we've Ilya told me some time ago that he takes a cold shower and his mindset of taking a cold shower was to embrace suffering. Yeah, excellent. I do the same. Is this your style? Yes, my style. I like this. So my style is actually I also sometimes take cold showers. It is purely observing how the water goes through my body, like purely being present, not trying to escape from there. Yeah. And I would say then it actually becomes pleasant. It's not like, well, that's interesting. I'm also that me. That's that's the way to deal with anything really difficult, especially in the physical space is to observe it to say it's pleasant. Hmm. It's a I would use a different word. You're you're accepting of the full beauty of reality, I would say, because say pleasant. But yeah, in some sense, it is pleasant. That's the only way to deal with a cold shower is to become an observer and to find joy in it. Same with like really difficult physical exercise or like running for a really long time endurance events anytime you're any kind of pain. I think the only way to survive it is not to resist it, just observe it. You mentioned Ilja Ilja Setskever is very he's our chief scientist, but also he's very close friend of mine. He co-founded OpenAI with you. I've spoken with him a few times. He's brilliant. I really enjoy talking to him. His mind, just like yours, works in fascinating ways. Both of you are not able to define deep learning simply. What's it like having him as somebody you have technical discussions with on in space machine learning, deep learning, AI, but also life? What's it like when these two agents get into a self-play situation in a room? What's it like collaborating with him? So I believe that we have extreme respect to each other. So I love Ilja's insight, both like I guess about consciousness, life, AI. But in terms of the it's interesting to me because you're a brilliant thinker in the space of machine learning, like intuition, like digging deep in what works, what doesn't, why it works, why it doesn't. And so is Ilja. I'm wondering if there's interesting deep discussions you've had with him in the past or disagreements that were very productive. So I can say I also understood over the time where are my strengths. So obviously we have plenty of AI discussions and I myself have plenty of ideas, but I consider Ilja one of the most prolific AI scientists in the entire world. And I think that I realized that maybe my super skill is being able to bring people to collaborate together, that I have some level of empathy that is unique in AI world. And that might come from either meditation, psychedelics, or let's say I read just hundreds of books on this topic. And I also went through a journey of I develop all sorts of algorithms. So I think that maybe I can, that's my super human skill. Ilja is one of the best AI scientists, but then I'm pretty good in assembling teams. And I'm also not holding to people like I'm growing people and then people become managers at OpenAI. I grew many of them like a research managers. You find places where you're excellent and he finds like his deep scientific insights is where he is and you find ways you can, the puzzle pieces fit together. Correct. Ultimately, for instance, let's say Ilja, he doesn't manage people. That's not what he likes or so. I like hanging out with people. By default, I'm an extrovert and I care about people. Okay. Cool. So that fits perfectly together. But I mean, I also just like your intuition about various problems in machine learning. He's definitely one I really enjoy. I remember talking to him about something I was struggling with, which is coming up with a good model for pedestrians, for human beings across the street in the context of autonomous vehicles. And he immediately started to formulate a framework within which you can evolve a model for pedestrians, like through self play, all that kind of mechanisms. The depth of thought on a particular problem, especially problems he doesn't know anything about is fascinating to watch. It makes you realize like, yeah, the limits that the human intellect might be limitless. I was just impressive to see a descendant of APE come up with clever ideas. Yeah, I mean, so even in the space of deep learning, when you look at various people, there are people who invented some breakthroughs once, but there are very few people who did it multiple times. And you can think if someone invented it once, that might be just a sheer luck. And if someone invented it multiple times, you know, if a probability of inventing it once is one over a million, then probability of inventing it twice or three times would be one over a million square or to the power of three, which would be just impossible. So it literally means that it's given that it's not the luck. And Ilya is one of these few people who have a lot of these inventions in his arsenal. It also feels that, for instance, if you think about folks like Gauss or Euler, you know, at first they read a lot of books, and then they did thinking, and then they figure out math. And that's how it feels with Ilya. You know, at first he read stuff and then like he spent his thinking cycles. That's a really good way to put it. When I talk to him, I see thinking. He's actually thinking. He makes me realize that there's like deep thinking that the human mind can do. Like most of us are not thinking deeply. Like you really have to put a lot of effort to think deeply. Like I have to really put myself in a place where I think deeply about a problem. It takes a lot of effort. It's like an airplane taking off or something. You have to achieve deep focus. He's just he's what is it? His brain is like a vertical takeoff in terms of airplane analogy. So it's interesting. But I mean, Cal Newport talks about this as ideas of deep work. It's you know, most of us don't work much at all in terms of like like deeply think about particular problems, whether it's a math, engineering, all that kind of stuff. You want to go to that place often. And that's real hard work. And some of us are better than others at that. So I think that the big piece has to do with actually even engineering your environment such that it's conducive to that. So see, both Ilya and I on the frequent basis, we kind of disconnect ourselves from the world in order to be able to do extensive amount of thinking. Yes. So Ilya usually he just leaves iPad at hand. He loves his iPad. And for me, I'm even sometimes, you know, just going for a few days to different location to Airbnb. I'm turning off my phone and there is no access to me. Yeah. And that's extremely important for me to be able to actually just formulate new thoughts, to do deep work rather than to be reactive. And the older I am, the more of these like random tasks are at hand. Before I go on to that thread, let me return to our friend GPT. Let me ask you another ridiculously big question. Can you give an overview of what GPT-3 is or like you say in your Twitter bio, GPT N plus one, how it works and why it works? So GPT-3 is a human-goose neural network. Let's assume that we know what is neural network by definition. And it is trained on the entire Internet just to predict next words. So let's say it sees part of the article and the only task that it has at hand is to say what would be the next word. What would be the next word? And it becomes really exceptional at the task of figuring out what's the next word. So you might ask, why would this be an important task? Why would it be important to predict what's the next word? And it turns out that a lot of problems can be formulated as a text completion problem. So GPT is purely learning to complete the text. You could imagine, for instance, if you are asking a question, who is the president of the United States, then GPT can give you an answer to it. It turns out that many more things can be formulated this way. You can format text in the way that you have sent us in English. You make it even look like some content of a website elsewhere, which would be teaching people how to translate things between languages. So it would be EN colon, text in English, FR colon. And then you ask the model to continue. And it turns out that such a model is predicting translation from English to French. The crazy thing is that this model can be used for way more sophisticated tasks. So you can format text such that it looks like a conversation between two people. And that might be a conversation between you and Elon Musk. And because the model read all the text about Elon Musk, it will be able to predict Elon Musk words as it would be Elon Musk. It will speak about colonization of Mars, about sustainable future and so on. And it's also possible to even give arbitrary personality to the model. You can say, here is a conversation with a friendly AI bot. And the model will complete the text as a friendly AI bot. So, I mean, how do I express how amazing this is? So just to clarify a conversation, generating a conversation between me and Elon Musk, it wouldn't just generate good examples of what Elon would say. It would get the syntax all correct. So like interview style, it would say like Elon colon and Lex colon. It's not just like inklings of semantic correctness. It's like the whole thing. Grammatical, syntactic, semantic. It's just really, really impressive generalization. Yeah, I mean, I also want to provide some caveats. So it can generate a few paragraphs of coherent text. But as you go to longer pieces, it actually goes off the rails. If you try to write a book, it won't work out this way. What way does it go off the rails, by the way? Is there interesting ways in which it goes off the rails? What falls apart first? So the model is trained on all the existing data that is out there, which means that it is not trained on its own mistakes. So, for instance, if it would make a mistake, then I can give you an example. So let's say I have a conversation with a model pretending that is Elon Musk. And then I start putting some, I start actually making up things which are not factual. I would say Twitter. But I got you. Sorry. Yeah. Okay. I don't know. I would say that Elon is my wife and the model will just keep on carrying it on. As if it's true. Yes. And in some sense, if you would have a normal conversation with Elon, he would be what the fuck. Yeah, there would be some feedback between. So the model is trained on things that humans have written, but through the generation process, there's no human in the loop feedback. Correct. That's fascinating. Makes sense. So it's magnifies like the errors get magnified and magnified. And it's also interesting. I mean, first of all, humans have the same problem is just that we will make fewer errors and magnify the errors slower. I think that actually what happens with humans is if you have a wrong belief about the world as a kid, then very quickly you will learn that it's not correct because they are grounded in reality and they are learning from your new experience. But do you think the model can correct itself to won't it through the power of the representation? And so the absence of Elon Musk being your wife, information on the Internet won't to correct itself. There won't be examples like that. So the errors will be subtle at first. Saddle at first. And in some sense, you can also say that the data that is not out there is a data which would represent how the human learns. And maybe model would be learned, trained on such a data, then it would be better off. How intelligent is GPT-3, do you think? Like when you think about the nature of intelligence, it seems exceptionally impressive. But then if you think about the big AGI problem, is this footsteps along the way to AGI? So let's see. It seems that intelligence itself is there are multiple axis of it. And I would expect that the the systems that we are building, they might end up being superhuman on some axis and subhuman on some other axis. It would be surprising to me on all axis simultaneously, they would become superhuman. Of course, people ask this question. Is GPT a spaceship that that would take us to the moon or are we building a ladder to heaven that we are just building bigger and bigger ladder? And we don't know in some sense which one of these two works. Which one is better? I'm trying to I like Stairway to Heaven as a good song, so I'm not exactly sure which one is better. But you're saying like the spaceship to the moon is actually effective? Correct. So people who criticize GPT, they say, you guys just building a taller ladder and it will never reach the moon. And at the moment, I would say the way I'm thinking is this is like a scientific question. And I'm also in heart, I'm a builder creator and like I'm thinking, let's try out. Let's see how far it goes. And so far we see constantly that there is a progress. Yeah. So what do you think GPT4, GPT5, GPTn plus one will there be a phase shift, like a transition to a place where we'll be truly surprised? Then again, like GPT3 is already very like truly surprising. The people that criticize GPT3 as it's there as a what is it ladder to heaven, I think too quickly get accustomed to how impressive it is that the prediction of the next word can achieve such depth of semantics, accuracy, syntax, grammar and semantics. Do you think GPT4 and 5 and 6 will continue to surprise us? I mean, definitely there will be more impressive models. There is a question, of course, if there will be a phase shift. And also even the way I'm thinking about these models is that when we build these models, you know, we see some level of the capabilities, but we don't even fully understand everything that the model can do. And actually one of the best things to do is to allow other people to probe the model to even see what is possible. Hence the using GPT as an API and opening it up to the world. Yeah, I mean, so when I'm thinking from perspective of, obviously, various people that have concerns about AGI, including myself. And then when I'm thinking from perspective, what's the strategy even to deploy these things to the world? The one strategy that I have seen many times working is the iterative deployment that you deploy slightly better versions and you allow other people to criticize you. So you actually are tried out. You see where are their fundamental issues. And it's almost you don't want to be in that situation that you are holding into powerful system and there's like a huge overhang. Then you deploy it and it might have a random chaotic impact on the world. So you actually want to be in the situation that they are gradually deploying systems. I asked this question of Ilya. Let me ask you this question. I've been reading a lot about Stalin and power. If you're in possession of a system that's like AGI, that's exceptionally powerful. Do you think your character and integrity might become corrupted? Like famously power corrupts and absolute power corrupts. Absolutely. So I believe that you want at some point to work toward distributing the power. I think that you want to be in the situation that actually AGI is not controlled by a small number of people, but essentially by a larger collective. So the thing is that requires a George Washington style move. In the ascent to power, there's always a moment when somebody gets a lot of power and they have to have the integrity and the moral compass to give away that power. That humans have been good and bad throughout history at this particular step. And I wonder, I wonder if we blind ourselves, for example, between nations, a race towards AI race between nations. We might blind ourselves and justify to ourselves the development of AI without distributing the power because we want to defend ourselves against China, against Russia, that kind of logic. And I wonder how we design governance mechanisms that prevent us from becoming power hungry and in the process destroying ourselves. So let's see. I have been thinking about this topic quite a bit, but I also want to admit that once again, I actually want to rely way more on Sam Altman on it. He wrote an excellent blog on how even to distribute wealth. And he proposed in his blog to tax equity of the companies rather than profit and to distribute it. And this is an example of a Washington move. I guess I personally have insane trust in Sam. I mean, he already spent plenty of money running universal basic income project that gives me, I guess, maybe some level of trust to him. But I also, I guess, love him as a friend. Yeah, I wonder because we're sort of summoning a new set of technologies. I wonder if we'll be cognizant like you're describing the process of open AI, but it could also be at other places like in the US government. Right. Both China and the US are now. Full steam ahead on autonomous weapons systems development. And that's really worrying to me because in the framework of something being a national security danger or a military danger, you can do a lot of pretty dark things that blind our moral compass. And I think AI will be one of those things. In some sense, the mission and the work you're doing at open AI is like the counterbalance to that. So you want to have more open AI and less autonomous weapons systems. I like this statement. To be clear, this is interesting and I'm thinking about it myself. But this is a place that I put my trust actually in Sam's hands because it's extremely hard for me to reason about it. Yeah. I mean, one important statement to make is it's good to think about this. Yeah, no question about it. No question about it. Even like low level, quote unquote, engineer. Like there's such a. I remember I programmed a car, RC car. They went really fast, like 30, 40 miles an hour. And I remember I was like sleep deprived, so I programmed it pretty crapily and it like the code froze. So it's doing some basic computer vision and it's going around on track, but it's going full speed. And there's a bug in the code that the car just went. It didn't turn. It went straight full speed and smash into the wall. And I remember thinking. The seriousness with which you need to approach the design of artificial intelligence systems and the programming of artificial intelligence systems is high because the consequences are high. That little car smashing into the wall. For some reason, I immediately thought of like an algorithm that controls nuclear weapons having the same kind of bug. And so like the lowest level engineer and the CEO of a company all need to have the seriousness in approaching this problem and thinking about the worst case consequences. So I think that is true. I mean, what I also recognize in myself and others even asking this question is that it evokes a lot of fear. And fear itself ends up being actually quite debilitating. The place where I arrived at the moment. Might sound cheesy or so, but it's almost to. Build things out of love rather than fear. I can focus on how I can maximize the value, how the systems that I'm building might be useful. I'm not saying that the fear doesn't exist out there and it totally makes sense to minimize it. But I don't want to be working because I'm scared. I want to be working out of passion, out of curiosity, out of the looking forward for the positive future. With the definition of love arising from rigorous practice of empathy. So not just like your own conception of what is good for the world, but always listening to others. Correct. Like at the love where I'm considering reward functions of others. Others to limit to infinity is like a sum of like one to N where N is seven billion or whatever it is. Not projecting my reward functions on others. Yeah, exactly. OK, can we just take a step back to something else? Super cool, which is open air codex. Can you give an overview of what open air codex and GitHub copilot is, how it works and why the hell it works so well? So with GPT-3, we noticed that the system, you know, that system trained on all the language out there. Started having some rudimentary coding capabilities. So we're able to ask it to implement addition function between two numbers. And indeed, it can write Python or JavaScript code for that. And then we thought we might as well just go full steam ahead and try to create a system that is actually good at what we are doing every day ourselves, which is programming. We optimized models for proficiency in coding. We actually even created models that both have a comprehension of language and code and codex is API for these models. So it's first pre-trained on language and then I don't know if you can say fine tuned because there's a lot of code, but it's language and code. It's language and code. It's also optimized for various things like, let's say, low latency and so on. Codex is the API, similar to GPT-3. We expect that there will be proliferation of the potential products that can use coding capabilities. And I can speak about it in a second. Copilot is the first product developed by GitHub. So as we're building models, we wanted to make sure that these models are useful. And we work together with GitHub on building the first product. Copilot is actually as you code, it suggests you code completions. And we have seen in the past there are like various tools that can suggest how to like a few characters of the code or the line of code. The thing about Copilot is it can generate 10 lines of code. It's often the way how it works is you often write in the comment what you want to happen because people in comments, they describe what happens next. So these days when I code, instead of going to Google to search for the appropriate code to solve my problem, I say, oh, for this array, could you smooth it? And then, you know, it imports some appropriate libraries and it uses NumPy convolution or so that I was not even aware that exists. And it does the appropriate thing. So you write a comment, maybe the header of a function, and it completes the function. Of course, you don't know what is the space of all the possible small programs it can generate. What are the failure cases? How many edge cases? How many subtle errors there are? How many big errors there are? It's hard to know. But the fact that it works at all in a large number of cases is incredible. It's like it's a kind of search engine into code that's been written on the Internet. Correct. So for instance, when you search things online, then usually you get to some particular case, like if you go to Stack Overflow, people describe one particular situation and then they seek for a solution. But in the case of Copilot, it's aware of your entire context. And in context, it's, oh, these are the libraries that they are using. That's the set of the variables that is initialized. And on the spot, it can actually tell you what to do. So the interesting thing is, and we think that the Copilot is one possible product using codecs, but there is a place for many more. So internally, we tried out to create other fun products. So it turns out that a lot of tools out there, let's say Google Calendar or Microsoft Word or so, they all have internal API to build plugins around them. So there is a way in a sophisticated way to control Calendar or Microsoft Word. Today, if you want more complicated behaviors from these programs, you have to add a new button for every behavior. But it is possible to use codecs and tell, for instance, to Calendar, could you schedule an appointment with Blacks next week after 2 p.m. and it writes corresponding piece of code. And that's the thing that actually you want. So interesting. So you figure out is there's a lot of programs with which you can interact through code. And so there you can generate that code from natural language. That's fascinating. And that's somewhat like also closest to what was the promise of Siri or Alexa. So previously, all these behaviors, they were hard coded. And it seems that codecs on the fly can pick up the API of, let's say, given software. And then it can turn language into use of this API. So without hard coding, you can find it can translate to machine language. So, for example, this would be really exciting for me like for Adobe products like Photoshop, which I think ActionScript, I think there's a scripting language that communicates with them. Same with Premiere. And you could imagine that that allows even to do coding by voice on your phone. So, for instance, in the past, as of today, I'm not editing Word documents on my phone because it's just the keyboard is too small. But if I would be able to tell to my phone, make the header large, then move the paragraphs around, and that's actually what I want. So I can tell you one more cool thing, or even how I'm thinking about codecs. So if you look actually at the evolution of computers, we started with very primitive interfaces, which is a punch card. And punch card, essentially, you make holes in the plastic card to indicate zeros and ones. And during that time, there was a small number of specialists who were able to use computers. And by the way, people even suspected that there is no need for many more people to use computers. But then we moved from punch cards to, at first, assembly, then C. And these programming languages, they were slightly higher level. They allowed many more people to code. And they also led to more of a proliferation of technology. And further on, there was a jump to, say, from C++ to Java and Python. And every time it has happened, more people are able to code, and we build more technology. And it's even hard to imagine now if someone will tell you that you should write code in assembly instead of, let's say, Python or Java or JavaScript. And codecs is yet another step toward bringing computers closer to humans, such that you communicate with a computer with your own language rather than with a specialized language. And I think that it will lead to an increase of number of people who can code. Yeah, and the kind of technologies that those people will create, it's innumerable. It could be a huge number of technologies we're not predicting at all. Because that's less and less a requirement of having a technical mind, a programming mind. You're now opening it to the world of other kinds of minds, creative minds, artistic minds, all that kind of stuff. I would like, for instance, biologists who work on DNA to be able to program and not need to spend a lot of time learning it. And I believe that's a good thing to the world. And I would actually add that. So at the moment, I'm a managing codecs team and also language team. And I believe that there is like a plenty of brilliant people out there and they should apply. Oh, OK. Yeah, awesome. So what's the language in the codecs? So those are kind of their overlapping teams. It's like GPT, the raw language, and then the codecs is applied to programming. Correct. And they are quite intertwined. There are many more teams involved making these models extremely efficient and deployable. For instance, there are people who are working to make our data centers amazing. Or there are people who work on putting these models into production or even pushing it at the very limit of the scale. So all aspects from the infrastructure to the actual machine learning. So I'm just saying there are multiple teams. While the team working on codecs and language, I guess I'm directly managing them. I would love to hire more. If you're interested in machine learning, this is probably one of the most exciting problems and systems to be working on. It's pretty cool. The program synthesis, the generating of programs is very interesting. Very interesting problem that has echoes of reasoning and intelligence in it. And I think there's a lot of fundamental questions that you might be able to sneak up to by generating programs. Yeah, one more exciting thing about the programs is that, so I said that in case of language, that one of the troubles is even evaluating language. So when the things are made up, you need somehow either a human to say that this doesn't make sense or so. In case of program, there is one extra lever that we can actually execute programs and see what they evaluate. So that process might be somewhat more automated in order to improve the qualities of generations. That's fascinating. So like the wow, that's really interesting. So for language, the simulation to actually execute it as a human mind. Yeah. For programs, there is a computer on which you can evaluate it. Wow. That's a brilliant little insight that the thing compiles and runs. That's first. And second, you can evaluate, do automated unit testing. And in some sense, it seems to me that we'll be able to make a tremendous progress. We are in the paradigm that there is way more data. There is like a transcription of millions of software engineers. Yeah. So I mean, you just mean because I was going to ask you about reliability. The thing about programs is you don't know if they're going to like a program that's controlling a nuclear power plant has to be very reliable. So I wouldn't start with controlling nuclear power plant. Maybe one day. But that's not actually that's not on the current roadmap. That's not the step one. You know, it's the Russian thing. You just want to go to the most powerful destructive thing right away run by JavaScript. But I got you. So it's a lower impact. But nevertheless, what you make me realize, it is possible to achieve some levels of reliability by doing testing. And you could you could imagine that, you know, maybe there are ways for model to write even code for testing itself and so on. And there exist ways to create a feedback loops that the model could keep on improving. By writing programs that generate tests for instance, for instance. And that's how we get consciousness, because it's meta compression. That's what you're going to write. That's the comment. That's the prompt that generates consciousness. Compressor of compressors. You just write that. Do you think the code that generates consciousness will be simple? So let's see. I mean, ultimately, the core idea behind will be simple, but there will be also decent amount of engineering involved. Like in some sense, it seems that, you know, spreading these models on many machines, it's not that trivial. And we find all sorts of innovations that make our models more efficient. I believe that first models that I guess are conscious are like a truly intelligent. They will have all sorts of tricks. But then again, there's a certain argument that maybe the tricks are temporary things. Yeah, they might be temporary things. And in some sense, it's also even important to know that even the cost of a trick. So sometimes people are eager to put the trick while forgetting that there is a cost of maintenance. Or like long term cost. Long term cost or maintenance or maybe even flexibility of code to actually implement new ideas. So even if you have something that gives you 2x, but it requires, you know, 1000 lines of code, I'm not sure if it's actually worth it. So in some sense, you know, if it's five lines of code and 2x, I would take it. And we see many of this, but also, you know, that requires some level of, I guess, lack of attachment to code that we are willing to remove it. Yeah. So you led the OpenAI robotics team. Can you give an overview of the cool things you were able to accomplish? What are you most proud of? So when we started robotics, we knew that actually reinforcement learning works and it is possible to solve fairly complicated problems. Like, for instance, AlphaGo is an evidence that it is possible to build superhuman Go players. Dota2 is an evidence that it's possible to build superhuman agents playing Dota. So I asked myself a question, you know, what about robots out there? Could we train machines to solve arbitrary tasks in the physical world? And the approach was, I guess, let's pick a complicated problem that if we would solve it, that means that we made some significant progress, the domain, and then we went after the problem. So we noticed that actually the robots out there, they are kind of at the moment optimized per task. So you can have a robot that it's like if you have a robot opening a battle, it's very likely that the end factor is a battle opener. And in some sense, that's a hack to be able to solve a task, which makes any task easier. And I asked myself, so what would be a robot that can actually solve many tasks? And we concluded that human hands have such a quality that indeed they are, you know, you have five kind of tiny arms attached. Individually, they can manipulate a pretty broad spectrum of objects. So we went after a single hand like trying to solve Rubik's Cube single handed. We picked this task because we thought that there is no way to hard code it. And it's also we picked a robot on which it would be hard to hard code it. And we went after the solution such that it could generalize to other problems. And just to clarify, it's one robotic hand solving the Rubik's Cube. The hard part is in the solution to the Rubik's Cube is the manipulation of the of like having it not fall out of the hand, having it use the five baby arms to what is it like rotate different parts of the Rubik's Cube to achieve the solution. Correct. Yeah. So what was the hardest part about that? What was the approach taken there? What are you most proud of? Obviously, we have like a strong belief in reinforcement learning. And, you know, one path it is to do reinforcement learning in the real world. Other path is to the simulation. In some sense, the tricky part about the real world is at the moment our models, they require a lot of data. There is essentially no data. And I think we decided to go through the path of the simulation. And in simulation, you can have infinite amount of data. The tricky part is the fidelity of the simulation and also can you in simulation represent everything that you represent otherwise in the real world? And, you know, it turned out that that, you know, because there is lack of fidelity, it is possible to what we what we arrived at is training a model that doesn't solve one simulation, but it actually solves the entire range of simulations, which vary in terms of like what's the exactly the friction of the cube or weight or so. And the single AI that can solve all of them ends up working well with the reality. How do you generate the different simulations? So, you know, there's plenty of parameters out there. We just pick them randomly and an in simulation model just goes for thousands of years and keeps on solving Rubik's Cube in each of them. And the thing is that neural network that we used, it has a memory and as it presses, for instance, the side of the of the cube, it can sense, oh, that's actually this side was difficult to press. I should press it stronger and throughout this process kind of learns even how to how to solve this particular instance of the Rubik's Cube back even mass. It's kind of like, you know, sometimes when you go to a gym and after after bench press, you try to leave the class and you kind of forgot. And your head goes like right away because kind of you got used to maybe different weight and it takes a second to adjust. Yeah. And this kind of of a memory, the model gained through the process of interacting with the cube in the simulation. I appreciate you speaking to the audience with the bench press, all the bros in the audience probably working out right now. There's probably somebody listening to this actually doing bench press. So maybe put the bar down and pick up the water bottle and you'll know exactly what what Jack is talking about. OK. So what what was the hardest part of getting the whole thing to work? So the hardest part is at the moment when it comes to physical work, when it comes to robots, they require maintenance. It's hard to replicate them a million times. It's also it's hard to replay things exactly. I remember this situation that one guy at our company, he had like a model that performs way better than other models in solving Rubik's Cube. And, you know, we kind of didn't know what's going on, why is that? And it turned out that, you know, he was running it from his laptop that had better CPU or better, maybe local GPU as well. And because of that, there was less of a latency and the model was the same. And that actually made solving Rubik's Cube more reliable. So in some sense, there might be some subtle bugs like that when it comes to running things in the real world. Even hinting on that, you could imagine that the initial models you would like to have models, which are insanely huge neural networks, and you would like to give them even more time for thinking. And when you have these real time systems, then you might be constrained actually by the amount of latency. And ultimately, I would like to build a system that is worth for you to wait five minutes because it gives you the answer that you are willing to wait for five minutes. So latency is a very unpleasant constraint under which to operate. Correct. And also there is actually one more thing which is tricky about robots. There is actually not much data. So the data that I'm speaking about would be data of first person experience from the robot and like a gigabytes of data like that. If we would have gigabytes of data like that of robot solving various problems, it would be very easy to make a progress on robotics. And you can see that in case of text or code, there is a lot of data like a first person perspective data on writing code. Yeah. So you had this you mentioned this really interesting idea that if you were to build like a successful robotics company, so open as mission is much bigger than robotics. This is one of the one of the things you worked on. But if it was a robotics company that you wouldn't so quickly dismiss supervised learning. Correct. That you would build a robot that was perhaps what like an empty shell like dumb and they would operate under teleoperation. So you would invest. That's just one way to do it. Invest in human super like direct human control of the robots as it's learning. And over time, add more and more automation. That's correct. So let's say that's how I would build a robotics company today if I would be building robotics company, which is spent 10 million dollars or so recording human trajectories controlling a robot. After you find a thing that the robot should be doing that there's a market fit for like that you can make a lot of money with that product. Correct. Correct. Yeah. So I would record data and then I would essentially train supervised learning model on it. That might be the path today. Long term, I think that actually what is needed is to train powerful models over video. So you have seen maybe a models that can generate images like Dali and people are looking into models generating videos. They're like a body's algorithmic questions even how to do it. And it's unclear if there is enough compute for this purpose. But I suspect that the models that which would have a level of understanding of video, same as GPT has a level of understanding of text, could be used to train robots to solve tasks. They would have a lot of common sense. If one day I'm pretty sure one day there will be a robotics company by robotics company. I mean, the primary source of income is from robots that is worth over one trillion dollars. What do you think that company will do? I think sell driving cars. It's interesting because my mind went to personal robotics robots in the home. It seems like there's much more market opportunity there. I think it's very difficult to achieve. I mean, this this this might speak to something important, which is I understand self driving much better than understand robotics in the home. So I understand how difficult it is to actually solve self driving to a level, not just the actual computer vision and the control problem and just the basic problem self driving, but creating a product that would undeniably be that would cost less money. Like it would save you a lot of money, like orders of magnitude, less money that could replace Uber drivers, for example. So car sharing is autonomous. It creates a similar or better experience in terms of how quickly you get from A to B or just whatever the pleasantness of the experience, the efficiency of the experience, the value of the experience. And at the same time, the car itself costs cheaper. I think that's very difficult to achieve. I think there's a lot more low hanging fruit in the home. That could be. I also want to give you a perspective on how challenging it would be at home or like it maybe kind of depends on the exact problem that you'd be solving. Like if we're speaking about these robotic arms and hence these things, they cost tens of thousands of dollars or maybe 100 K. And, you know, maybe obviously maybe there would be economy of scale. These things would be cheaper. But actually, for any household to buy, the price would have to go down to maybe thousand bucks. Yeah, I personally think that so self-driving car provides a clear service. I don't think robots in the home, there'll be a trillion dollar company will just be all about service, meaning it will not necessarily be about like a robotic arm that helps you. I don't know, open a bottle or wash dishes or any of that kind of stuff. It has to be able to take care of that whole the therapist thing you mentioned. I think that's of course there's a line between what is a robot and what is not like doesn't really need a body. But, you know, some AI system with some embodiment, I think. So the tricky part, when you think actually what's the difficult part is when the robot has like when there is a diversity of the environment with which the robot has to interact, that becomes hard. So, you know, on one spectrum, you have industrial robots as they are doing over and over the same thing. It is possible to some extent to prescribe the movements. And with very small amount of intelligence, the movement can be repeated millions of times. There are also, you know, various pieces of industrial robots where it becomes harder and harder. Like, for instance, in case of Tesla, it might be a matter of putting a rock inside of a car. And, you know, because the rock kind of moves around, it's not that easy. It's not exactly the same every time. That's being the case that you need actually humans to do it. And while, you know, welding cars together, it's a very repetitive process. Then in case of self-driving itself, the difficulty has to do with the diversity of the environment, but still the car itself. The problem that you are solving is you try to avoid even interacting with things. You are not touching anything around because touching itself is hard. And then if you would have in the home robot that, you know, has to touch things and like if these things, they change the shape, if there is a huge variety of things to be touched, then that's difficult. If you are speaking about the robot, which there is, you know, head that is smiling in some way with cameras, that it doesn't, you know, touch things, that's relatively simple. OK, so to both agree and to push back. So you're referring to touch like soft robotics, like the actual touch. But I would argue that you could formulate just basic interaction between like non-contact interaction is also a kind of touch. And that might be very difficult to solve. That's the basic this not disagreement, but that's the basic open question to me with self-driving cars and this agreement with Elon, which is how much interaction is required to solve self-driving cars, how much touch is required. You said that in your intuition, touch is not required in my intuition to create a product that's compelling to use. You're going to have to interact with pedestrians, not just avoid pedestrians, but interact with them. When we drive around in major cities, we're constantly threatening everybody's life with our movements. And that's how they respect us. There's a game theory going on with pedestrians. And I'm afraid you can't just formulate autonomous driving as a collision avoidance problem. So I think it goes beyond like a collision avoidance is the first order approximation. But then at least in case of test, they are gathering data from people driving their cars. And I believe that's an example of supervised data that they can train their models on. And they are doing it, which can give a model this like another level of behavior that is needed to actually interact with the real world. Yeah, it's interesting how much data is required to achieve that. What do you think of the whole Tesla autopilot approach, the computer vision based approach with multiple cameras and as a data engine, it's a multitask, multi-headed neural network, and it's this fascinating process of similar to what you're talking about with the robotics approach, which is you deploy a neural network and then there's humans that use it and then it runs into trouble in a bunch of places and that stuff is sent back. So the deployment discovers a bunch of edge cases and those edge cases are sent back for supervised annotation, thereby improving the neural network. And that's deployed again, it goes over and over until the network becomes really good at the task of driving, becomes safer and safer. What do you think of that kind of approach to robotics? I believe that's the way to go. So in some sense, even when I was speaking about collecting trajectories from humans, that's like a first step. And then you deploy the system and then you have humans revising all the issues. And in some sense, like this approach converges to a system that doesn't make mistakes because for the cases where there are mistakes, you gather data how to fix them and the system will keep on improving. So there's a very, to me, difficult question of how hard that, you know, how long that converging takes, how hard it is. The other aspect of autonomous vehicles probably applies to certain robotics applications is society, right? They put as the quality of the system converges. So one, there's a human factors perspective, a psychology of humans being able to supervise those, even with teleoperation, those robots. And the other society willing to accept robots. Currently, society is much harsher on self-driving cars than it is on human driven cars in terms of the expectation of safety. So the bar is set much higher than for humans. And so if there's a death in an autonomous vehicle, that's seen as much more dramatic than a death in a human driven vehicle. Part of the success of deployment of robots is figuring out how to make robots part of society, both on the just the human side, on the media journalist side and also on the policy government side. And that seems to be maybe you can put that into the objective function to optimize. But that is that is definitely a tricky one. And I wonder if that is actually the trickiest part for self-driving cars or any system that's safety critical. It's not the algorithm. It's the society accepting it. Yeah, I would say I believe that the part of the process of deployment is actually showing people that the given things can be trusted. Yeah. And trust is also like a glass that is actually really easy to crack it. Yeah. And damage it. And I think that's actually very common with innovation, that there is some resistance toward it. Yeah. And it's just a natural progression. So sometimes people have to keep on proving that indeed the systems are worth being used. And I would say I also found out that often the best way to convince people is by letting them experience it. Yeah, absolutely. That's the case with Tesla Autopilot, for example. That's the case with basically robots in general. It's kind of funny to hear people talk about robots like there's a lot of fear, even with like legged robots. But when they actually interact with them, there's joy. I love interacting with them. And the same with the car. With a robot, if it starts being useful, I think people immediately understand. And if the product is designed well, they fall in love. You're right. It's actually even similar when I'm thinking about Copilot, the GitHub Copilot. There was a spectrum of responses that people had. And ultimately, the important piece was to let people try it out. And then many people just loved it. Especially like programmers. Yeah, programmers. But like some of them, you know, they came with a fear. Yeah. But then you try it out and you think, actually, that's cool. And you know, you can try to resist the same way as you could resist moving from punch cards to, let's say, C++ or so. And it's a little bit futile. So we talked about generation of program, generation of language, even self-supervised learning in the visual space for robotics and then reinforcement learning. What do you and like this whole beautiful spectrum of AI, do you think is a good benchmark, a good test to strive for to achieve intelligence? That's a strong test of intelligence. You know, it started with Alan Turing and the Turing test. Maybe you think natural language conversation is a good test. So, you know, it would be nice if, for instance, machine would be able to solve Riemann hypothesis in math. That would be, I think that would be very impressive. So theorem proving, is that to you proving theorems is a good, oh, like one thing that the machine did, you would say, damn. Exactly. OK. That would be quite impressive. I mean, the tricky part about the benchmarks is, you know, as we are getting closer with them, we have to invent new benchmarks. There is actually no ultimate benchmark out there. Yes. My thought with the Riemann hypothesis would be the moment the machine proves it would say, OK, well, then the problem was easy. That's what happens. And I mean, in some sense, that's actually what happens over the years in AI that like we get used to things very quickly. You know something I talked to Rodney Brooks. I don't know if you know that is he called Alpha Zero homework problem because you're saying like there's nothing special about it. It's not a big leap. And I didn't. Well, he's coming from one of the aspects that we refer to as he was part of the founding of iRobot, which deployed now tens of millions of robot in the home. So if you see robots that are actually in the homes of people as the legitimate instantiation of artificial intelligence, then yes, maybe an AI that plays a silly game like going chess is not a real accomplishment. To me, it's a fundamental leap. But I think we as humans then say, OK, well, then that that game of chess or go wasn't that difficult compared to the thing that's currently unsolved. So my intuition is that from perspective of the evolution of these AI systems, we'll at first see the tremendous progress in digital space. And the main thing about this space is also that you can everything is that there is a lot of recorded data, plus you can very rapidly deploy things to billions of people. Well, in case of physical space, the deployment part takes multiple years. You have to manufacture things and delivering it to actual people is very hard. So I'm expecting that the first and that prices in digital space of goods, they would go down to the let's say marginal costs are zero. And also the question is how much of our life will be in digital? Because it seems like we're heading towards more and more of our lives being in the digital space. So like innovation in the physical space might become less and less significant. Like why do you need to drive anywhere if most of your life is spent in virtual reality? I still would like, you know, to at least at the moment, my impression is that I would like to have a physical contact with other people. And that's very important to me. We don't have a way to replicate it in the computer. It might be the case that over the time it will change like in 10 years from now. Why not have like an arbitrary infinite number of people you can interact with? Some of them are real, some are not with arbitrary characteristics that you can define based on your own preferences. I think that's maybe where we are heading and maybe I'm resisting the future. Yeah, I'm telling you, I if I got to choose if I could live in Elder Scrolls Skyrim versus the real world, I'm not so sure I would stay with the real world. Yeah, I mean, the question is, so will VR be sufficient to get us there or do you need to, you know, plug electrodes in the brain? And it would be nice if these electrodes wouldn't be invasive. Yeah, or at least like provably non-destructive. But in the digital space, do you think we'll be able to solve the Turing test, the spirit of the Turing test, which is do you think we'll be able to achieve compelling natural language conversation between people, like have friends that are AI systems on the Internet? I totally think it's doable. Do you think the current approach of GPT will take us there? Yes, so there is, you know, the part of at first learning all the content out there. And I think that still system should keep on learning as it speaks with you. Yeah. And I think that should work. The question is how exactly to do it. And, you know, obviously we have people at OpenAI asking these questions and kind of at first pre-training on all existing content is like a backbone and is a decent backbone. Do you think AI needs a body connecting to our robotics question to truly connect with humans? Or can most of the connection be in the digital space? So let's see. We know that there are people who met each other online and they fell in love. Yeah. So it seems that it's conceivable to establish connection, which is purely through Internet. Of course, it might be more compelling the more modalities you add. So it would be like you're proposing like a Tinder, but for AI. You like swipe right and left and half the systems are AI and the other is humans and you don't know which is which. That would be our formulation of Turing test. The moment AI is able to achieve more swipe right or left, whatever, the moment it's able to be more attractive than other humans, it passes the Turing test. Then you pass the Turing test in attractiveness. That's right. Well, no, like attractiveness just to clarify. There will be conversation. Not just visual. Right. Right. It's also attractiveness with wit and humor and whatever. Whatever makes conversations pleasant for humans. OK. All right. So you're saying it's possible to achieve in the digital space. In some sense, I would almost ask the question, why wouldn't that be possible? Right. Well, I have this argument with my dad all the time. He thinks that touch and smell are really important. So they can be very important. And I'm saying the initial systems, they won't have it. Still, I wouldn't like their people being born without the senses. And, you know, I believe that they can still fall in love and have meaningful life. Yeah. I wonder if it's possible to go close to all the way by just training on transcripts of conversations. Like, I wonder how far that takes us. So I think that actually still you want images like I would like. So I don't have kids, but like I could imagine having a tutor. It has to see, you know, kids drawing some pictures on a paper and facial expressions, all that kind of stuff. We use dogs and humans use their eyes to communicate with each other. I think that's a really powerful mechanism of communication. Body language, too, that words are much lower bandwidth. And for body language, we still, you know, we kind of have a system that displays an image of its facial expression on the computer. It doesn't have to move, you know, mechanical pieces or so. So I think that there is like kind of a progression. You can imagine that text might be the simplest to tackle, but this is not the complete human experience at all. You expand it to, let's say, images both for input and output. And what you describe is actually the final, I guess, frontier. What makes us human? The fact that we can touch each other or smell or so. And it's the hardest from perspective of data and deployment. And I believe that these things might happen gradually. Are you excited by that possibility, this particular application of human to AI friendship and interaction? So let's see. Like, would you do you look forward to a world you said you're living with a few folks and you're very close friends with them? Do you look forward to a day where one or two of those friends are AI systems? So if the system would be truly wishing me well, rather than being in the situation that it optimizes for my time to interact with the system. The line between those is it's a gray, it's a gray area. I think that's the distinction between love and possession. And these things, they might be often correlated for humans, but it's like you might find that there are some friends with whom you haven't spoken for months. Yeah. And then, you know, you pick up the phone as the time hasn't passed. They are not holding to you. And I wouldn't like to have AI system that, you know, it's trying to convince me to spend time with it. I would like the system to optimize for what I care about and help me in achieving my own goals. But there's some, I mean, I don't know. There's some manipulation, there's some possessiveness, there's some insecurities, there's fragility. All those things are necessary to form a close friendship over time, to go through some dark shit together, some bliss and happiness together. I feel like there's a lot of greedy, self-centered behavior within that process. My intuition, but I might be wrong, is that human-computer interaction doesn't have to go through a computer being greedy, possessive, and so on. It is possible to train systems, maybe, that they actually, you know, they are, I guess, prompted or fine-tuned or so to truly optimize for what you care about. And you could imagine that, you know, the way how the process would look like is at some point, we as humans, we look at the transcript of the conversation or like an entire interaction. And we say, actually, here there was more loving way to go about it. And we supervise system toward being more loving. Or maybe we train the system such that it has a reward function toward being more loving. Yeah. Or maybe the possibility of the system being an asshole and manipulative and possessive every once in a while is a feature, not a bug. Because some of the happiness that we experience when two souls meet each other, when two humans meet each other, is a kind of break from the assholes in the world. And so you need assholes in AI as well, because like it'll be like a breath of fresh air to discover an AI that the three previous AIs you had are too friendly. Or cruel or whatever. It's like some kind of mix. And then this one is just right. But you need to experience the full spectrum. Like I think you need to be able to engineer assholes. So let's see. Because there's some level to us being appreciated, to appreciate the human experience. We need the dark and the light. So that kind of reminds me. I met a while ago at the meditation retreat, one woman and a beautiful, beautiful woman. And she had a crutch. She had trouble walking on one leg. I asked her what has happened. And she said that five years ago she was in Maui, Hawaii. And she was eating a salad and some snail fell into the salad. And apparently there are neurotoxic snails over there. And she got into coma for a year. And apparently there is a high chance of event just dying. But she was in the coma. At some point she regained partially consciousness. She was able to hear people in the room. People behaved as she wouldn't be there. At some point she started being able to speak, but she was mumbling. Like barely able to express herself. At some point she got into wheelchair. Then at some point she actually noticed that she can move her toe. And then she knew that she will be able to walk. And then that's where she was five years after. And she said that since then she appreciates the fact that she can move her toe. And I was thinking, do I need to go through such experience to appreciate that I can move my toe? That's a really good story and a really deep example. And in some sense it might be the case that we don't see light if we haven't went through the darkness. But I wouldn't say that we should. We shouldn't assume that that's the case. We may be able to engineer shortcuts. Ilya had this belief that maybe one has to go for a week or six months to some challenging camp to just experience a lot of difficulties. And then comes back and actually everything is bright. Everything is beautiful. I'm with Ilya on this. It must be a Russian thing. Where are you from originally? I'm Polish. Polish. Okay. I'm tempted to say that explains a lot. But yeah, there's something about the Russian, the necessity of suffering. I believe suffering or rather struggle is necessary. I believe that struggle is necessary. I mean, in some sense, you even look at the story of any superhero in the movie. It's not that it was like everything goes easy, easy, easy, easy. I like how that's your ground truth is the story of superheroes. Okay. You mentioned that you used to do research at night and go to bed at like 6 a.m. or 7 a.m. I still do that often. What sleep schedules have you tried to make for a productive and happy life? Like, is there is there some interesting wild sleeping patterns that you engaged that you found that works really well for you? I tried at some point decreasing number of hours of sleep like a gradually. Like half an hour every few days to this. You know, I was hoping to just save time. That clearly didn't work for me. Like at some point there's like a phase shift and I felt tired all the time. You know, there was a time that I used to work during the nights. The nice thing about the nights is that no one disturbs you. And even I remember when I was meeting for the first time with Greg Brockman, his CTO and chairman of OpenAI. Our meeting was scheduled to 5 p.m. and I overstepped for the meeting. Overslept for the meeting. 5 p.m. Yeah. Now you sound like me. That's hilarious. Okay. Yeah. And at the moment, in some sense, my sleeping schedule also has to do with the fact that I'm interacting with people. I sleep without an alarm. So. So, yeah, the the team thing you mentioned extrovert thing, because most humans operate during a certain set of hours. You're forced to then operate at the same set of hours. But I'm not quite there yet. I found a lot of joy, just like you said, working through the night because it's quiet, because the world doesn't disturb you. And there's some aspect counter to everything you're saying. There's some joyful aspect to sleeping through the mess of the day because people are having meetings and sending e-mails and there's drama meetings. I can sleep through all the meetings. You know, I have meetings every day and they prevent me from having sufficient amount of time for focus work. And then I modified my calendar and I said that I'm out of office Wednesday, Thursday and Friday every day. And I'm having meetings only Monday and Tuesday. And that bus, the positively influenced my mood that I have literally like three days for fully focused work. Yeah. So there's better solutions to this problem than staying awake all night. OK. You've been part of development of some of the greatest ideas in artificial intelligence. What would you say is your process for developing good novel ideas? You have to be aware that clearly there are many other brilliant people around. So you have to ask yourself a question. Why the given idea, let's say, wasn't tried by someone else? And in some sense, it has to do with, you know, kind of simple. It might sound simple, but like thinking outside of the box. And what do I mean here? So, for instance, for a while, people in academia, they assumed that you have a fixed data set and then you optimize the algorithms in order to get the best performance. And that was so in great assumption that no one thought about training models on anti-internet or like that. Maybe some people thought about it, but it felt too many as unfair. And in some sense, that's almost like it's not my idea or so, but that's an example of breaking a typical assumption. So you want to be in the paradigm that you are breaking a typical assumption. In the context of the AI community, getting to pick your data set is cheating. Correct. And in some sense, so that was a that was assumption that many people had out there. And then if you free yourself from assumptions, then they are likely to achieve something that others cannot do. And in some sense, if you are trying to do exactly the same things as others, it's very likely that you're going to have the same results. Yeah. But there's also that kind of tension, which is asking yourself the question, why haven't others done this? Because, I mean, I get a lot of good ideas, but I think probably most of them suck when they meet reality. So actually, I think the other big piece is getting into a habit of generating ideas, training your brain to generating ideas and not even suspending judgment of the ideas. So in some sense, I noticed myself that even if I'm in the process of generating ideas, if I tell myself, oh, that was a bad idea, then that actually interrupts the process and I cannot generate more ideas because I'm actually focused on the negative part, why it won't work. But I created also an environment in the way that it's very easy for me to store new ideas. So, for instance, next to my bed, I have a voice recorder and it happens to me often. I wake up during the night and I have some idea. In the past, I was writing them down on my phone, but that means turning on the screen and that wakes me up. Or like pulling a paper, which requires turning on the light. These days, I just start recording it. What do you think? I don't know if you know who Jim Keller is. I know Jim Keller. He's a big proponent of thinking harder on a problem right before sleep so that he can sleep through it and solve it in his sleep or like come up with radical stuff in his sleep. He was trying to get me to do this. So it happened from my experience perspective. It happened to me many times during the high school days when I was doing mathematics that I had the solution to my problem as I woke up. At the moment, regarding thinking hard about the given problem is I'm trying to actually devote substantial amount of time to think about important problems, not just before the sleep. I am organizing the huge chunks of time such that I'm not constantly working on the urgent problems, but I actually have time to think about the important one. So you do it naturally. But his idea is that you kind of prime your brain to make sure that that's the focus. You know, oftentimes people have other worries in their life that's not fundamentally deep problems like, I don't know, just stupid drama in your life. And even at work, all that kind of stuff, he wants to kind of pick the most important problem that you're thinking about and go to bed on that. I think that's why I mean, the other thing that comes to my mind is also I feel the most fresh in the morning. So during the morning, I try to work on the most important things rather than just being pulled by urgent things or checking email or so. What do you do with the because I've been doing the voice recorder thing, too, but I end up recording so many messages. It's hard to organize. I have the same problem. Now I have heard that Google Pixel is really good in transcribing text and I might get a Google Pixel just for the sake of transcribing text. Yeah, people listening to this, if you have a good voice recorder suggestion that transcribes, please let me know. Some of it is this has to do with the OpenAI Codex, too. Like some of it is simply like the friction. I need apps that remove that friction between voice and the organization of the resulting transcripts and all that kind of stuff. But yes, you're right. Absolutely. Like during for me is walking sleep, too, but walking and running, especially running, get a lot of thoughts during running. And there's there's no good mechanism for recording thoughts. So one more thing that I do, I have a separate phone, which I which has no apps. Maybe it's like audible or let's say no one has this phone number, this kind of my meditation phone. Yeah. And I try to expand the amount of time that that's the phone that I'm having. It has also Google Maps if I need to go somewhere. And I also use this phone to write down ideas. Ah, that's a really good idea. That's a really good idea. I often actually what I end up doing is even sending a message from that phone to the other phone. So that's actually my way of recording messages or I just put them into notes. I love it. What advice would you give to a young person? High school, college about how to be successful. You've done a lot of incredible things in the past decade. So maybe maybe you have some something that might be something that might be something. I mean, it might sound like a simplistic or so, but I would say literally just follow your passion, double down on it. And if you don't know what's your passion, just figure out what could be what could be a passion. So this that might be an exploration. When I was in elementary school was math and chemistry. And I remember for some time I gave up on math because my school teacher, she told me that I'm dumb. And I guess maybe an advice would be just ignore people if they tell you that you're dumb. You're dumb. You mentioned something offline about chemistry and explosives. What was that about? So let's see. So a story goes like that. I got into chemistry. Maybe I was like a second grade of my elementary school, third grade. I started going to chemistry classes. I really love building stuff. And I did all the experiments that they describe in the book. I can you know how to create oxygen with vinegar and baking soda or so. So I did all the experiments and at some point I was, you know, so what's next? What can I do? And explosives they also it's like you have a clear reward signal, you know, if the thing worked or not. So I remember at first I got I got interested in producing hydrogen. That was kind of a funny experiment from school. You can just burn it. And then I moved to nitroglycerin. So that's also relatively easy to synthesize. I started producing essentially dynamite and detonating it with a friend. Remember there was a there was at first like maybe two attempts that I went with a friend to detonate what we built. And it didn't work out. And like a third time he was like, it won't work. Like, let's don't waste time. And we were I was carrying this this, you know, that tube with dynamite. I don't know, pound or so dynamite in my backpack. We're like riding on the bike to the edges of the city. Yeah. And attempt number three. This was be attempt number three. Attempt number three. And we dig a hole to put it inside. It actually had the, you know, electrical detonator. We draw a cable behind the tree. I even I never I haven't ever seen like an explosion before. So I thought that there would be a lot of sound. But, you know, we're like laying down and I'm holding the cable and the battery. At some point, you know, we kind of like a three to one. And I just connected it. And it felt like at the ground shake. It was like more like a sound. And then the soil started kind of lifting up and started falling on us. Yeah. Wow. And then the friends said, let's make sure next time we have helmets. But also, you know, I'm happy that nothing happened to me. It could have been the case that I lost the limbo or so. Yeah. But that's childhood of of an engineering mind with a strong reward signal of an explosion. I love it. My there's some aspect of chemists, the chemists. I know like my dad with plasma chemistry, plasma physics, he was very much into explosives, too. It's a worrying quality of people that work in chemistry that they love. I think it is that exactly is the the strong signal that the thing worked. There is no doubt. There's no doubt. There's some magic. It's almost like a reminder that physics works, that chemistry works. It's cool. It's almost like a little glimpse at nature that you yourself engineer. That's why I really like artificial intelligence, especially robotics, is you create a little piece of nature. And in some sense, even for me, with explosives, the motivation was creation rather than destruction. Yes, exactly. In terms of advice, I forgot to ask about just machine learning and deep learning for people who are specifically interested in machine learning. How would you recommend they get into the field? So I would say, re-implement everything. And also there is plenty of courses. So like from scratch? So on different levels of abstraction in some sense. But I would say re-implement something from scratch, re-implement something from a paper, re-implement something from podcasts that you have heard about. I would say that's a powerful way to understand things. So it's often the case that you read the description and you think you understand, but you truly understand once you build it. Then you actually know what really meant that in the description. Is there particular topics that you find people just fall in love with? I tend to really enjoy reinforcement learning because it's much easier to get to a point where you feel like you created something special. Like fun games kind of things that are rewarding. It's rewarding, yeah. As opposed to like re-implementing from scratch, more like supervised learning kind of things. So if someone would optimize for things to be rewarding, then it feels that the things that are somewhat generative, they have such a property. So you have, for instance, adversarial networks or you have just even generative language models. And you can even see internally, we have seen this thing with our releases. So we released recently two models. There is one model called Dali that generates images and there is other model called Clip that actually you provide various possibilities. What could be the answer to what is on the picture? And it can tell you which one is the most likely. And in some sense, in case of the first one, Dali, it is very easy for you to understand that actually there is magic going on. And in case of the second one, even though it is insanely powerful and people from the vision community, as they started probing it inside, they actually understood how far it goes. So it's difficult for a person at first to see how well it works. And that's the same, as you said, that in case of supervised learning models, you might not kind of see or it's not that easy for you to understand the strength. Even though you don't believe in magic, to see the magic. To see the magic, yes. It's generative. That's really brilliant. So anything that's generative, because then you are at the core of the creation. You get to experience creation without much effort, unless you have to do it from scratch. And it feels that humans are wired. There is some level of reward for creating stuff. Yeah. Of course, different people have a different weight on this reward. Yeah. In the big objective function. In the big objective function of a person. Of a person. You wrote that beautiful is what you intensely pay attention to. Even a cockroach is beautiful if you look very closely. Can you expand on this? What is beauty? So what I wrote here actually corresponds to my subjective experience that I had through extended periods of meditation. It's pretty crazy that at some point the meditation gets you to the place that you have really increased focus, increased attention. And then you look at the very simple objects that were all the time around you. You can look at the table or on the pen or at the nature. And you notice more and more details and it becomes very pleasant to look at it. And it once again, it kind of reminds me of my childhood. Like just pure joy of being. It's also, I have seen even the reverse effect that by default, regardless of what we possess, we very quickly get used to it. And you know, you can have a very beautiful house. And if you don't put sufficient effort, you're just going to get used to it. And it doesn't bring any more joy, regardless of what you have. Yeah. Well, I actually. I find that material possessions get in the way of that experience of pure joy. So I've always I've been very fortunate. To just find joy in simple things, just just like you're saying. Just like, I don't know, objects in my life, just stupid objects like this cup thing, you know, just objects. Sounds OK. I'm not being eloquent, but literally objects in the world. They're just full of joy because it's like I can't believe one. I can't believe that I'm fortunate enough to be alive, to experience these objects. And then, too, I can't believe humans are clever enough to have built these objects. The hierarchy of pleasure that that provides is infinite. I mean, even if you look at the cup of water, so, you know, you see first like a level of like a reflection of light. But then you think, no, man, there's like a trillions upon trillions of particles bouncing against each other. There is also the tension on the surface that, you know, if the back back could like a stand on it and move around. And you think it also has this like a magical property that as you decrease temperature, it actually expands in volume, which allows for the, you know, legs to freeze on the on the surface and at the bottom to have actually not freeze, which allows for life like a crazy. Yeah, you look in detail at some object and you think actually, you know, this table that was just the figment of someone's imagination at some point. And then there was like a thousands of people involved to actually manufacture it and put it here. And by default, no one cares. And then you can start thinking about evolution, how it all started from single cell organisms that led to this table. And these thoughts, they give me life appreciation. Yeah, and even lack of thoughts, just the pure raw signal also gives the life appreciation. See, the thing is, and then that's coupled for me with the sadness that the whole ride ends and perhaps is deeply coupled in that the fact that this experience, this moment ends gives it gives it an intensity that I'm not sure I would otherwise have. So in that same way, I try to meditate on my own death. Often. Do you think about your mortality? Are you afraid of death? So fear of death is like one of the most fundamental fears that each of us has. We might be not even aware of it. It requires to look inside to even recognize that it's out there. And there is still, let's say, this property of nature that if things would last forever, then they would be also boring to us. The fact that the things change in some way gives any meaning to them. I also found out that it seems to be very healing to people to have these short experiences, like, I guess, psychedelic experiences in which they experience death of self in which they let go of this fear and then maybe can even increase the appreciation of the moment. It seems that many people, they can easily comprehend the fact that their money is finite while they don't see that time is finite. I have this like a discussion with Ilya from time to time. He's like, you know, man, like the life will pass very fast. At some point I will be 40, 50, 60, 70 and then it's over. This is true, which also makes me believe that, you know, that every single moment it is so unique that should be appreciated. And it's also makes me think that I should be acting on my life because otherwise it will pass. I also like this framework of thinking from Jeff Bezos on regret minimization that like I would like if I will be at that deathbed to look back on my life and not regret that I haven't done something. It's usually you might regret that you haven't tried. I'm fine with failing. I haven't tried. I was in each year, eternal occurrence, tried to live a life that if you had to live it infinitely many times, that would be the you'd be OK with that kind of life. So try to live it optimally. I can say that it's almost like I'm. Unprecedented to me where I am in my life. I'm extremely grateful for actually people whom I met. I would say I think that I'm decent smart and so on. But I think that actually to great extent where I am has to do with the people who I met. Would you be OK if after this conversation you died. So if I'm dead, then it kind of I don't have a choice anymore. So in some sense, there's like a plenty of things that I would like to try out in my life. I feel that I'm gradually going one by one and I'm just doing them. I think that the least will be always infinite. Yeah, so might as well go today. Yeah, I mean, to be clear, I'm not looking forward to dying. I would say if there is no choice, I would accept it. But like in some sense, I'm if there would be a choice, if there would be possibility to leave, I would fight for living. I find. It's more honest and real to think about, you know, dying today at the end of the day. That seems to me to at least in my brain, more honest slap in the face as opposed to I still have 10 years like today. Then then I'm much more about appreciating the cup and the table and so on and less about like silly worldly accomplishments and all those kinds of things. We we we have in the company a person who, say, at some point found out that they have cancer. And that also gives you a huge perspective with respect to what matters now. And, you know, often people in situations like that, they conclude that actually what matters is human connection. And love and that's people conclude also if you have kids, kids, his family, you I think tweeted, we don't assign the minus infinity reward to our death. Such a reward would prevent us from taking any risk. We wouldn't be able to cross the road in fear of being hit by a car. So in the objective function, you mentioned fear of death might be fundamental to the human condition. So, as I said, let's assume that there are like a reward functions in our brain. And and the interesting thing is even realization how different reward functions can play with your behavior. As a matter of fact, I wouldn't say that you should assign infinite negative reward to anything because that messes up the math. The math doesn't work out. It doesn't work out. And as you said, even, you know, government or some insurance companies said they assign nine million dollars to human life. And I'm just saying it with respect to that might be a hard statement to ourselves, but in some sense that there is a finite value of our own life. I'm trying to put it from perspective of being less of being more ego less and realizing fragility of my own life. And in some sense, the fear of death might prevent you from acting because anything can cause death. Yeah. And I'm sure actually, if you were to put death in the objective function, there's probably so many aspects to death and fear of death and realization of death and mortality. There's just whole components of finiteness of not just your life, but every experience and so on that you're going to have to formalize mathematically. And also, you know, that might lead to you spending a lot of compute cycles on this like a deliberating this terrible future instead of experiencing now. And then in some sense, it's also kind of unpleasant simulation to run in your head. Yeah. Do you think there's an objective function that describes the entirety of human life? So, you know, usually the way you ask that is what is the meaning of life? Is there a universal objective functions that captures the why of life? So, yeah, I mean, I suspect that they will ask this question, but it's also a question that I asked myself many, many times. See, I can tell you a framework that I have these days to think about this question. So I think that fundamentally meaning of life has to do with some of our reward functions that we have in brain. And they might have to do with, let's say, for instance, curiosity or human connection, which might mean understanding others. It's also possible for a person to slightly modify their reward function. Usually they mostly stay fixed, but it's possible to modify reward function. And you can pretty much choose. So in some sense, the reward functions, optimizing reward functions, they will give you a life satisfaction. Is there some randomness in the function? I think when you are born, there is some randomness. Like you can see that some people, for instance, they they care more about building stuff. Some people care more about caring for others. Some people that there are all sorts of default reward functions. And then in some sense, you can ask yourself, what's this like? What is the satisfying way for you to go after this reward function? And you just go after this reward function. And, you know, some people ask, are these reward functions real? I almost think about it as. Let's say if you would have to discover mathematics in mathematics, you are likely to run into various objects like a complex numbers or differentiation, some other objects. And these are very natural objects that arise. And similarly, the reward functions that we are having in our brain, they are somewhat very natural that, you know, there is a reward function for for understanding like a comprehension. Yeah. Curiosity and so on. So in some sense, they are in the same way natural as their natural objects in mathematics. Interesting. So, you know, there's the the old sort of debate is mathematics invented or discovered. You're saying reward functions are discovered. So nature provides a sum you can still, let's say, expand it throughout the life. Some of the reward functions, they might be futile. Like, for instance, there might be a reward function, maximize amount of wealth. Yeah. And this is more like a learned reward function. And but we know also that some reward functions, if you optimize them, you won't be quite satisfied. Well, I don't know which part of your word function resulted in you coming today, but I am deeply appreciative that you did spend your valuable time with me. Wojciech is really fun talking to you. You're you're brilliant. You're a good human being. And it's an honor to meet you and an honor to talk to you. Thanks for talking to the brother. Thank you. Thanks a lot. I appreciated your questions. I had a lot of time being here. Thanks for listening to this conversation with Wojciech Sremba. To support this podcast, please check out our sponsors in the description. And now let me leave you with some words from Arthur C. Clark, who is the author of 2001, A Space Odyssey. It may be that our role on this planet is not to worship God, but to create him. Thank you for listening and hope to see you next time.
 The following is a conversation with Wojciech Zaremba,
 co-founder of OpenAI,
 which is one of the top organizations in the world
 doing artificial intelligence research and development.
 Wojciech is the head of language and cogeneration teams,
 building and doing research on GitHub Copilot,
 OpenAI Codex, and GPT-3 and who knows,
 four, five, six, N, and N plus one.
 And he also previously led OpenAI's robotic efforts.
 These are incredibly exciting projects to me
 that deeply challenge and expand our understanding
 of the structure and nature of intelligence.
 The 21st century, I think, may very well be remembered
 for a handful of revolutionary AI systems
 and their implementations.
 GPT, Codex, and applications of language models
 and transformers in general to the language
 and visual domains may very well be
 at the core of these AI systems.
 To support this podcast, please check out our sponsors.
 They're listed in the description.
 This is the Lex Friedman podcast,
 and here is my conversation with Wojciech Zaremba.
 You mentioned that Sam Altman asked
 about the Fermi paradox,
 and the people at OpenAI had really sophisticated,
 interesting answers.
 So that's when you knew this is the right team
 to be working with.
 So let me ask you about the Fermi paradox, about aliens.
 Why have we not found overwhelming evidence
 for aliens visiting Earth?
 I don't have a conviction in the answer,
 but rather kind of probabilistic perspective
 on what might be a, let's say, possible answer.
 It's also interesting that the question itself
 even can touch on your typical question
 of what's the meaning of life?
 Because if you assume that we don't see aliens
 because they destroy themselves,
 that kind of upweighs the focus on making sure
 that we won't destroy ourselves.
 But at the moment, the place where I am,
 actually with my belief,
 and these things also change over the time,
 is I think that we might be alone in the universe,
 which actually makes life more, or let's say consciousness,
 life more kind of valuable.
 And that means that we should more appreciate it.
 Have we always been alone?
 So what's your intuition about our galaxy, our universe?
 Is it just sprinkled with graveyards
 of intelligent civilizations,
 or is life, intelligent life, truly unique?
 At the moment, my belief that it is unique,
 but I would say I could also,
 there was some footage released
 with UFO objects,
 which makes me actually doubt my own belief.
 Yeah, I can tell you one crazy answer that I have heard.
 So apparently when you look actually
 at the limits of computation,
 you can compute more if the temperature
 of the universe would drop down.
 So one of the things that aliens might want to do
 if they are truly optimizing to maximize amount of compute,
 which maybe can lead to more, let's say simulations or so,
 it's instead of wasting current entropy of the universe,
 because we by living,
 we are actually somewhat wasting entropy,
 then you can wait for the universe to cool down
 such that you have more computation.
 That's kind of a funny answer.
 I'm not sure if I believe in it,
 but that would be one of the reasons
 why you don't see aliens.
 It's also possible to some people say that
 maybe there is not that much point
 in actually going to other galaxies
 if you can go inwards.
 So there is no limits of what could be an experience
 if we could connect machines to our brains,
 while there are still some limits
 if we want to explore the universe.
 Yeah, there could be a lot of ways to go inwards too.
 Once you figure out some aspect of physics
 we haven't figured out yet,
 maybe you can travel to different dimensions.
 I mean, travel in three-dimensional space
 may not be the most fun kind of travel.
 There may be just a huge amount
 of different ways to travel,
 and it doesn't require a spaceship going slowly
 in 3D space to space time.
 It also feels like one of the problems
 is that speed of light is low and the universe is vast.
 And it seems that actually most likely
 if we want to travel very far,
 then we would instead of actually sending spaceships
 with humans that weight a lot,
 we would send something similar
 to what Yuri Miller is working on.
 These are like a huge sail,
 which is at first powered,
 or there is a shot of laser from an air
 and it can propel it to quarter of speed of light.
 And sail itself contains a few grams of equipment.
 And that might be the way to actually transport matter
 through universe.
 But then when you think what would it mean for humans,
 it means that we would need to actually
 put their 3D printer and 3D print a human on other planet.
 I don't know, play them YouTube or let's say,
 or like a 3D print like huge human right away
 or maybe a womb or so.
 With our current techniques of archeology,
 if a civilization was born and died long enough ago on Earth,
 we wouldn't be able to tell.
 And so that makes me really sad.
 And so I think about Earth in that same way,
 how can we leave some remnants if we do destroy ourselves?
 How can we leave remnants for aliens in the future
 to discover?
 Like here's some nice stuff we've done.
 Like Wikipedia and YouTube,
 do we have it like in a satellite orbiting Earth
 with a hard drive?
 Like how do we say, how do we back up human civilization
 for the good parts or all of it is good parts
 so that it can be preserved longer than our bodies can?
 It's a difficult question.
 It also requires the difficult acceptance
 of the fact that we may die.
 And if we die, we may die suddenly as a civilization.
 So let's see, I think it kind of depends on the cataclysm.
 We have observed in other parts of the universe
 that birds of gamma rays,
 these are high energy rays of light
 that actually can apparently kill entire galaxy.
 So there might be actually nothing even to,
 nothing to protect us from it.
 I'm also, and I'm looking actually at the past civilizations.
 So it's like Aztecs or so,
 they disappear from the surface of the Earth
 and one can ask why is it the case?
 And the way I'm thinking about it is,
 you know, that definitely they had some problem
 that they couldn't solve.
 And maybe there was a flood and all of a sudden
 they couldn't drink, there was no potable water
 and they all died.
 And I think that so far,
 the best solution to such a problems is, I guess, technology.
 So, I mean, if they would know that you can just boil water
 and then drink it after,
 then that would save their civilization.
 And even now when we look actually at the current pandemic,
 it seems that once again, actually science comes to rescue
 and somehow science increases size of the action space.
 And I think that's a good thing.
 Yeah, but nature has a vastly larger action space.
 But still it might be a good thing for us
 to keep on increasing action space.
 Well, okay.
 Looking at past civilizations, yes.
 But looking at the destruction of human civilization,
 perhaps expanding the action space will add actions
 that are easily acted upon, easily executed,
 and as a result, destroy us.
 So let's see.
 I was pondering why actually even we have negative impact
 on the globe, because if you ask every single individual,
 they would like to have clean air,
 they would like healthy planet,
 but somehow it actually is not the case that as a collective,
 we are not going in this direction.
 I think that there exists very powerful system
 to describe what we value.
 That's capitalism.
 It assigns actually monetary values to various activities.
 At the moment, the problem in the current system
 is that there are some things which we value.
 There is no cost assigned to it.
 So even though we value clean air,
 or maybe we also value lack of destruction
 on the internet or so, at the moment, these quantities,
 companies, corporations can pollute them for free.
 So in some sense, I wished,
 or like that's I guess purpose of politics,
 to align the incentive systems.
 And we are kind of maybe even moving in this direction.
 The first issue is even to be able to measure
 the things that we value,
 then we can actually assign the monetary value to them.
 Yeah, and that's so it's getting the data
 and also probably through technology,
 enabling people to vote and to move money around
 in a way that is aligned with their values.
 And that's very much a technology question.
 So like having one president and Congress
 and voting that happens every four years
 or something like that, that's a very outdated idea.
 There could be some technological improvements
 to that kind of idea.
 So I'm thinking from time to time about these topics,
 but it also feels to me that it's a little bit like,
 it's hard for me to actually make correct predictions
 what is the appropriate thing to do.
 I extremely trust Sam Altman, our CEO on these topics here.
 Okay, I'm more on the side of being, I guess, naive hippie.
 That's your life philosophy.
 Well, I think self-doubt and I think hippie implies optimism.
 Those two things are pretty good way to operate.
 I mean, still it is hard for me to actually understand
 how the politics works or exactly how the things would play out.
 And Sam is really excellent with it.
 What do you think is rarest in the universe?
 You said we might be alone.
 What's hardest to build is another engineering way
 to ask that, life, intelligence or consciousness.
 So like you said that we might be alone,
 which is the thing that's hardest to get to?
 Is it just the origin of life?
 Is it the origin of intelligence?
 Is it the origin of consciousness?
 So let me at first explain you my kind of mental model,
 what I think is needed for life to appear.
 So I imagined that at some point there was this primordial soup
 of amino acids and maybe some proteins in the ocean.
 And some proteins were turning into some other proteins through reaction.
 And you can almost think about this cycle of what turns into what.
 There is a graph essentially describing
 which substance turns into some other substance.
 And essentially life means that all of a sudden in the graph
 has been created a cycle such that the same thing
 keeps on happening over and over again.
 That's what is needed for life to happen.
 And in some sense you can think almost that you have this gigantic graph
 and it needs a sufficient number of edges for the cycle to appear.
 Then from perspective of intelligence and consciousness,
 my current intuition is that they might be quite intertwined.
 First of all, it might not be that it's like a binary thing
 that you have intelligence or consciousness.
 It seems to be more a continuous component.
 Let's see, if we look for instance on the event networks recognizing images,
 people are able to show that the activations of these networks
 correlate very strongly with activations in visual cortex of some monkeys.
 The same seems to be true about language models.
 Also, if you for instance look, if you train agent in 3D world,
 at first it barely recognizes what is going on.
 Over time it recognizes foreground from background.
 Over time it knows where there is a foot and it just follows it.
 Over time it actually starts having a 3D perception.
 It is possible for instance to look inside of the head of an agent
 and ask what would it see if it looks to the right.
 The crazy thing is initially when the agents are barely trained
 these predictions are pretty bad.
 Over time they become better and better.
 You can still see that if you ask what happens when the head is turned by 360°
 for some time they think that the different thing appears
 and then at some stage they understand actually that the same thing is supposed to appear.
 So they get an understanding of 3D structure.
 It's also very likely that they have inside some level of symbolic reasoning,
 like particularly symbols for other agents.
 So when you look at DOTA agents, they collaborate together
 and they have some anticipation of if they would win battle.
 They have some expectations with respect to other agents.
 I might be too much anthropomorphizing how things look for me.
 But the fact that they have a symbol for other agents
 makes me believe that at some stage as they are optimizing for skills
 they would have also a symbol to describe themselves.
 This is a very useful symbol to have.
 This particularity I would call it self-consciousness or self-awareness.
 Still it might be different from the consciousness.
 I guess the way how I'm understanding the word consciousness,
 let's say the experience of drinking a coffee or let's say experience of being a butt,
 that's the meaning of the word consciousness.
 It doesn't mean to be awake.
 It might be also somewhat related to memory and recurrent connections.
 So it's kind of like if you look at anesthetic drugs, they essentially disturb brainwaves
 such that maybe memory is not formed.
 So there's a lessening of consciousness when you do that.
 Correct.
 And so that's the one way to intuit what is consciousness.
 There's also kind of another element here.
 It could be that it's this kind of self-awareness module that you described.
 Plus the actual subjective experience is a storytelling module
 that tells us a story about what we're experiencing.
 The crazy thing.
 So let's say, I mean, in meditation they teach people not to speak story inside of the head.
 There is also some fraction of population who doesn't have actually narrator.
 I know people who don't have a narrator and they have to use external people
 in order to kind of solve tasks that require internal narrator.
 So it seems that it's possible to have that experience without the talk.
 What are we talking about when we talk about the internal narrator?
 Is that the voice when you're reading a book?
 Yeah, I thought that that's what you are referring to.
 Well, I was referring more on not an actual voice.
 I meant like there's some kind of subjective experience feels like it's fundamentally about storytelling to ourselves.
 It feels like the feeling is a story that is much simpler abstraction than the raw sensory information.
 So there feels like it's a very high level abstraction that is useful for me to feel like entity in this world.
 Most useful aspect of it is that because I'm conscious,
 I think there's an intricate connection to me not wanting to die.
 So like it's a useful hack to really prioritize not dying.
 Those seem to be somewhat connected.
 So I'm telling the story of like it's rich.
 Feels like something to be me and the fact that me exists in this world.
 I want to preserve me.
 So that makes it a useful agent hack.
 So I will just refer maybe to that first part, as you said, about that kind of story of describing who you are.
 I was thinking about it even.
 Obviously, I like thinking about consciousness.
 I like thinking about AI as well.
 And I'm trying to see analogies of these things in AI.
 What would it correspond to?
 So OpenAI trained a model called GPT, which can generate pretty amusing text on arbitrary topic.
 And one way to control GPT is by putting into prefix at the beginning of the text some information.
 What would be the story about?
 You can have even chat with GPT by saying that the chat is with Lex or Elon Musk or so.
 And GPT would just pretend to be you or Elon Musk or so.
 And it almost feels that this story that we give ourselves to describe our life,
 it's almost like things that you put into context of GPT.
 Yeah, but the context we provide to GPT is multimodal.
 So GPT itself is multimodal.
 GPT itself hasn't learned actually from experience of single human, but from the experience of humanity.
 It's a chameleon.
 You can turn it into anything.
 And in some sense, by providing context, it behaves as the thing that you wanted it to be.
 It's interesting that people have stories of who they are.
 And as I said, these stories, they help them to operate in the world.
 But it's also interesting, I guess, various people find it out through meditation or so that there might be some patterns that you have learned when you were a kid that actually are not serving you anymore.
 And you also might be thinking that that's who you are. And that's actually just a story.
 Yes, it's a useful hack, but sometimes it gets us into trouble.
 It's a local optima. It's a local optima.
 You wrote that Stephen Hawking, he tweeted, Stephen Hawking asked what breathes fire into equations,
 which meant what makes given mathematical equations realize the physics of a universe.
 Similarly, I wonder what breathes fire into computation.
 What makes given computation conscious?
 OK, so how do we engineer consciousness?
 How do you breathe fire and magic into the machine?
 So it seems clear to me that not every computation is conscious.
 I mean, you can, let's say, just keep on multiplying one matrix over and over again and maybe gigantic matrix.
 You can put a lot of computation. I don't think it would be conscious.
 So in some sense, the question is, what are the computations which could be conscious?
 I mean, so one assumption is that it has to do purely with computation, that you can abstract away matter.
 Other possibilities that it's very important was the realization of computation that it has to do with some force fields or so.
 And they bring consciousness. At the moment, my intuition is that it can be fully abstracted away.
 So in case of computation, you can ask yourself, what are the mathematical objects or so that could bring such properties?
 So for instance, if we think about the models, AI models, what they truly try to do, or like models like GPT,
 is they try to predict next word or so. And this turns out to be equivalent to compressing text.
 And because in some sense, compression means that you learn the model of reality and you have just to remember where are your mistakes.
 The better you are in predicting. And in some sense, when we look at our experience, also when you look, for instance, at the car driving,
 you know in which direction it will go. You are good in prediction.
 And it might be the case that the consciousness is intertwined with compression.
 It might be also the case that self-consciousness has to do with compressor trying to compress itself.
 So I was just wondering what are the objects in mathematics or computer science, which are mysterious,
 that could have to do with consciousness. And then I thought, you see in mathematics, there is something called Gadel theorem,
 which means if you have sufficiently complicated mathematical system, it is possible to point the mathematical system back on itself.
 In computer science, there is something called halting problem. It's somewhat similar construction.
 So I thought that if we believe that under assumption that consciousness has to do with compression,
 then you could imagine that as you keep on compressing things, then at some point it actually makes sense for the compressor to compress itself.
 Metacompression. Consciousness is metacompression.
 That's an idea. And in some sense, you know, the crazy...
 Thank you. But do you think if we think of a touring machine, a universal touring machine, can that achieve consciousness?
 So is there something beyond our traditional definition of computation that's required?
 So it's a specific computation. And I said this computation has to do with compression.
 And the compression itself, maybe other way of putting it is like you are internally creating the model of reality.
 In order that you try inside to simplify reality in order to predict what's going to happen.
 And that also feels somewhat similar to how I think actually about my own conscious experience.
 So clearly I don't have access to reality. The only access to reality is through, you know, cable going to my brain.
 And my brain is creating a simulation of reality and I have access to the simulation of reality.
 Are you by any chance aware of the Hutter Prize, Marcus Hutter?
 He made this prize for compression of Wikipedia pages. And there's a few qualities to it.
 One, I think, has to be perfect compression, which makes...
 I think that little quirk makes it much less applicable to the general task of intelligence.
 Because it feels like intelligence is always going to be messy.
 Like perfect compression feels like it's not the right goal, but it's nevertheless a very interesting goal.
 So for him, intelligence equals compression.
 And so the smaller you make the file, given a large Wikipedia page, the more intelligent the system has to be.
 Yeah, that makes sense. So you can make perfect compression if you store errors.
 And I think that actually what he meant is you have algorithm plus errors.
 By the way, Hutter is a... he was a PhD advisor of Chen Leck, who is a DeepMind co-founder.
 Yeah, yeah. So there's an interesting... and now he's a DeepMind.
 There's an interesting network of people.
 He's one of the people that I think seriously took on the task of what would an AGI system look like?
 I think for the longest time, the question of AGI was not taken seriously or rather rigorously.
 And he did just that. Like mathematically speaking, what would the model look like?
 If you remove the constraints of it having to be... having to have a reasonable amount of memory,
 reasonable amount of running time complexity, computation time, what would it look like?
 And essentially, it's a half math, half philosophical discussion of how would a reinforcement learning type of framework look like for an AGI?
 So he developed the framework even to describe what's optimal with respect to reinforcement learning.
 Like there is a theoretical framework, which is, as you said, under assumption, there is infinite amount of memory and compute.
 There was actually one person before his name is Solomonov.
 Hutter extended Solomonov work to reinforcement learning, but there exists a theoretical algorithm,
 which is optimal algorithm to build intelligence. And I can actually explain you the algorithm.
 Yes, let's go. Let's go. Let's go.
 So the task itself, you can...
 Can I just pause how absurd it is for brain in a skull trying to explain the algorithm for intelligence? Just go ahead.
 It is pretty crazy. It is pretty crazy that, you know, the brain itself is actually so small and it can't ponder.
 How to design algorithms that optimally solve the problem of intelligence.
 OK. So what's the algorithm?
 So let's see. So first of all, the task itself is described as you have infinite sequence of zeros and ones.
 OK, you read n bits and you are about to predict n plus 1 bit.
 So that's the task. And you could imagine that every task could be casted as such a task.
 So if, for instance, you have images and labels, you can just turn every image into a sequence of zeros and ones, then label, you concatenate labels.
 And that's actually... You could start by having training data first and then afterwards you have test data.
 So theoretically any problem could be casted as a problem of predicting zeros and ones on this infinite tape.
 So let's say you read already n bits and you want to predict n plus 1 bit.
 And I will ask you to write every possible program that generates these n bits.
 OK, so and you can have... You choose programming language. It can be Python or C++.
 And the difference between programming languages might be there is a difference by constant.
 Asymptotically, your predictions will be equivalent.
 So you read n bits, you enumerate all the programs that produce these n bits in their output.
 And then in order to predict n plus 1 bit, you actually weight the programs according to their length.
 There is some specific formula how you weight them.
 And then the n plus 1 bit prediction is the prediction from each of these programs according to that weight.
 Like statistically. So the smaller the program, the more likely you are to pick its output.
 So that algorithm is grounded in the hope or the intuition that the simple answer is the right one.
 It is a formalization of it. It also means like if you would ask the question after how many years would sun explode.
 You can say it's more likely the answer is 2 to some power because it's a shorter program.
 Yeah.
 I don't have a good intuition about how different the space of short programs are from the space of large programs.
 Like what is the universe where short programs run things?
 So as I said, the things have to agree with n bits. So even if you have you need to start.
 If you have a very short program and they're like a still some has if it's not perfect with prediction of n bits, you have to start errors.
 What are the errors? And that gives you the full program that agrees on n bits.
 Oh, so you don't agree perfectly with the n bits and you store.
 That's like a longer, longer programs, like longer program, because it contains these extra bits of errors.
 That's fascinating. What's what's your intuition about the programs that are able to do cool stuff like intelligence and consciousness?
 Are they perfectly like is it is there if then statements in them?
 So like is there a lot of exceptions that they're storing?
 So you could imagine if there would be tremendous amount of statements, then they wouldn't be that short.
 In case of neural networks, you could imagine that what happens is they when you start with an uninitialized neural network,
 it stores internally many possibilities how the how the problem can be solved.
 And HDD is kind of magnifying some some some paths which are slightly similar to the correct answers.
 So it's kind of magnifying correct programs and in some sense HDD is the search algorithm in the program space.
 And the program space is represented by, you know, kind of the wiring inside of the neural network.
 And there's like an insane number of ways how the features can be computed.
 Let me ask you the high level basic question that's not so basic.
 What is deep learning? Is there a way you'd like to think of it that is different than like a generic?
 Textbook definition?
 The thing that I hinted just a second ago is maybe that closest to how I'm thinking these days about deep learning.
 So now the statement is neural networks can represent some programs.
 Seems that various modules that we are actually adding up to are like, you know, we we want networks to be deep because we we want multiple steps of the computation.
 And and deep learning provides the way to represent space of programs, which is searchable and it's searchable with stochastic gradient descent.
 So we have an algorithm to search over a humongous number of programs and gradient descent kind of bubbles up the things that tend to give correct answers.
 So a neural network with a with fixed weights that's optimized.
 Do you think of that as a single program?
 So there is a work by Christopher Ola, where he so he works on interpretability of neural networks and he was able to to identify inside of the neural network, for instance,
 a detector of a wheel for a car or the detector of a mask for a car.
 And then he was able to separate them out and assemble them together using a simple program for the detector for a car detector.
 That's like if you think of traditionally defined programs, that's like a function within a program that this particular neural network was able to find.
 And you can tear that out just like you can copy and paste from Stack Overflow that.
 So any program is a composition of smaller programs.
 Yeah, I mean, the nice thing about the neural networks is that it allows the things to be more fuzzy than in case of programs.
 In case of programs, you have this like a branching this way or that way.
 And the neural networks, they have an easier way to to be somewhere in between or to share things.
 What to use the most beautiful or surprising idea in deep learning and the utilization of these neural networks,
 which, by the way, for people who are not familiar, neural networks is a bunch of what would you say?
 It's inspired by the human brain. There's neurons. There's connection between those neurons.
 There's inputs and there's outputs and there's millions or billions of those neurons.
 And the learning happens by adjusting the weights on the edges that connect these neurons.
 Thank you for giving definition that I supposed to do it.
 But I guess you have enough empathy to listeners to actually know that that might be useful.
 No, that's like some asking Plato of like, what is the meaning of life?
 He's not going to answer. You're being philosophical and deep and quite profound talking about the space of programs,
 which is just very interesting. But also people are just not familiar with the hell we're talking about when we talk about deep learning.
 Anyway, sorry. What is the most beautiful or surprising idea to you in in in all the time you've worked at deep learning?
 And you worked on a lot of fascinating projects, applications of neural networks.
 It doesn't have to be big and profound. It can be a cool trick.
 Yeah, I mean, I'm thinking about the trick, but like it's still amusing to me that it works at all.
 Yeah. That let's say that the extremely simple algorithm stochastic gradient descent,
 which is something that I would be able to derive on the piece of paper to high school student when put at the scale of thousands of machines
 actually can create the behaviors which we called kind of human like behaviors.
 So in general, any application is stochastic gradient descent to neural networks is is amazing to you.
 So that or is there a particular application in natural language reinforcement learning?
 And also, what do you attribute that success to? Is it just scale?
 What profound insight can we take from the fact that the thing works for gigantic sets of variables?
 I mean, the interesting thing is this algorithms, they were invented decades ago and people actually gave up on the idea.
 Yeah. And, you know, back then they thought that we need profoundly different algorithms
 and they spent a lot of cycles on very different algorithms.
 And I believe that, you know, we have seen that various innovations that say like transformer or dropout or so they can, you know, pass the help.
 But it's also remarkable to me that this algorithm from 60s or so or I mean,
 you can even say that the gradient descent was invented by Leibniz in, I guess, 18th century or so.
 That actually is the core of learning.
 In the past, people are it's almost like out of the maybe an ego.
 People are saying that it cannot be the case that such a simple algorithm is there, you know, could solve complicated problems.
 So they were in search for the other algorithms.
 And as I'm saying, like, I believe that actually we are in the game where there is there are actually, frankly, three levers.
 There is compute, there are algorithms and there is data.
 And if we want to build intelligent systems, we have to pull all three levers and they are actually multiplicative.
 And it's also interesting. So yes, is it only compute?
 People internally, they did the studies to determine how much gains they were coming from different levers.
 And so far we have seen that more gains came from compute than algorithms.
 But also we are in the world that in case of compute, there is a kind of, you know, exponential increase in funding.
 And at some point, it's impossible to invest more. It's impossible to, you know, invest $10 trillion.
 We are speaking about the, let's say, all taxes in the US.
 But you're talking about money. There could be innovation in the compute.
 That's that's true as well. So, I mean, there are like a few pieces.
 So one piece is human brain is an incredible supercomputer.
 And they're like it has 100 trillion parameters.
 Or like if you try to count various quantities in the brain, they're like a neuron synapses, that small number of neurons.
 There is a lot of synapses. It's unclear even how to map synapses to parameters of neural networks.
 But it's clear that there are many more. So it might be the case that our networks are still somewhat small.
 It also might be the case that they are more efficient than brain or less efficient by some by some huge factor.
 I also believe that there will be like, you know, at the moment we are at the stage that these neural networks, they require 1000x or like a huge factor of more data than humans do.
 And it will be a matter of there will be algorithms that vastly decrease sample complexity. I believe so.
 But the place where we are heading today is there are domains which contains million X more data.
 And even though computers might be 1000 times slower than humans in learning, that's not the problem.
 For instance, I believe that it should be possible to create superhuman therapies by and there are even simple steps of doing it.
 And, you know, that the core reason is there is just machine will be able to read way more transcripts of therapies and then it should be able to speak simultaneously with many more people.
 And it should be possible to optimize it all in parallel.
 Well, there's now you're touching on something I deeply care about and think is way harder than we imagine.
 What's the goal of a therapist? What's the goal of therapists?
 So, OK, so one goal. Now, this is terrifying to me, but there's a lot of people that contemplate suicide, suffer from depression, and they could significantly be helped with therapy.
 And the idea that an AI algorithm might be in charge of that, it's like a life and death task.
 It's the stakes are high. So one goal for a therapist, whether human or AI, is to prevent suicide ideation to prevent suicide.
 How do you achieve that?
 So let's see. So to be clear, I don't think that the current models are good enough for such a task because it requires insane amount of understanding and patting.
 And the models are far from this place. But it's.
 But do you think that understanding empathy, that signal is in the data?
 I think there is some signal in the data. Yes. I mean, there are plenty of transcripts of conversations and it is possible to it is possible from it to understand personalities.
 It is possible from it to understand if conversation is friendly, amicable, antagonistic.
 It is, I believe that, you know, given the fact that the models that we train now, they can.
 They can have they are chameleons that they can have any personality.
 They might turn out to be better in understanding personality of other people than anyone else.
 And they are pathetic to be empathetic. Yeah. Interesting.
 But I wonder if there's some level of multiple modalities required to be able to be empathetic of the human experience, whether language is not enough to understand death, to understand fear, to understand childhood trauma,
 to understand wit and humor required when you're dancing with a person who might be depressed or suffering, both humor and hope and love and all those kinds of things.
 So there's another underlying question, which is self supervised versus supervised.
 So can you get that from the data by just reading a huge number of transcripts? I actually so I think that reading huge number of transcripts is a step one.
 It's like the same way as you cannot learn to dance if just from YouTube by watching it.
 You have to actually try it out yourself. And so I think that here that's a similar situation.
 I also wouldn't deploy the system in the high stakes situations right away, but kind of see gradually where it goes.
 And obviously, initially, it would have to go hand in hand with humans.
 But at the moment, we are in the situation that actually there is many more people who actually would like to have a therapy or speak with someone than there are therapies out there.
 I can you know, I was so fundamentally, I was thinking, what are the things that can vastly increase people well-being?
 Therapy is one of them. Being meditation is other one.
 I guess maybe human connection is the third one. And I guess pharmacologically, it's also possible.
 Maybe direct brain stimulation or something like that. But these are pretty much options out there.
 Then let's say the way I'm thinking about the AGI endeavor is by default, that's an endeavor to increase the amount of wealth.
 And I believe that we can vastly increase the amount of wealth for everyone.
 And simultaneously, so I mean, there are like two endeavors that make sense to me.
 One is like essentially increase the amount of wealth. And the second one is increase overall human well-being.
 And those are coupled together.
 I would say these are different topics. One can help another.
 And you know, therapist is a funny word because I see friendship and love as therapy.
 I mean, so therapist broadly defined is just friendship as a friend.
 So like therapist has a very kind of clinical sense to it. But what is human connection?
 You're like not to get all Camus and Dostoevsky on you, but you know, life is suffering and we draw.
 We seek connection with other humans as we desperately try to make sense of this world in a deep, overwhelming loneliness that we feel inside.
 So I think connection has to do with understanding.
 And I think that almost like a lack of understanding causes suffering. If you speak with someone and you feel ignored, that actually causes pain.
 If you are feeling deeply understood that actually they might not even tell you what to do in life, but like a pure understanding.
 Or just being heard. Understanding is a kind of, it's a lot, you know, just being heard.
 Feel like you're being heard. Like somehow that's alleviation temporarily of the loneliness.
 That if somebody knows you're here with their body language, with the way they are, with the way they look at you, with the way they talk, you feel less alone for a brief moment.
 Yeah, very much agree. So I thought in the past about somewhat similar question to yours, which is what is love rather what is connection?
 And obviously I think about these things from AI perspective. What would it mean?
 So I said that intelligence has to do with some compression, which is more or less like you can say almost understanding of what is going around.
 It seems to me that other aspect is there seem to be reward functions and you can have a reward for food, for maybe human connection, for let's say warmth, sex, and so on.
 And it turns out that the various people might be optimizing slightly different reward functions. They essentially might care about different things.
 And in case of love, at least the love between two people, you can say that the boundary between people dissolves to such extent that they end up optimizing each other reward functions.
 Yeah, that's interesting.
 Celebrate the success of each other.
 Yeah, in some sense, I would say love means helping others to optimize their reward functions, not your reward functions, not the things that you think are important, but the things that the person cares about.
 You try to help them to optimize it.
 So love is, if you think of two reward functions, you just it's a condition. You combine them together.
 Yeah, pretty much.
 Maybe like with a weight and it depends like the dynamic of the relationship.
 Yeah, I mean, you could imagine that if you are fully optimizing someone's reward function without yours, then maybe you are creating codependency or something like that.
 I'm not sure what's the appropriate weight. But the interesting thing is I even think that the individual person, we ourselves, we are actually less of a unified insight.
 So, for instance, if you look at the donut on the one level, you might think, oh, this looks tasty. I would like to eat it. On the other level, you might tell yourself, I shouldn't be doing it because I want to gain muscles.
 So and you know, you might do it regardless, kind of against yourself. So it seems that even within ourselves, they're almost like a kind of intertwined personas.
 And I believe that the self love means that the love between all these personas, which also means being able to love yourself when you are angry or stressed or so.
 Combining all those reward functions of the different selves you have.
 Yeah, and accepting that they are there. Like, you know, often people, they have a negative self-talk or they say, I don't like when I'm angry.
 And like I tried to imagine, tried to imagine if there would be like a small baby Lex, like a five years old, angry.
 And then they are like, you shouldn't be angry. Like, stop being angry.
 Yeah. But like instead, actually, you want the Lex to come over, give him a hug and is like, I say, it's fine.
 OK, you can be angry as long as you want. And then he would stop.
 Or maybe not.
 Or maybe not. But you cannot expect it even.
 Yeah. But still, that doesn't explain the why of love. Like, why is love part of the human condition?
 Why is it useful to combine the reward functions?
 It seems like that doesn't I mean, I don't think reinforcement learning frameworks can give us answers to why even even the Hutter framework has an objective function that's static.
 So we came to existence as a consequence of evolutionary process.
 And in some sense, the purpose of evolution is survival.
 And then this complicated optimization objective baked into us, let's say compression, which might help us operate in the real world.
 And it baked into us various reward functions.
 Yeah.
 Then to be clear, at the moment, we are operating in the regime, which is somewhat out of distribution where the even evolution optimized us.
 It's almost like love is a consequence of cooperation that we've discovered is useful.
 Correct. In some way, it's even the case if you.
 I just love the idea that love is like the out of distribution.
 Or it's not out of distribution. It's like as you said, it evolved for cooperation.
 Yes.
 And I believe that the cop like in some sense, cooperation ends up helping each of us individually.
 So it makes sense evolutionary.
 And there is in some sense, and you know, love means there is this dissolution of boundaries that you have a shared reward function.
 And we evolved to actually identify ourselves with larger groups.
 So we can identify ourselves with a family.
 We can identify ourselves with a country to such extent that people are willing to give away their life for country.
 So there is we are wired, actually, even for love.
 And at the moment, I guess the maybe it would be somewhat more beneficial if you will, if we would identify ourselves with all the humanity as a whole.
 So you can clearly see when people travel around the world, when they run into person from the same country, they say, oh, which city are and all this?
 Like all the sudden they find all these similarities. They they they find some they befriended those folks earlier than others.
 So there is like a sense, some sense of the belonging.
 And I would say I think it would be overall good thing to the world for people to move towards.
 I think it's even called open individualism, move toward the mindset of a larger and larger groups.
 So the challenge there, that's a beautiful vision. And I share it to expand that circle of empathy, that circle of love towards the entirety of humanity.
 But then you start to ask, well, where do you draw the line? Because why not expand it to other conscious beings?
 And then finally, for our discussion, something I think about is why not expand it to AI systems?
 Like we we start respecting each other when the person, the entity on the other side has the capacity to suffer, because then we develop a capacity to sort of empathize.
 And so I could see AI systems that are interacting with humans more and more having conscious like displays.
 So like they display consciousness through language and through other means.
 And so then the question is like, well, is that consciousness? Because they're acting conscious.
 And so, you know, the reason we don't like torturing animals is because they look like they're suffering when they're tortured.
 And if AI looks like it's suffering when it's tortured, how is that not requiring of the same kind of empathy from us and respect and rights that animals do and other humans do?
 I think it requires empathy as well. I mean, I would like, I guess, us or humanity or so make a progress in understanding what consciousness is,
 because I don't want just to be speaking about that philosophy, but rather actually make a scientific.
 To have a, like, you know, there was a time that people thought that there is a force of life and the things that have this force, they are alive.
 And I think that there is actually a path to understand exactly what consciousness is.
 And in some sense, it might require essentially putting probes inside of a human brain.
 What Neuralink does.
 So the goal there, I mean, there's several things with consciousness that make it a real discipline, which is one is rigorous measurement of consciousness.
 And then the other is the engineering of consciousness, which may or may not be related.
 You could also run into trouble, like, for example, in the United States, the Department of Department of Transportation and a lot of different places put a value on human life.
 I think DOT's values nine million dollars per person.
 Sort of in that same way, you can get into trouble if you put a number on how conscious a being is, because then you can start making policy.
 If a cow is point one or like 10 percent as conscious as a human, then you can start making calculations and might get you into trouble.
 But then again, that might be a very good way to do it.
 I would like to move to that place that actually we have scientific understanding what consciousness is.
 And then we'll be able to actually assign value.
 And I believe that there is even the path for the experimentation in it.
 So, you know, we said that, you know, you could put the probes inside of the brain.
 There is actually a few other things that you could do with devices like Neuralink.
 So you could imagine that the way even to measure if a system is conscious is by literally just plugging into the brain.
 I mean, that seems that's kind of easy.
 But the plugging into the brain and asking person if they feel that their consciousness expanded.
 And this direction, of course, has some issues.
 You can say, you know, if someone takes a psychedelic drug, they might feel that their consciousness expanded, even though that drug itself is not conscious.
 Right. So like you can't fully trust the self-report of a person saying their their consciousness is expanded or not.
 Let me ask you a little bit about psychedelics.
 There's been a lot of excellent research on different psychedelics, psilocybin, MDMA, even DMT, drugs in general, marijuana, too.
 What do you think psychedelics do to the human mind?
 It seems they take the human mind to some interesting places.
 Is that just a little hack, a visual hack, or is there some profound expansion of the mind?
 So let's see. I don't believe in magic.
 I believe in I believe in in science, in in causality.
 Still, let's say. And then, as I said, like, I think that the brain.
 That our subjective experience of reality is a.
 We live in the simulation run by our brain and the simulation that our brain runs, they can be very pleasant or very hellish.
 Drugs, they are changing some hyperparameters of the simulation.
 It is possible thanks to change of these hyperparameters to actually look back on your experience and even see that the given things that we took for granted, they are changeable.
 So they allow to have an amazing perspective.
 There is also, for instance, the fact that after DMT, people can see the full movie inside of their head gives me further belief.
 That brain can generate that full movie that the brain is actually.
 Learning the model of reality to such extent that it tries to predict what's going to happen next.
 Yeah, very high resolution so it can replay realities at extremely high resolution.
 Yeah, it's also kind of interesting to me that somehow there seems to be some similarity between.
 These drugs and meditation itself, and I actually started even these days to think about meditation as a psychedelic.
 Do you practice meditation?
 I practice meditation. I mean, I once few times on the retreats and it feels after like after second or third day of meditation.
 There is a there is almost like a sense of, you know, tripping.
 What is the meditation retreat entail?
 So you wake up early in the morning and you meditate for extended period of time.
 And alone.
 Yeah, so it's optimized even though there are other people, it's optimized for isolation.
 So you don't speak with anyone. You don't actually look into other people eyes.
 And you know you sit on the chair.
 Say the personal meditation tells you to focus on the breath.
 So you try to put all the all attention into breathing and breathing in and breathing out.
 And the crazy thing is that as you focus attention like that, after some time.
 There's some starts coming back like some memories that you're completely forgotten.
 It almost feels like that you have a mailbox and then you know you are just like archiving email one by one.
 And at some point, at some point there is like an amazing feeling of getting to mailbox zero, zero emails.
 And it's very pleasant.
 It's kind of it's it's it's crazy to me that that once.
 You resolve these inner store stories or like inner traumas.
 Then once there is nothing left, the default state of human mind is extremely peaceful and happy.
 Extreme like some sense it it feels that the.
 It feels at least to me in that way how when I was a child that I can look at any object and it's very beautiful.
 I have a lot of curiosity about the simple things.
 And that's where the usual meditation takes me.
 Are you what are you experiencing?
 Are you just taking in simple sensory information and are just enjoying the rawness of that sensory information?
 So there's no there's no memories or all that kind of stuff.
 You're just enjoying being.
 Yeah, pretty much. I mean, still there is a that it's it's thoughts are slowing down.
 Sometimes they pop up, but it's also somehow the extended meditation takes you to the space that they are way more friendly.
 You're way more positive.
 There is also this this thing that we've extracted.
 It almost feels that the.
 It almost feels that the we are constantly getting a little bit of a reward function and we are just spreading this reward function on various activities.
 But if you stay still for extended period of time, it kind of accumulates, accumulates, accumulates and.
 There is a there is a sense there is a sense that some point it passes some threshold and it feels as.
 Drop is falling into kind of ocean of love and bliss.
 And that's like a this is like a very pleasant and that's I'm saying like.
 That corresponds to the subjective experience.
 Some people, I guess, in spiritual community, they describe it that that's the reality.
 And I would say I believe that there are like all sorts of subjective experience that one can have.
 And I believe that, for instance, meditation might take you to the subjective experiences, which are very pleasant, collaborative.
 And I would like a word to move toward a more collaborative place.
 Yeah, I would say that's very pleasant and I enjoy doing stuff like that.
 I wonder how that maps to your mathematical model of love with the reward function combining a bunch of things.
 It seems like our life then is we're just we have this reward function and we're accumulating a bunch of stuff in it with weights.
 It's like like multi objective.
 And what meditation is, is you just remove them, remove them until the weight on one or just a few is very high.
 And that's where the pleasure comes from. Yeah.
 So something similar, how I'm thinking about it.
 So I told you that there is this like that there is a story of who you are.
 And I think almost about it as a, you know, text prepended to GPT.
 Yeah. And some people refer to it as ego.
 OK, it's like a story. Who, who, who you are.
 OK, so ego is the prompt for GPT3.
 Yeah, yes, yes. And that's description of you.
 And then with meditation, you can get to the point that actually you experience things without the prompt and you experience things like as they are.
 You are not biased or the description how they supposed to be.
 That's very pleasant. And then with respect to the reward function, it's possible to get to the point that there is the solution of self.
 And therefore you can say that the you're having your or like a your brain attempts to simulate the reward function of everyone else or like everything that's that there is this like a love, which feels like a oneness with everything.
 And that's also, you know, very beautiful, very pleasant.
 At some point you might have a lot of altruistic thoughts during that moment.
 And then the self always comes back.
 How would you recommend if somebody is interested in meditation, like a big thing to take on as a project?
 Would you recommend a meditation retreat? How many days? What kind of thing would you recommend?
 I think that actually retreat is the way to go.
 It almost feels that the say meditation is a psychedelic, but when you take it in the small dose, you might barely feel it.
 Once you get the high dose, actually you're going to feel it.
 So even cold turkey, if you haven't really seriously meditated for prolonged period of time, just go to a retreat.
 Yeah, how many days? How many days?
 Start the weekend one weekend. So like two, three days.
 And it's interesting that first or second day, it's hard. And at some point it becomes easy.
 There's a lot of seconds in a day. How hard is the meditation retreat? Just sitting there in a chair.
 So the thing is actually, it literally just depends on your own framing.
 Like if you are in the mindset that you are waiting for it to be over or you are waiting for Nirvana to happen, it will be very unpleasant.
 And in some sense, even the difficulty, it's not even in the lack of being able to speak with others.
 Like you're sitting there, your legs will hurt from sitting.
 In terms of like the practical things, do you experience kind of discomfort, like physical discomfort of just sitting, like your your butt being numb,
 your legs being sore, all that kind of stuff?
 Yes, you experience it. And then they teach you to observe it rather than it's like the crazy thing is you at first might have a feeling toward trying to escape it.
 And that becomes very apparent that that's extremely unpleasant.
 And then you just just observe it. And at some point it just becomes it just is.
 It's like I remember we've Ilya told me some time ago that he takes a cold shower and his mindset of taking a cold shower was to embrace suffering.
 Yeah, excellent. I do the same.
 Is this your style?
 Yes, my style. I like this.
 So my style is actually I also sometimes take cold showers.
 It is purely observing how the water goes through my body, like purely being present, not trying to escape from there.
 Yeah. And I would say then it actually becomes pleasant.
 It's not like, well, that's interesting.
 I'm also that me. That's that's the way to deal with anything really difficult, especially in the physical space is to observe it to say it's pleasant.
 Hmm. It's a I would use a different word.
 You're you're accepting of the full beauty of reality, I would say, because say pleasant.
 But yeah, in some sense, it is pleasant. That's the only way to deal with a cold shower is to become an observer and to find joy in it.
 Same with like really difficult physical exercise or like running for a really long time endurance events anytime you're any kind of pain.
 I think the only way to survive it is not to resist it, just observe it.
 You mentioned Ilja Ilja Setskever is very he's our chief scientist, but also he's very close friend of mine.
 He co-founded OpenAI with you.
 I've spoken with him a few times. He's brilliant. I really enjoy talking to him.
 His mind, just like yours, works in fascinating ways.
 Both of you are not able to define deep learning simply.
 What's it like having him as somebody you have technical discussions with on in space machine learning, deep learning, AI, but also life?
 What's it like when these two agents get into a self-play situation in a room?
 What's it like collaborating with him?
 So I believe that we have extreme respect to each other.
 So I love Ilja's insight, both like I guess about consciousness, life, AI.
 But in terms of the it's interesting to me because you're a brilliant thinker in the space of machine learning, like intuition, like digging deep in what works, what doesn't, why it works, why it doesn't.
 And so is Ilja. I'm wondering if there's interesting deep discussions you've had with him in the past or disagreements that were very productive.
 So I can say I also understood over the time where are my strengths.
 So obviously we have plenty of AI discussions and I myself have plenty of ideas, but I consider Ilja one of the most prolific AI scientists in the entire world.
 And I think that I realized that maybe my super skill is being able to bring people to collaborate together, that I have some level of empathy that is unique in AI world.
 And that might come from either meditation, psychedelics, or let's say I read just hundreds of books on this topic.
 And I also went through a journey of I develop all sorts of algorithms.
 So I think that maybe I can, that's my super human skill.
 Ilja is one of the best AI scientists, but then I'm pretty good in assembling teams.
 And I'm also not holding to people like I'm growing people and then people become managers at OpenAI. I grew many of them like a research managers.
 You find places where you're excellent and he finds like his deep scientific insights is where he is and you find ways you can, the puzzle pieces fit together.
 Correct. Ultimately, for instance, let's say Ilja, he doesn't manage people.
 That's not what he likes or so. I like hanging out with people. By default, I'm an extrovert and I care about people.
 Okay. Cool. So that fits perfectly together.
 But I mean, I also just like your intuition about various problems in machine learning. He's definitely one I really enjoy.
 I remember talking to him about something I was struggling with, which is coming up with a good model for pedestrians, for human beings across the street in the context of autonomous vehicles.
 And he immediately started to formulate a framework within which you can evolve a model for pedestrians, like through self play, all that kind of mechanisms.
 The depth of thought on a particular problem, especially problems he doesn't know anything about is fascinating to watch.
 It makes you realize like, yeah, the limits that the human intellect might be limitless. I was just impressive to see a descendant of APE come up with clever ideas.
 Yeah, I mean, so even in the space of deep learning, when you look at various people, there are people who invented some breakthroughs once, but there are very few people who did it multiple times.
 And you can think if someone invented it once, that might be just a sheer luck. And if someone invented it multiple times, you know, if a probability of inventing it once is one over a million, then probability of inventing it twice or three times would be one over a million square or to the power of three, which would be just impossible.
 So it literally means that it's given that it's not the luck. And Ilya is one of these few people who have a lot of these inventions in his arsenal.
 It also feels that, for instance, if you think about folks like Gauss or Euler, you know, at first they read a lot of books, and then they did thinking, and then they figure out math.
 And that's how it feels with Ilya. You know, at first he read stuff and then like he spent his thinking cycles.
 That's a really good way to put it. When I talk to him, I see thinking. He's actually thinking.
 He makes me realize that there's like deep thinking that the human mind can do. Like most of us are not thinking deeply. Like you really have to put a lot of effort to think deeply.
 Like I have to really put myself in a place where I think deeply about a problem. It takes a lot of effort. It's like an airplane taking off or something. You have to achieve deep focus.
 He's just he's what is it? His brain is like a vertical takeoff in terms of airplane analogy. So it's interesting.
 But I mean, Cal Newport talks about this as ideas of deep work. It's you know, most of us don't work much at all in terms of like like deeply think about particular problems, whether it's a math, engineering, all that kind of stuff.
 You want to go to that place often. And that's real hard work. And some of us are better than others at that.
 So I think that the big piece has to do with actually even engineering your environment such that it's conducive to that.
 So see, both Ilya and I on the frequent basis, we kind of disconnect ourselves from the world in order to be able to do extensive amount of thinking.
 Yes. So Ilya usually he just leaves iPad at hand. He loves his iPad.
 And for me, I'm even sometimes, you know, just going for a few days to different location to Airbnb. I'm turning off my phone and there is no access to me.
 Yeah. And that's extremely important for me to be able to actually just formulate new thoughts, to do deep work rather than to be reactive.
 And the older I am, the more of these like random tasks are at hand.
 Before I go on to that thread, let me return to our friend GPT.
 Let me ask you another ridiculously big question. Can you give an overview of what GPT-3 is or like you say in your Twitter bio, GPT N plus one, how it works and why it works?
 So GPT-3 is a human-goose neural network. Let's assume that we know what is neural network by definition.
 And it is trained on the entire Internet just to predict next words. So let's say it sees part of the article and the only task that it has at hand is to say what would be the next word.
 What would be the next word? And it becomes really exceptional at the task of figuring out what's the next word.
 So you might ask, why would this be an important task? Why would it be important to predict what's the next word?
 And it turns out that a lot of problems can be formulated as a text completion problem. So GPT is purely learning to complete the text.
 You could imagine, for instance, if you are asking a question, who is the president of the United States, then GPT can give you an answer to it.
 It turns out that many more things can be formulated this way. You can format text in the way that you have sent us in English.
 You make it even look like some content of a website elsewhere, which would be teaching people how to translate things between languages. So it would be EN colon, text in English, FR colon.
 And then you ask the model to continue. And it turns out that such a model is predicting translation from English to French.
 The crazy thing is that this model can be used for way more sophisticated tasks.
 So you can format text such that it looks like a conversation between two people. And that might be a conversation between you and Elon Musk.
 And because the model read all the text about Elon Musk, it will be able to predict Elon Musk words as it would be Elon Musk.
 It will speak about colonization of Mars, about sustainable future and so on. And it's also possible to even give arbitrary personality to the model.
 You can say, here is a conversation with a friendly AI bot. And the model will complete the text as a friendly AI bot.
 So, I mean, how do I express how amazing this is? So just to clarify a conversation, generating a conversation between me and Elon Musk, it wouldn't just generate good examples of what Elon would say.
 It would get the syntax all correct. So like interview style, it would say like Elon colon and Lex colon. It's not just like inklings of semantic correctness.
 It's like the whole thing. Grammatical, syntactic, semantic. It's just really, really impressive generalization.
 Yeah, I mean, I also want to provide some caveats. So it can generate a few paragraphs of coherent text.
 But as you go to longer pieces, it actually goes off the rails. If you try to write a book, it won't work out this way.
 What way does it go off the rails, by the way? Is there interesting ways in which it goes off the rails? What falls apart first?
 So the model is trained on all the existing data that is out there, which means that it is not trained on its own mistakes.
 So, for instance, if it would make a mistake, then I can give you an example.
 So let's say I have a conversation with a model pretending that is Elon Musk. And then I start putting some, I start actually making up things which are not factual.
 I would say Twitter. But I got you. Sorry. Yeah.
 Okay. I don't know. I would say that Elon is my wife and the model will just keep on carrying it on.
 As if it's true.
 Yes. And in some sense, if you would have a normal conversation with Elon, he would be what the fuck.
 Yeah, there would be some feedback between. So the model is trained on things that humans have written, but through the generation process, there's no human in the loop feedback.
 Correct. That's fascinating. Makes sense. So it's magnifies like the errors get magnified and magnified.
 And it's also interesting. I mean, first of all, humans have the same problem is just that we will make fewer errors and magnify the errors slower.
 I think that actually what happens with humans is if you have a wrong belief about the world as a kid, then very quickly you will learn that it's not correct because they are grounded in reality and they are learning from your new experience.
 But do you think the model can correct itself to won't it through the power of the representation?
 And so the absence of Elon Musk being your wife, information on the Internet won't to correct itself.
 There won't be examples like that. So the errors will be subtle at first. Saddle at first.
 And in some sense, you can also say that the data that is not out there is a data which would represent how the human learns.
 And maybe model would be learned, trained on such a data, then it would be better off.
 How intelligent is GPT-3, do you think? Like when you think about the nature of intelligence, it seems exceptionally impressive.
 But then if you think about the big AGI problem, is this footsteps along the way to AGI?
 So let's see. It seems that intelligence itself is there are multiple axis of it.
 And I would expect that the the systems that we are building, they might end up being superhuman on some axis and subhuman on some other axis.
 It would be surprising to me on all axis simultaneously, they would become superhuman. Of course, people ask this question.
 Is GPT a spaceship that that would take us to the moon or are we building a ladder to heaven that we are just building bigger and bigger ladder?
 And we don't know in some sense which one of these two works.
 Which one is better? I'm trying to I like Stairway to Heaven as a good song, so I'm not exactly sure which one is better.
 But you're saying like the spaceship to the moon is actually effective?
 Correct. So people who criticize GPT, they say, you guys just building a taller ladder and it will never reach the moon.
 And at the moment, I would say the way I'm thinking is this is like a scientific question.
 And I'm also in heart, I'm a builder creator and like I'm thinking, let's try out. Let's see how far it goes.
 And so far we see constantly that there is a progress.
 Yeah. So what do you think GPT4, GPT5, GPTn plus one will there be a phase shift, like a transition to a place where we'll be truly surprised?
 Then again, like GPT3 is already very like truly surprising.
 The people that criticize GPT3 as it's there as a what is it ladder to heaven, I think too quickly get accustomed to how impressive it is that the prediction of the next word can achieve such depth of semantics, accuracy, syntax, grammar and semantics.
 Do you think GPT4 and 5 and 6 will continue to surprise us?
 I mean, definitely there will be more impressive models. There is a question, of course, if there will be a phase shift.
 And also even the way I'm thinking about these models is that when we build these models, you know, we see some level of the capabilities, but we don't even fully understand everything that the model can do.
 And actually one of the best things to do is to allow other people to probe the model to even see what is possible.
 Hence the using GPT as an API and opening it up to the world.
 Yeah, I mean, so when I'm thinking from perspective of, obviously, various people that have concerns about AGI, including myself.
 And then when I'm thinking from perspective, what's the strategy even to deploy these things to the world?
 The one strategy that I have seen many times working is the iterative deployment that you deploy slightly better versions and you allow other people to criticize you.
 So you actually are tried out. You see where are their fundamental issues.
 And it's almost you don't want to be in that situation that you are holding into powerful system and there's like a huge overhang.
 Then you deploy it and it might have a random chaotic impact on the world.
 So you actually want to be in the situation that they are gradually deploying systems.
 I asked this question of Ilya. Let me ask you this question.
 I've been reading a lot about Stalin and power.
 If you're in possession of a system that's like AGI, that's exceptionally powerful.
 Do you think your character and integrity might become corrupted?
 Like famously power corrupts and absolute power corrupts. Absolutely.
 So I believe that you want at some point to work toward distributing the power.
 I think that you want to be in the situation that actually AGI is not controlled by a small number of people, but essentially by a larger collective.
 So the thing is that requires a George Washington style move.
 In the ascent to power, there's always a moment when somebody gets a lot of power and they have to have the integrity and the moral compass to give away that power.
 That humans have been good and bad throughout history at this particular step.
 And I wonder, I wonder if we blind ourselves, for example, between nations, a race towards AI race between nations.
 We might blind ourselves and justify to ourselves the development of AI without distributing the power because we want to defend ourselves against China, against Russia, that kind of logic.
 And I wonder how we design governance mechanisms that prevent us from becoming power hungry and in the process destroying ourselves.
 So let's see. I have been thinking about this topic quite a bit, but I also want to admit that once again, I actually want to rely way more on Sam Altman on it.
 He wrote an excellent blog on how even to distribute wealth.
 And he proposed in his blog to tax equity of the companies rather than profit and to distribute it.
 And this is an example of a Washington move.
 I guess I personally have insane trust in Sam.
 I mean, he already spent plenty of money running universal basic income project that gives me, I guess, maybe some level of trust to him.
 But I also, I guess, love him as a friend.
 Yeah, I wonder because we're sort of summoning a new set of technologies.
 I wonder if we'll be cognizant like you're describing the process of open AI, but it could also be at other places like in the US government.
 Right. Both China and the US are now.
 Full steam ahead on autonomous weapons systems development.
 And that's really worrying to me because in the framework of something being a national security danger or a military danger, you can do a lot of pretty dark things that blind our moral compass.
 And I think AI will be one of those things.
 In some sense, the mission and the work you're doing at open AI is like the counterbalance to that.
 So you want to have more open AI and less autonomous weapons systems.
 I like this statement. To be clear, this is interesting and I'm thinking about it myself.
 But this is a place that I put my trust actually in Sam's hands because it's extremely hard for me to reason about it.
 Yeah. I mean, one important statement to make is it's good to think about this.
 Yeah, no question about it.
 No question about it. Even like low level, quote unquote, engineer.
 Like there's such a. I remember I programmed a car, RC car.
 They went really fast, like 30, 40 miles an hour.
 And I remember I was like sleep deprived, so I programmed it pretty crapily and it like the code froze.
 So it's doing some basic computer vision and it's going around on track, but it's going full speed.
 And there's a bug in the code that the car just went.
 It didn't turn. It went straight full speed and smash into the wall.
 And I remember thinking. The seriousness with which you need to approach the design of artificial intelligence systems
 and the programming of artificial intelligence systems is high because the consequences are high.
 That little car smashing into the wall.
 For some reason, I immediately thought of like an algorithm that controls nuclear weapons having the same kind of bug.
 And so like the lowest level engineer and the CEO of a company all need to have the seriousness in approaching this problem and thinking about the worst case consequences.
 So I think that is true. I mean, what I also recognize in myself and others even asking this question is that it evokes a lot of fear.
 And fear itself ends up being actually quite debilitating.
 The place where I arrived at the moment.
 Might sound cheesy or so, but it's almost to.
 Build things out of love rather than fear.
 I can focus on how I can maximize the value, how the systems that I'm building might be useful.
 I'm not saying that the fear doesn't exist out there and it totally makes sense to minimize it.
 But I don't want to be working because I'm scared.
 I want to be working out of passion, out of curiosity, out of the looking forward for the positive future.
 With the definition of love arising from rigorous practice of empathy.
 So not just like your own conception of what is good for the world, but always listening to others.
 Correct. Like at the love where I'm considering reward functions of others.
 Others to limit to infinity is like a sum of like one to N where N is seven billion or whatever it is.
 Not projecting my reward functions on others.
 Yeah, exactly.
 OK, can we just take a step back to something else?
 Super cool, which is open air codex.
 Can you give an overview of what open air codex and GitHub copilot is, how it works and why the hell it works so well?
 So with GPT-3, we noticed that the system, you know, that system trained on all the language out there.
 Started having some rudimentary coding capabilities.
 So we're able to ask it to implement addition function between two numbers.
 And indeed, it can write Python or JavaScript code for that.
 And then we thought we might as well just go full steam ahead and try to create a system that is actually good at what we are doing every day ourselves, which is programming.
 We optimized models for proficiency in coding.
 We actually even created models that both have a comprehension of language and code and codex is API for these models.
 So it's first pre-trained on language and then I don't know if you can say fine tuned because there's a lot of code, but it's language and code.
 It's language and code.
 It's also optimized for various things like, let's say, low latency and so on.
 Codex is the API, similar to GPT-3.
 We expect that there will be proliferation of the potential products that can use coding capabilities.
 And I can speak about it in a second.
 Copilot is the first product developed by GitHub.
 So as we're building models, we wanted to make sure that these models are useful.
 And we work together with GitHub on building the first product.
 Copilot is actually as you code, it suggests you code completions.
 And we have seen in the past there are like various tools that can suggest how to like a few characters of the code or the line of code.
 The thing about Copilot is it can generate 10 lines of code.
 It's often the way how it works is you often write in the comment what you want to happen because people in comments, they describe what happens next.
 So these days when I code, instead of going to Google to search for the appropriate code to solve my problem, I say, oh, for this array, could you smooth it?
 And then, you know, it imports some appropriate libraries and it uses NumPy convolution or so that I was not even aware that exists.
 And it does the appropriate thing.
 So you write a comment, maybe the header of a function, and it completes the function.
 Of course, you don't know what is the space of all the possible small programs it can generate.
 What are the failure cases? How many edge cases? How many subtle errors there are?
 How many big errors there are? It's hard to know.
 But the fact that it works at all in a large number of cases is incredible.
 It's like it's a kind of search engine into code that's been written on the Internet.
 Correct. So for instance, when you search things online,
 then usually you get to some particular case, like if you go to Stack Overflow,
 people describe one particular situation and then they seek for a solution.
 But in the case of Copilot, it's aware of your entire context.
 And in context, it's, oh, these are the libraries that they are using.
 That's the set of the variables that is initialized.
 And on the spot, it can actually tell you what to do.
 So the interesting thing is, and we think that the Copilot is one possible product using codecs,
 but there is a place for many more.
 So internally, we tried out to create other fun products.
 So it turns out that a lot of tools out there, let's say Google Calendar or Microsoft Word or so,
 they all have internal API to build plugins around them.
 So there is a way in a sophisticated way to control Calendar or Microsoft Word.
 Today, if you want more complicated behaviors from these programs,
 you have to add a new button for every behavior.
 But it is possible to use codecs and tell, for instance, to Calendar,
 could you schedule an appointment with Blacks next week after 2 p.m.
 and it writes corresponding piece of code.
 And that's the thing that actually you want.
 So interesting. So you figure out is there's a lot of programs with which you can interact through code.
 And so there you can generate that code from natural language. That's fascinating.
 And that's somewhat like also closest to what was the promise of Siri or Alexa.
 So previously, all these behaviors, they were hard coded.
 And it seems that codecs on the fly can pick up the API of, let's say, given software.
 And then it can turn language into use of this API.
 So without hard coding, you can find it can translate to machine language.
 So, for example, this would be really exciting for me like for Adobe products like Photoshop,
 which I think ActionScript, I think there's a scripting language that communicates with them.
 Same with Premiere.
 And you could imagine that that allows even to do coding by voice on your phone.
 So, for instance, in the past, as of today, I'm not editing Word documents on my phone
 because it's just the keyboard is too small.
 But if I would be able to tell to my phone, make the header large,
 then move the paragraphs around, and that's actually what I want.
 So I can tell you one more cool thing, or even how I'm thinking about codecs.
 So if you look actually at the evolution of computers,
 we started with very primitive interfaces, which is a punch card.
 And punch card, essentially, you make holes in the plastic card to indicate zeros and ones.
 And during that time, there was a small number of specialists who were able to use computers.
 And by the way, people even suspected that there is no need for many more people to use computers.
 But then we moved from punch cards to, at first, assembly, then C.
 And these programming languages, they were slightly higher level.
 They allowed many more people to code.
 And they also led to more of a proliferation of technology.
 And further on, there was a jump to, say, from C++ to Java and Python.
 And every time it has happened, more people are able to code, and we build more technology.
 And it's even hard to imagine now if someone will tell you that you should write code in assembly
 instead of, let's say, Python or Java or JavaScript.
 And codecs is yet another step toward bringing computers closer to humans,
 such that you communicate with a computer with your own language rather than with a specialized language.
 And I think that it will lead to an increase of number of people who can code.
 Yeah, and the kind of technologies that those people will create, it's innumerable.
 It could be a huge number of technologies we're not predicting at all.
 Because that's less and less a requirement of having a technical mind, a programming mind.
 You're now opening it to the world of other kinds of minds, creative minds, artistic minds, all that kind of stuff.
 I would like, for instance, biologists who work on DNA to be able to program and not need to spend a lot of time learning it.
 And I believe that's a good thing to the world.
 And I would actually add that.
 So at the moment, I'm a managing codecs team and also language team.
 And I believe that there is like a plenty of brilliant people out there and they should apply.
 Oh, OK. Yeah, awesome. So what's the language in the codecs?
 So those are kind of their overlapping teams.
 It's like GPT, the raw language, and then the codecs is applied to programming.
 Correct. And they are quite intertwined.
 There are many more teams involved making these models extremely efficient and deployable.
 For instance, there are people who are working to make our data centers amazing.
 Or there are people who work on putting these models into production or even pushing it at the very limit of the scale.
 So all aspects from the infrastructure to the actual machine learning.
 So I'm just saying there are multiple teams.
 While the team working on codecs and language, I guess I'm directly managing them.
 I would love to hire more.
 If you're interested in machine learning, this is probably one of the most exciting problems and systems to be working on.
 It's pretty cool.
 The program synthesis, the generating of programs is very interesting.
 Very interesting problem that has echoes of reasoning and intelligence in it.
 And I think there's a lot of fundamental questions that you might be able to sneak up to by generating programs.
 Yeah, one more exciting thing about the programs is that, so I said that in case of language,
 that one of the troubles is even evaluating language.
 So when the things are made up, you need somehow either a human to say that this doesn't make sense or so.
 In case of program, there is one extra lever that we can actually execute programs and see what they evaluate.
 So that process might be somewhat more automated in order to improve the qualities of generations.
 That's fascinating. So like the wow, that's really interesting.
 So for language, the simulation to actually execute it as a human mind.
 Yeah. For programs, there is a computer on which you can evaluate it.
 Wow. That's a brilliant little insight that the thing compiles and runs.
 That's first. And second, you can evaluate, do automated unit testing.
 And in some sense, it seems to me that we'll be able to make a tremendous progress.
 We are in the paradigm that there is way more data.
 There is like a transcription of millions of software engineers.
 Yeah. So I mean, you just mean because I was going to ask you about reliability.
 The thing about programs is you don't know if they're going to like a program that's controlling a nuclear power plant has to be very reliable.
 So I wouldn't start with controlling nuclear power plant.
 Maybe one day. But that's not actually that's not on the current roadmap.
 That's not the step one. You know, it's the Russian thing.
 You just want to go to the most powerful destructive thing right away run by JavaScript.
 But I got you. So it's a lower impact.
 But nevertheless, what you make me realize, it is possible to achieve some levels of reliability by doing testing.
 And you could you could imagine that, you know, maybe there are ways for model to write even code for testing itself and so on.
 And there exist ways to create a feedback loops that the model could keep on improving.
 By writing programs that generate tests for instance, for instance.
 And that's how we get consciousness, because it's meta compression.
 That's what you're going to write. That's the comment.
 That's the prompt that generates consciousness. Compressor of compressors.
 You just write that. Do you think the code that generates consciousness will be simple?
 So let's see. I mean, ultimately, the core idea behind will be simple, but there will be also decent amount of engineering involved.
 Like in some sense, it seems that, you know, spreading these models on many machines, it's not that trivial.
 And we find all sorts of innovations that make our models more efficient.
 I believe that first models that I guess are conscious are like a truly intelligent.
 They will have all sorts of tricks.
 But then again, there's a certain argument that maybe the tricks are temporary things.
 Yeah, they might be temporary things. And in some sense, it's also even important to know that even the cost of a trick.
 So sometimes people are eager to put the trick while forgetting that there is a cost of maintenance.
 Or like long term cost. Long term cost or maintenance or maybe even flexibility of code to actually implement new ideas.
 So even if you have something that gives you 2x, but it requires, you know, 1000 lines of code, I'm not sure if it's actually worth it.
 So in some sense, you know, if it's five lines of code and 2x, I would take it.
 And we see many of this, but also, you know, that requires some level of, I guess, lack of attachment to code that we are willing to remove it.
 Yeah. So you led the OpenAI robotics team.
 Can you give an overview of the cool things you were able to accomplish? What are you most proud of?
 So when we started robotics, we knew that actually reinforcement learning works and it is possible to solve fairly complicated problems.
 Like, for instance, AlphaGo is an evidence that it is possible to build superhuman Go players.
 Dota2 is an evidence that it's possible to build superhuman agents playing Dota.
 So I asked myself a question, you know, what about robots out there?
 Could we train machines to solve arbitrary tasks in the physical world?
 And the approach was, I guess, let's pick a complicated problem that if we would solve it, that means that we made some significant progress, the domain, and then we went after the problem.
 So we noticed that actually the robots out there, they are kind of at the moment optimized per task.
 So you can have a robot that it's like if you have a robot opening a battle, it's very likely that the end factor is a battle opener.
 And in some sense, that's a hack to be able to solve a task, which makes any task easier.
 And I asked myself, so what would be a robot that can actually solve many tasks?
 And we concluded that human hands have such a quality that indeed they are, you know, you have five kind of tiny arms attached.
 Individually, they can manipulate a pretty broad spectrum of objects.
 So we went after a single hand like trying to solve Rubik's Cube single handed.
 We picked this task because we thought that there is no way to hard code it.
 And it's also we picked a robot on which it would be hard to hard code it.
 And we went after the solution such that it could generalize to other problems.
 And just to clarify, it's one robotic hand solving the Rubik's Cube.
 The hard part is in the solution to the Rubik's Cube is the manipulation of the of like having it not fall out of the hand,
 having it use the five baby arms to what is it like rotate different parts of the Rubik's Cube to achieve the solution.
 Correct. Yeah. So what was the hardest part about that?
 What was the approach taken there? What are you most proud of?
 Obviously, we have like a strong belief in reinforcement learning.
 And, you know, one path it is to do reinforcement learning in the real world.
 Other path is to the simulation.
 In some sense, the tricky part about the real world is at the moment our models, they require a lot of data.
 There is essentially no data.
 And I think we decided to go through the path of the simulation.
 And in simulation, you can have infinite amount of data.
 The tricky part is the fidelity of the simulation and also can you in simulation represent everything that you represent otherwise in the real world?
 And, you know, it turned out that that, you know, because there is lack of fidelity, it is possible to what we what we arrived at is training a model that doesn't solve one simulation,
 but it actually solves the entire range of simulations, which vary in terms of like what's the exactly the friction of the cube or weight or so.
 And the single AI that can solve all of them ends up working well with the reality.
 How do you generate the different simulations?
 So, you know, there's plenty of parameters out there.
 We just pick them randomly and an in simulation model just goes for thousands of years and keeps on solving Rubik's Cube in each of them.
 And the thing is that neural network that we used, it has a memory and as it presses, for instance, the side of the of the cube,
 it can sense, oh, that's actually this side was difficult to press.
 I should press it stronger and throughout this process kind of learns even how to how to solve this particular instance of the Rubik's Cube back even mass.
 It's kind of like, you know, sometimes when you go to a gym and after after bench press, you try to leave the class and you kind of forgot.
 And your head goes like right away because kind of you got used to maybe different weight and it takes a second to adjust.
 Yeah. And this kind of of a memory, the model gained through the process of interacting with the cube in the simulation.
 I appreciate you speaking to the audience with the bench press, all the bros in the audience probably working out right now.
 There's probably somebody listening to this actually doing bench press.
 So maybe put the bar down and pick up the water bottle and you'll know exactly what what Jack is talking about.
 OK. So what what was the hardest part of getting the whole thing to work?
 So the hardest part is at the moment when it comes to physical work, when it comes to robots, they require maintenance.
 It's hard to replicate them a million times. It's also it's hard to replay things exactly.
 I remember this situation that one guy at our company, he had like a model that performs way better than other models in solving Rubik's Cube.
 And, you know, we kind of didn't know what's going on, why is that?
 And it turned out that, you know, he was running it from his laptop that had better CPU or better, maybe local GPU as well.
 And because of that, there was less of a latency and the model was the same.
 And that actually made solving Rubik's Cube more reliable.
 So in some sense, there might be some subtle bugs like that when it comes to running things in the real world.
 Even hinting on that, you could imagine that the initial models you would like to have models, which are insanely huge neural networks,
 and you would like to give them even more time for thinking.
 And when you have these real time systems, then you might be constrained actually by the amount of latency.
 And ultimately, I would like to build a system that is worth for you to wait five minutes because it gives you the answer that you are willing to wait for five minutes.
 So latency is a very unpleasant constraint under which to operate.
 Correct. And also there is actually one more thing which is tricky about robots.
 There is actually not much data.
 So the data that I'm speaking about would be data of first person experience from the robot and like a gigabytes of data like that.
 If we would have gigabytes of data like that of robot solving various problems, it would be very easy to make a progress on robotics.
 And you can see that in case of text or code, there is a lot of data like a first person perspective data on writing code.
 Yeah. So you had this you mentioned this really interesting idea that if you were to build like a successful robotics company,
 so open as mission is much bigger than robotics. This is one of the one of the things you worked on.
 But if it was a robotics company that you wouldn't so quickly dismiss supervised learning.
 Correct. That you would build a robot that was perhaps what like an empty shell like dumb and they would operate under teleoperation.
 So you would invest. That's just one way to do it. Invest in human super like direct human control of the robots as it's learning.
 And over time, add more and more automation. That's correct.
 So let's say that's how I would build a robotics company today if I would be building robotics company,
 which is spent 10 million dollars or so recording human trajectories controlling a robot.
 After you find a thing that the robot should be doing that there's a market fit for like that you can make a lot of money with that product.
 Correct. Correct. Yeah.
 So I would record data and then I would essentially train supervised learning model on it. That might be the path today.
 Long term, I think that actually what is needed is to train powerful models over video.
 So you have seen maybe a models that can generate images like Dali and people are looking into models generating videos.
 They're like a body's algorithmic questions even how to do it. And it's unclear if there is enough compute for this purpose.
 But I suspect that the models that which would have a level of understanding of video,
 same as GPT has a level of understanding of text, could be used to train robots to solve tasks.
 They would have a lot of common sense.
 If one day I'm pretty sure one day there will be a robotics company by robotics company.
 I mean, the primary source of income is from robots that is worth over one trillion dollars.
 What do you think that company will do? I think sell driving cars.
 It's interesting because my mind went to personal robotics robots in the home.
 It seems like there's much more market opportunity there. I think it's very difficult to achieve.
 I mean, this this this might speak to something important, which is I understand self driving much better than understand robotics in the home.
 So I understand how difficult it is to actually solve self driving to a level,
 not just the actual computer vision and the control problem and just the basic problem self driving,
 but creating a product that would undeniably be that would cost less money.
 Like it would save you a lot of money, like orders of magnitude, less money that could replace Uber drivers, for example.
 So car sharing is autonomous.
 It creates a similar or better experience in terms of how quickly you get from A to B or just whatever the pleasantness of the experience,
 the efficiency of the experience, the value of the experience. And at the same time, the car itself costs cheaper.
 I think that's very difficult to achieve. I think there's a lot more low hanging fruit in the home.
 That could be. I also want to give you a perspective on how challenging it would be at home or like it maybe kind of depends on the exact problem that you'd be solving.
 Like if we're speaking about these robotic arms and hence these things, they cost tens of thousands of dollars or maybe 100 K.
 And, you know, maybe obviously maybe there would be economy of scale.
 These things would be cheaper. But actually, for any household to buy, the price would have to go down to maybe thousand bucks.
 Yeah, I personally think that so self-driving car provides a clear service.
 I don't think robots in the home, there'll be a trillion dollar company will just be all about service, meaning it will not necessarily be about like a robotic arm that helps you.
 I don't know, open a bottle or wash dishes or any of that kind of stuff.
 It has to be able to take care of that whole the therapist thing you mentioned.
 I think that's of course there's a line between what is a robot and what is not like doesn't really need a body.
 But, you know, some AI system with some embodiment, I think.
 So the tricky part, when you think actually what's the difficult part is when the robot has like when there is a diversity of the environment with which the robot has to interact, that becomes hard.
 So, you know, on one spectrum, you have industrial robots as they are doing over and over the same thing.
 It is possible to some extent to prescribe the movements.
 And with very small amount of intelligence, the movement can be repeated millions of times.
 There are also, you know, various pieces of industrial robots where it becomes harder and harder.
 Like, for instance, in case of Tesla, it might be a matter of putting a rock inside of a car.
 And, you know, because the rock kind of moves around, it's not that easy.
 It's not exactly the same every time.
 That's being the case that you need actually humans to do it.
 And while, you know, welding cars together, it's a very repetitive process.
 Then in case of self-driving itself, the difficulty has to do with the diversity of the environment, but still the car itself.
 The problem that you are solving is you try to avoid even interacting with things.
 You are not touching anything around because touching itself is hard.
 And then if you would have in the home robot that, you know, has to touch things and like if these things, they change the shape, if there is a huge variety of things to be touched,
 then that's difficult. If you are speaking about the robot, which there is, you know, head that is smiling in some way with cameras, that it doesn't, you know, touch things, that's relatively simple.
 OK, so to both agree and to push back.
 So you're referring to touch like soft robotics, like the actual touch.
 But I would argue that you could formulate just basic interaction between like non-contact interaction is also a kind of touch.
 And that might be very difficult to solve. That's the basic this not disagreement, but that's the basic open question to me with self-driving cars and this agreement with Elon,
 which is how much interaction is required to solve self-driving cars, how much touch is required.
 You said that in your intuition, touch is not required in my intuition to create a product that's compelling to use.
 You're going to have to interact with pedestrians, not just avoid pedestrians, but interact with them.
 When we drive around in major cities, we're constantly threatening everybody's life with our movements.
 And that's how they respect us. There's a game theory going on with pedestrians.
 And I'm afraid you can't just formulate autonomous driving as a collision avoidance problem.
 So I think it goes beyond like a collision avoidance is the first order approximation.
 But then at least in case of test, they are gathering data from people driving their cars.
 And I believe that's an example of supervised data that they can train their models on.
 And they are doing it, which can give a model this like another level of behavior that is needed to actually interact with the real world.
 Yeah, it's interesting how much data is required to achieve that.
 What do you think of the whole Tesla autopilot approach, the computer vision based approach with multiple cameras and as a data engine, it's a multitask,
 multi-headed neural network, and it's this fascinating process of similar to what you're talking about with the robotics approach,
 which is you deploy a neural network and then there's humans that use it and then it runs into trouble in a bunch of places and that stuff is sent back.
 So the deployment discovers a bunch of edge cases and those edge cases are sent back for supervised annotation, thereby improving the neural network.
 And that's deployed again, it goes over and over until the network becomes really good at the task of driving, becomes safer and safer.
 What do you think of that kind of approach to robotics?
 I believe that's the way to go.
 So in some sense, even when I was speaking about collecting trajectories from humans, that's like a first step.
 And then you deploy the system and then you have humans revising all the issues.
 And in some sense, like this approach converges to a system that doesn't make mistakes because for the cases where there are mistakes,
 you gather data how to fix them and the system will keep on improving.
 So there's a very, to me, difficult question of how hard that, you know, how long that converging takes, how hard it is.
 The other aspect of autonomous vehicles probably applies to certain robotics applications is society, right?
 They put as the quality of the system converges.
 So one, there's a human factors perspective, a psychology of humans being able to supervise those, even with teleoperation, those robots.
 And the other society willing to accept robots.
 Currently, society is much harsher on self-driving cars than it is on human driven cars in terms of the expectation of safety.
 So the bar is set much higher than for humans.
 And so if there's a death in an autonomous vehicle, that's seen as much more dramatic than a death in a human driven vehicle.
 Part of the success of deployment of robots is figuring out how to make robots part of society, both on the just the human side,
 on the media journalist side and also on the policy government side.
 And that seems to be maybe you can put that into the objective function to optimize.
 But that is that is definitely a tricky one.
 And I wonder if that is actually the trickiest part for self-driving cars or any system that's safety critical.
 It's not the algorithm. It's the society accepting it.
 Yeah, I would say I believe that the part of the process of deployment is actually showing people that the given things can be trusted.
 Yeah. And trust is also like a glass that is actually really easy to crack it.
 Yeah. And damage it.
 And I think that's actually very common with innovation, that there is some resistance toward it.
 Yeah. And it's just a natural progression.
 So sometimes people have to keep on proving that indeed the systems are worth being used.
 And I would say I also found out that often the best way to convince people is by letting them experience it.
 Yeah, absolutely. That's the case with Tesla Autopilot, for example.
 That's the case with basically robots in general.
 It's kind of funny to hear people talk about robots like there's a lot of fear, even with like legged robots.
 But when they actually interact with them, there's joy.
 I love interacting with them. And the same with the car.
 With a robot, if it starts being useful, I think people immediately understand.
 And if the product is designed well, they fall in love. You're right.
 It's actually even similar when I'm thinking about Copilot, the GitHub Copilot.
 There was a spectrum of responses that people had.
 And ultimately, the important piece was to let people try it out.
 And then many people just loved it.
 Especially like programmers.
 Yeah, programmers. But like some of them, you know, they came with a fear.
 Yeah.
 But then you try it out and you think, actually, that's cool.
 And you know, you can try to resist the same way as you could resist moving from punch cards to, let's say, C++ or so.
 And it's a little bit futile.
 So we talked about generation of program, generation of language, even self-supervised learning in the visual space for robotics and then reinforcement learning.
 What do you and like this whole beautiful spectrum of AI, do you think is a good benchmark, a good test to strive for to achieve intelligence?
 That's a strong test of intelligence.
 You know, it started with Alan Turing and the Turing test.
 Maybe you think natural language conversation is a good test.
 So, you know, it would be nice if, for instance, machine would be able to solve Riemann hypothesis in math.
 That would be, I think that would be very impressive.
 So theorem proving, is that to you proving theorems is a good, oh, like one thing that the machine did, you would say, damn.
 Exactly.
 OK.
 That would be quite impressive.
 I mean, the tricky part about the benchmarks is, you know, as we are getting closer with them, we have to invent new benchmarks.
 There is actually no ultimate benchmark out there.
 Yes. My thought with the Riemann hypothesis would be the moment the machine proves it would say, OK, well, then the problem was easy.
 That's what happens.
 And I mean, in some sense, that's actually what happens over the years in AI that like we get used to things very quickly.
 You know something I talked to Rodney Brooks.
 I don't know if you know that is he called Alpha Zero homework problem because you're saying like there's nothing special about it.
 It's not a big leap. And I didn't.
 Well, he's coming from one of the aspects that we refer to as he was part of the founding of iRobot, which deployed now tens of millions of robot in the home.
 So if you see robots that are actually in the homes of people as the legitimate instantiation of artificial intelligence, then yes, maybe an AI that plays a silly game like going chess is not a real accomplishment.
 To me, it's a fundamental leap. But I think we as humans then say, OK, well, then that that game of chess or go wasn't that difficult compared to the thing that's currently unsolved.
 So my intuition is that from perspective of the evolution of these AI systems, we'll at first see the tremendous progress in digital space.
 And the main thing about this space is also that you can everything is that there is a lot of recorded data, plus you can very rapidly deploy things to billions of people.
 Well, in case of physical space, the deployment part takes multiple years.
 You have to manufacture things and delivering it to actual people is very hard.
 So I'm expecting that the first and that prices in digital space of goods, they would go down to the let's say marginal costs are zero.
 And also the question is how much of our life will be in digital?
 Because it seems like we're heading towards more and more of our lives being in the digital space.
 So like innovation in the physical space might become less and less significant.
 Like why do you need to drive anywhere if most of your life is spent in virtual reality?
 I still would like, you know, to at least at the moment, my impression is that I would like to have a physical contact with other people.
 And that's very important to me. We don't have a way to replicate it in the computer.
 It might be the case that over the time it will change like in 10 years from now.
 Why not have like an arbitrary infinite number of people you can interact with?
 Some of them are real, some are not with arbitrary characteristics that you can define based on your own preferences.
 I think that's maybe where we are heading and maybe I'm resisting the future.
 Yeah, I'm telling you, I if I got to choose if I could live in Elder Scrolls Skyrim versus the real world, I'm not so sure I would stay with the real world.
 Yeah, I mean, the question is, so will VR be sufficient to get us there or do you need to, you know, plug electrodes in the brain?
 And it would be nice if these electrodes wouldn't be invasive.
 Yeah, or at least like provably non-destructive.
 But in the digital space, do you think we'll be able to solve the Turing test, the spirit of the Turing test,
 which is do you think we'll be able to achieve compelling natural language conversation between people,
 like have friends that are AI systems on the Internet?
 I totally think it's doable.
 Do you think the current approach of GPT will take us there?
 Yes, so there is, you know, the part of at first learning all the content out there.
 And I think that still system should keep on learning as it speaks with you.
 Yeah.
 And I think that should work.
 The question is how exactly to do it.
 And, you know, obviously we have people at OpenAI asking these questions
 and kind of at first pre-training on all existing content is like a backbone and is a decent backbone.
 Do you think AI needs a body connecting to our robotics question to truly connect with humans?
 Or can most of the connection be in the digital space?
 So let's see. We know that there are people who met each other online and they fell in love.
 Yeah.
 So it seems that it's conceivable to establish connection, which is purely through Internet.
 Of course, it might be more compelling the more modalities you add.
 So it would be like you're proposing like a Tinder, but for AI.
 You like swipe right and left and half the systems are AI and the other is humans and you don't know which is which.
 That would be our formulation of Turing test.
 The moment AI is able to achieve more swipe right or left, whatever, the moment it's able to be more attractive than other humans, it passes the Turing test.
 Then you pass the Turing test in attractiveness.
 That's right. Well, no, like attractiveness just to clarify.
 There will be conversation.
 Not just visual. Right. Right. It's also attractiveness with wit and humor and whatever.
 Whatever makes conversations pleasant for humans.
 OK. All right. So you're saying it's possible to achieve in the digital space.
 In some sense, I would almost ask the question, why wouldn't that be possible?
 Right. Well, I have this argument with my dad all the time. He thinks that touch and smell are really important.
 So they can be very important. And I'm saying the initial systems, they won't have it.
 Still, I wouldn't like their people being born without the senses.
 And, you know, I believe that they can still fall in love and have meaningful life.
 Yeah. I wonder if it's possible to go close to all the way by just training on transcripts of conversations.
 Like, I wonder how far that takes us.
 So I think that actually still you want images like I would like.
 So I don't have kids, but like I could imagine having a tutor.
 It has to see, you know, kids drawing some pictures on a paper and facial expressions, all that kind of stuff.
 We use dogs and humans use their eyes to communicate with each other.
 I think that's a really powerful mechanism of communication.
 Body language, too, that words are much lower bandwidth.
 And for body language, we still, you know, we kind of have a system that displays an image of its facial expression on the computer.
 It doesn't have to move, you know, mechanical pieces or so.
 So I think that there is like kind of a progression.
 You can imagine that text might be the simplest to tackle, but this is not the complete human experience at all.
 You expand it to, let's say, images both for input and output.
 And what you describe is actually the final, I guess, frontier.
 What makes us human? The fact that we can touch each other or smell or so.
 And it's the hardest from perspective of data and deployment.
 And I believe that these things might happen gradually.
 Are you excited by that possibility, this particular application of human to AI friendship and interaction?
 So let's see.
 Like, would you do you look forward to a world you said you're living with a few folks and you're very close friends with them?
 Do you look forward to a day where one or two of those friends are AI systems?
 So if the system would be truly wishing me well, rather than being in the situation that it optimizes for my time to interact with the system.
 The line between those is it's a gray, it's a gray area.
 I think that's the distinction between love and possession.
 And these things, they might be often correlated for humans, but it's like you might find that there are some friends with whom you haven't spoken for months.
 Yeah.
 And then, you know, you pick up the phone as the time hasn't passed.
 They are not holding to you.
 And I wouldn't like to have AI system that, you know, it's trying to convince me to spend time with it.
 I would like the system to optimize for what I care about and help me in achieving my own goals.
 But there's some, I mean, I don't know.
 There's some manipulation, there's some possessiveness, there's some insecurities, there's fragility.
 All those things are necessary to form a close friendship over time, to go through some dark shit together, some bliss and happiness together.
 I feel like there's a lot of greedy, self-centered behavior within that process.
 My intuition, but I might be wrong, is that human-computer interaction doesn't have to go through a computer being greedy, possessive, and so on.
 It is possible to train systems, maybe, that they actually, you know, they are, I guess, prompted or fine-tuned or so to truly optimize for what you care about.
 And you could imagine that, you know, the way how the process would look like is at some point, we as humans, we look at the transcript of the conversation or like an entire interaction.
 And we say, actually, here there was more loving way to go about it.
 And we supervise system toward being more loving.
 Or maybe we train the system such that it has a reward function toward being more loving.
 Yeah. Or maybe the possibility of the system being an asshole and manipulative and possessive every once in a while is a feature, not a bug.
 Because some of the happiness that we experience when two souls meet each other, when two humans meet each other, is a kind of break from the assholes in the world.
 And so you need assholes in AI as well, because like it'll be like a breath of fresh air to discover an AI that the three previous AIs you had are too friendly.
 Or cruel or whatever. It's like some kind of mix. And then this one is just right. But you need to experience the full spectrum.
 Like I think you need to be able to engineer assholes.
 So let's see. Because there's some level to us being appreciated, to appreciate the human experience.
 We need the dark and the light. So that kind of reminds me.
 I met a while ago at the meditation retreat, one woman and a beautiful, beautiful woman.
 And she had a crutch. She had trouble walking on one leg. I asked her what has happened.
 And she said that five years ago she was in Maui, Hawaii.
 And she was eating a salad and some snail fell into the salad. And apparently there are neurotoxic snails over there.
 And she got into coma for a year.
 And apparently there is a high chance of event just dying. But she was in the coma.
 At some point she regained partially consciousness. She was able to hear people in the room.
 People behaved as she wouldn't be there.
 At some point she started being able to speak, but she was mumbling. Like barely able to express herself.
 At some point she got into wheelchair. Then at some point she actually noticed that she can move her toe.
 And then she knew that she will be able to walk. And then that's where she was five years after.
 And she said that since then she appreciates the fact that she can move her toe.
 And I was thinking, do I need to go through such experience to appreciate that I can move my toe?
 That's a really good story and a really deep example.
 And in some sense it might be the case that we don't see light if we haven't went through the darkness.
 But I wouldn't say that we should.
 We shouldn't assume that that's the case. We may be able to engineer shortcuts.
 Ilya had this belief that maybe one has to go for a week or six months to some challenging camp to just experience a lot of difficulties.
 And then comes back and actually everything is bright. Everything is beautiful.
 I'm with Ilya on this. It must be a Russian thing. Where are you from originally?
 I'm Polish.
 Polish. Okay.
 I'm tempted to say that explains a lot. But yeah, there's something about the Russian, the necessity of suffering.
 I believe suffering or rather struggle is necessary.
 I believe that struggle is necessary. I mean, in some sense, you even look at the story of any superhero in the movie.
 It's not that it was like everything goes easy, easy, easy, easy.
 I like how that's your ground truth is the story of superheroes.
 Okay. You mentioned that you used to do research at night and go to bed at like 6 a.m. or 7 a.m.
 I still do that often.
 What sleep schedules have you tried to make for a productive and happy life?
 Like, is there is there some interesting wild sleeping patterns that you engaged that you found that works really well for you?
 I tried at some point decreasing number of hours of sleep like a gradually.
 Like half an hour every few days to this.
 You know, I was hoping to just save time.
 That clearly didn't work for me. Like at some point there's like a phase shift and I felt tired all the time.
 You know, there was a time that I used to work during the nights.
 The nice thing about the nights is that no one disturbs you.
 And even I remember when I was meeting for the first time with Greg Brockman, his CTO and chairman of OpenAI.
 Our meeting was scheduled to 5 p.m. and I overstepped for the meeting.
 Overslept for the meeting. 5 p.m. Yeah.
 Now you sound like me. That's hilarious. Okay. Yeah.
 And at the moment, in some sense, my sleeping schedule also has to do with the fact that I'm interacting with people.
 I sleep without an alarm. So.
 So, yeah, the the team thing you mentioned extrovert thing, because most humans operate during a certain set of hours.
 You're forced to then operate at the same set of hours.
 But I'm not quite there yet.
 I found a lot of joy, just like you said, working through the night because it's quiet, because the world doesn't disturb you.
 And there's some aspect counter to everything you're saying.
 There's some joyful aspect to sleeping through the mess of the day because people are having meetings and sending e-mails and there's drama meetings.
 I can sleep through all the meetings.
 You know, I have meetings every day and they prevent me from having sufficient amount of time for focus work.
 And then I modified my calendar and I said that I'm out of office Wednesday, Thursday and Friday every day.
 And I'm having meetings only Monday and Tuesday.
 And that bus, the positively influenced my mood that I have literally like three days for fully focused work.
 Yeah. So there's better solutions to this problem than staying awake all night.
 OK. You've been part of development of some of the greatest ideas in artificial intelligence.
 What would you say is your process for developing good novel ideas?
 You have to be aware that clearly there are many other brilliant people around.
 So you have to ask yourself a question.
 Why the given idea, let's say, wasn't tried by someone else?
 And in some sense, it has to do with, you know, kind of simple.
 It might sound simple, but like thinking outside of the box.
 And what do I mean here?
 So, for instance, for a while, people in academia, they assumed that you have a fixed data set and then you optimize the algorithms in order to get the best performance.
 And that was so in great assumption that no one thought about training models on anti-internet or like that.
 Maybe some people thought about it, but it felt too many as unfair.
 And in some sense, that's almost like it's not my idea or so, but that's an example of breaking a typical assumption.
 So you want to be in the paradigm that you are breaking a typical assumption.
 In the context of the AI community, getting to pick your data set is cheating.
 Correct. And in some sense, so that was a that was assumption that many people had out there.
 And then if you free yourself from assumptions, then they are likely to achieve something that others cannot do.
 And in some sense, if you are trying to do exactly the same things as others, it's very likely that you're going to have the same results.
 Yeah. But there's also that kind of tension, which is asking yourself the question, why haven't others done this?
 Because, I mean, I get a lot of good ideas, but I think probably most of them suck when they meet reality.
 So actually, I think the other big piece is getting into a habit of generating ideas, training your brain to generating ideas and not even suspending judgment of the ideas.
 So in some sense, I noticed myself that even if I'm in the process of generating ideas, if I tell myself, oh, that was a bad idea,
 then that actually interrupts the process and I cannot generate more ideas because I'm actually focused on the negative part, why it won't work.
 But I created also an environment in the way that it's very easy for me to store new ideas.
 So, for instance, next to my bed, I have a voice recorder and it happens to me often.
 I wake up during the night and I have some idea. In the past, I was writing them down on my phone, but that means turning on the screen and that wakes me up.
 Or like pulling a paper, which requires turning on the light. These days, I just start recording it.
 What do you think? I don't know if you know who Jim Keller is.
 I know Jim Keller.
 He's a big proponent of thinking harder on a problem right before sleep so that he can sleep through it and solve it in his sleep or like come up with radical stuff in his sleep.
 He was trying to get me to do this.
 So it happened from my experience perspective.
 It happened to me many times during the high school days when I was doing mathematics that I had the solution to my problem as I woke up.
 At the moment, regarding thinking hard about the given problem is I'm trying to actually devote substantial amount of time to think about important problems, not just before the sleep.
 I am organizing the huge chunks of time such that I'm not constantly working on the urgent problems, but I actually have time to think about the important one.
 So you do it naturally. But his idea is that you kind of prime your brain to make sure that that's the focus.
 You know, oftentimes people have other worries in their life that's not fundamentally deep problems like, I don't know, just stupid drama in your life.
 And even at work, all that kind of stuff, he wants to kind of pick the most important problem that you're thinking about and go to bed on that.
 I think that's why I mean, the other thing that comes to my mind is also I feel the most fresh in the morning.
 So during the morning, I try to work on the most important things rather than just being pulled by urgent things or checking email or so.
 What do you do with the because I've been doing the voice recorder thing, too, but I end up recording so many messages.
 It's hard to organize. I have the same problem.
 Now I have heard that Google Pixel is really good in transcribing text and I might get a Google Pixel just for the sake of transcribing text.
 Yeah, people listening to this, if you have a good voice recorder suggestion that transcribes, please let me know.
 Some of it is this has to do with the OpenAI Codex, too.
 Like some of it is simply like the friction. I need apps that remove that friction between voice and the organization of the resulting transcripts and all that kind of stuff.
 But yes, you're right. Absolutely. Like during for me is walking sleep, too, but walking and running, especially running, get a lot of thoughts during running.
 And there's there's no good mechanism for recording thoughts.
 So one more thing that I do, I have a separate phone, which I which has no apps.
 Maybe it's like audible or let's say no one has this phone number, this kind of my meditation phone.
 Yeah. And I try to expand the amount of time that that's the phone that I'm having.
 It has also Google Maps if I need to go somewhere. And I also use this phone to write down ideas.
 Ah, that's a really good idea. That's a really good idea.
 I often actually what I end up doing is even sending a message from that phone to the other phone.
 So that's actually my way of recording messages or I just put them into notes.
 I love it. What advice would you give to a young person? High school, college about how to be successful.
 You've done a lot of incredible things in the past decade.
 So maybe maybe you have some something that might be something that might be something.
 I mean, it might sound like a simplistic or so, but I would say literally just follow your passion, double down on it.
 And if you don't know what's your passion, just figure out what could be what could be a passion.
 So this that might be an exploration.
 When I was in elementary school was math and chemistry.
 And I remember for some time I gave up on math because my school teacher, she told me that I'm dumb.
 And I guess maybe an advice would be just ignore people if they tell you that you're dumb.
 You're dumb. You mentioned something offline about chemistry and explosives.
 What was that about? So let's see. So a story goes like that.
 I got into chemistry. Maybe I was like a second grade of my elementary school, third grade.
 I started going to chemistry classes. I really love building stuff.
 And I did all the experiments that they describe in the book.
 I can you know how to create oxygen with vinegar and baking soda or so.
 So I did all the experiments and at some point I was, you know, so what's next? What can I do?
 And explosives they also it's like you have a clear reward signal, you know, if the thing worked or not.
 So I remember at first I got I got interested in producing hydrogen.
 That was kind of a funny experiment from school. You can just burn it.
 And then I moved to nitroglycerin. So that's also relatively easy to synthesize.
 I started producing essentially dynamite and detonating it with a friend.
 Remember there was a there was at first like maybe two attempts that I went with a friend to detonate what we built.
 And it didn't work out. And like a third time he was like, it won't work.
 Like, let's don't waste time. And we were I was carrying this this, you know, that tube with dynamite.
 I don't know, pound or so dynamite in my backpack.
 We're like riding on the bike to the edges of the city.
 Yeah. And attempt number three. This was be attempt number three.
 Attempt number three. And we dig a hole to put it inside.
 It actually had the, you know, electrical detonator.
 We draw a cable behind the tree.
 I even I never I haven't ever seen like an explosion before.
 So I thought that there would be a lot of sound.
 But, you know, we're like laying down and I'm holding the cable and the battery.
 At some point, you know, we kind of like a three to one. And I just connected it.
 And it felt like at the ground shake.
 It was like more like a sound. And then the soil started kind of lifting up and started falling on us.
 Yeah. Wow. And then the friends said, let's make sure next time we have helmets.
 But also, you know, I'm happy that nothing happened to me.
 It could have been the case that I lost the limbo or so.
 Yeah. But that's childhood of of an engineering mind with a strong reward signal of an explosion.
 I love it. My there's some aspect of chemists, the chemists.
 I know like my dad with plasma chemistry, plasma physics, he was very much into explosives, too.
 It's a worrying quality of people that work in chemistry that they love.
 I think it is that exactly is the the strong signal that the thing worked.
 There is no doubt. There's no doubt. There's some magic.
 It's almost like a reminder that physics works, that chemistry works. It's cool.
 It's almost like a little glimpse at nature that you yourself engineer.
 That's why I really like artificial intelligence, especially robotics, is you create a little piece of nature.
 And in some sense, even for me, with explosives, the motivation was creation rather than destruction.
 Yes, exactly. In terms of advice, I forgot to ask about just machine learning and deep learning for people who are specifically interested in machine learning.
 How would you recommend they get into the field?
 So I would say, re-implement everything. And also there is plenty of courses.
 So like from scratch?
 So on different levels of abstraction in some sense.
 But I would say re-implement something from scratch, re-implement something from a paper, re-implement something from podcasts that you have heard about.
 I would say that's a powerful way to understand things.
 So it's often the case that you read the description and you think you understand, but you truly understand once you build it.
 Then you actually know what really meant that in the description.
 Is there particular topics that you find people just fall in love with?
 I tend to really enjoy reinforcement learning because it's much easier to get to a point where you feel like you created something special.
 Like fun games kind of things that are rewarding.
 It's rewarding, yeah.
 As opposed to like re-implementing from scratch, more like supervised learning kind of things.
 So if someone would optimize for things to be rewarding, then it feels that the things that are somewhat generative, they have such a property.
 So you have, for instance, adversarial networks or you have just even generative language models.
 And you can even see internally, we have seen this thing with our releases.
 So we released recently two models.
 There is one model called Dali that generates images and there is other model called Clip that actually you provide various possibilities.
 What could be the answer to what is on the picture?
 And it can tell you which one is the most likely.
 And in some sense, in case of the first one, Dali, it is very easy for you to understand that actually there is magic going on.
 And in case of the second one, even though it is insanely powerful and people from the vision community, as they started probing it inside, they actually understood how far it goes.
 So it's difficult for a person at first to see how well it works.
 And that's the same, as you said, that in case of supervised learning models, you might not kind of see or it's not that easy for you to understand the strength.
 Even though you don't believe in magic, to see the magic.
 To see the magic, yes.
 It's generative. That's really brilliant.
 So anything that's generative, because then you are at the core of the creation.
 You get to experience creation without much effort, unless you have to do it from scratch.
 And it feels that humans are wired.
 There is some level of reward for creating stuff.
 Yeah.
 Of course, different people have a different weight on this reward.
 Yeah. In the big objective function.
 In the big objective function of a person.
 Of a person.
 You wrote that beautiful is what you intensely pay attention to.
 Even a cockroach is beautiful if you look very closely.
 Can you expand on this?
 What is beauty?
 So what I wrote here actually corresponds to my subjective experience that I had through extended periods of meditation.
 It's pretty crazy that at some point the meditation gets you to the place that you have really increased focus, increased attention.
 And then you look at the very simple objects that were all the time around you.
 You can look at the table or on the pen or at the nature.
 And you notice more and more details and it becomes very pleasant to look at it.
 And it once again, it kind of reminds me of my childhood.
 Like just pure joy of being.
 It's also, I have seen even the reverse effect that by default, regardless of what we possess, we very quickly get used to it.
 And you know, you can have a very beautiful house.
 And if you don't put sufficient effort, you're just going to get used to it.
 And it doesn't bring any more joy, regardless of what you have.
 Yeah.
 Well, I actually.
 I find that material possessions get in the way of that experience of pure joy.
 So I've always I've been very fortunate.
 To just find joy in simple things, just just like you're saying.
 Just like, I don't know, objects in my life, just stupid objects like this cup thing, you know, just objects.
 Sounds OK. I'm not being eloquent, but literally objects in the world.
 They're just full of joy because it's like I can't believe one.
 I can't believe that I'm fortunate enough to be alive, to experience these objects.
 And then, too, I can't believe humans are clever enough to have built these objects.
 The hierarchy of pleasure that that provides is infinite.
 I mean, even if you look at the cup of water, so, you know, you see first like a level of like a reflection of light.
 But then you think, no, man, there's like a trillions upon trillions of particles bouncing against each other.
 There is also the tension on the surface that, you know, if the back back could like a stand on it and move around.
 And you think it also has this like a magical property that as you decrease temperature, it actually expands in volume,
 which allows for the, you know, legs to freeze on the on the surface and at the bottom to have actually not freeze, which allows for life like a crazy.
 Yeah, you look in detail at some object and you think actually, you know, this table that was just the figment of someone's imagination at some point.
 And then there was like a thousands of people involved to actually manufacture it and put it here.
 And by default, no one cares.
 And then you can start thinking about evolution, how it all started from single cell organisms that led to this table.
 And these thoughts, they give me life appreciation. Yeah, and even lack of thoughts, just the pure raw signal also gives the life appreciation.
 See, the thing is, and then that's coupled for me with the sadness that the whole ride ends and perhaps is deeply coupled in that the fact that this experience, this moment ends gives it gives it an intensity that I'm not sure I would otherwise have.
 So in that same way, I try to meditate on my own death.
 Often. Do you think about your mortality? Are you afraid of death?
 So fear of death is like one of the most fundamental fears that each of us has.
 We might be not even aware of it. It requires to look inside to even recognize that it's out there.
 And there is still, let's say, this property of nature that if things would last forever, then they would be also boring to us.
 The fact that the things change in some way gives any meaning to them.
 I also found out that it seems to be very healing to people to have these short experiences, like, I guess, psychedelic experiences in which they experience death of self in which they let go of this fear and then maybe can even increase the appreciation of the moment.
 It seems that many people, they can easily comprehend the fact that their money is finite while they don't see that time is finite.
 I have this like a discussion with Ilya from time to time. He's like, you know, man, like the life will pass very fast. At some point I will be 40, 50, 60, 70 and then it's over.
 This is true, which also makes me believe that, you know, that every single moment it is so unique that should be appreciated.
 And it's also makes me think that I should be acting on my life because otherwise it will pass.
 I also like this framework of thinking from Jeff Bezos on regret minimization that like I would like if I will be at that deathbed to look back on my life and not regret that I haven't done something.
 It's usually you might regret that you haven't tried. I'm fine with failing.
 I haven't tried.
 I was in each year, eternal occurrence, tried to live a life that if you had to live it infinitely many times, that would be the you'd be OK with that kind of life.
 So try to live it optimally.
 I can say that it's almost like I'm.
 Unprecedented to me where I am in my life. I'm extremely grateful for actually people whom I met. I would say I think that I'm decent smart and so on.
 But I think that actually to great extent where I am has to do with the people who I met.
 Would you be OK if after this conversation you died.
 So if I'm dead, then it kind of I don't have a choice anymore.
 So in some sense, there's like a plenty of things that I would like to try out in my life.
 I feel that I'm gradually going one by one and I'm just doing them.
 I think that the least will be always infinite.
 Yeah, so might as well go today.
 Yeah, I mean, to be clear, I'm not looking forward to dying.
 I would say if there is no choice, I would accept it.
 But like in some sense, I'm if there would be a choice, if there would be possibility to leave, I would fight for living.
 I find. It's more honest and real to think about, you know, dying today at the end of the day.
 That seems to me to at least in my brain, more honest slap in the face as opposed to I still have 10 years like today.
 Then then I'm much more about appreciating the cup and the table and so on and less about like silly worldly accomplishments and all those kinds of things.
 We we we have in the company a person who, say, at some point found out that they have cancer.
 And that also gives you a huge perspective with respect to what matters now.
 And, you know, often people in situations like that, they conclude that actually what matters is human connection.
 And love and that's people conclude also if you have kids, kids, his family, you I think tweeted,
 we don't assign the minus infinity reward to our death.
 Such a reward would prevent us from taking any risk.
 We wouldn't be able to cross the road in fear of being hit by a car.
 So in the objective function, you mentioned fear of death might be fundamental to the human condition.
 So, as I said, let's assume that there are like a reward functions in our brain.
 And and the interesting thing is even realization how different reward functions can play with your behavior.
 As a matter of fact, I wouldn't say that you should assign infinite negative reward to anything because that messes up the math.
 The math doesn't work out. It doesn't work out.
 And as you said, even, you know, government or some insurance companies said they assign nine million dollars to human life.
 And I'm just saying it with respect to that might be a hard statement to ourselves,
 but in some sense that there is a finite value of our own life.
 I'm trying to put it from perspective of being less of being more ego less and realizing fragility of my own life.
 And in some sense, the fear of death might prevent you from acting because anything can cause death.
 Yeah. And I'm sure actually, if you were to put death in the objective function,
 there's probably so many aspects to death and fear of death and realization of death and mortality.
 There's just whole components of finiteness of not just your life, but every experience and so on that you're going to have to formalize mathematically.
 And also, you know, that might lead to you spending a lot of compute cycles on this like a deliberating this terrible future instead of experiencing now.
 And then in some sense, it's also kind of unpleasant simulation to run in your head.
 Yeah. Do you think there's an objective function that describes the entirety of human life?
 So, you know, usually the way you ask that is what is the meaning of life?
 Is there a universal objective functions that captures the why of life?
 So, yeah, I mean, I suspect that they will ask this question, but it's also a question that I asked myself many, many times.
 See, I can tell you a framework that I have these days to think about this question.
 So I think that fundamentally meaning of life has to do with some of our reward functions that we have in brain.
 And they might have to do with, let's say, for instance, curiosity or human connection, which might mean understanding others.
 It's also possible for a person to slightly modify their reward function.
 Usually they mostly stay fixed, but it's possible to modify reward function.
 And you can pretty much choose. So in some sense, the reward functions, optimizing reward functions, they will give you a life satisfaction.
 Is there some randomness in the function?
 I think when you are born, there is some randomness.
 Like you can see that some people, for instance, they they care more about building stuff.
 Some people care more about caring for others.
 Some people that there are all sorts of default reward functions.
 And then in some sense, you can ask yourself, what's this like?
 What is the satisfying way for you to go after this reward function?
 And you just go after this reward function.
 And, you know, some people ask, are these reward functions real?
 I almost think about it as.
 Let's say if you would have to discover mathematics in mathematics, you are likely to run into various objects like a complex numbers or differentiation, some other objects.
 And these are very natural objects that arise.
 And similarly, the reward functions that we are having in our brain, they are somewhat very natural that, you know, there is a reward function for for understanding like a comprehension.
 Yeah. Curiosity and so on.
 So in some sense, they are in the same way natural as their natural objects in mathematics.
 Interesting. So, you know, there's the the old sort of debate is mathematics invented or discovered.
 You're saying reward functions are discovered.
 So nature provides a sum you can still, let's say, expand it throughout the life.
 Some of the reward functions, they might be futile.
 Like, for instance, there might be a reward function, maximize amount of wealth.
 Yeah. And this is more like a learned reward function.
 And but we know also that some reward functions, if you optimize them, you won't be quite satisfied.
 Well, I don't know which part of your word function resulted in you coming today, but I am deeply appreciative that you did spend your valuable time with me.
 Wojciech is really fun talking to you.
 You're you're brilliant. You're a good human being.
 And it's an honor to meet you and an honor to talk to you.
 Thanks for talking to the brother. Thank you.
 Thanks a lot. I appreciated your questions.
 I had a lot of time being here.
 Thanks for listening to this conversation with Wojciech Sremba.
 To support this podcast, please check out our sponsors in the description.
 And now let me leave you with some words from Arthur C. Clark, who is the author of 2001, A Space Odyssey.
 It may be that our role on this planet is not to worship God, but to create him.
 Thank you for listening and hope to see you next time.