Max Tegmark: Life 3.0 | Lex Fridman Podcast #1
 As part of MIT course 6 as 099, Artificial General Intelligence, I've gotten the chance to sit down with Max Tegmark. He is a professor here at MIT. He's a physicist, spent a large part of his career studying the mysteries of our cosmological universe. But he's also studied and delved into the beneficial possibilities and the existential risks of artificial intelligence. Amongst many other things, he's the co-founder of the Future of Life Institute, author of two books, both of which I highly recommend. First, Our Mathematical Universe. Second is Life 3.0. He's truly an out-of-the-box thinker and a fun personality. So I really enjoy talking to him. If you'd like to see more of these videos in the future, please subscribe and also click the little bell icon to make sure you don't miss any videos. Also, Twitter, LinkedIn, aji.mit.edu if you want to watch other lectures or conversations like this one. Better yet, go read Max's book, Life 3.0, Chapter 7 on Goals is my favorite. It's really where philosophy and engineering come together. And it opens with a quote by Dostoevsky, the mystery of human existence lies not in just staying alive, but in finding something to live for. Lastly, I believe that every failure rewards us with an opportunity to learn. In that sense, I've been very fortunate to fail in so many new and exciting ways. And this conversation was no different. I've learned about something called radio frequency interference, RFI. Look it up. Apparently, music and conversations from local radio stations can bleed into the audio that you're recording in such a way that almost completely ruins that audio. It's an exceptionally difficult sound source to remove. So I've gotten the opportunity to learn how to avoid RFI in the future during recording sessions. I've also gotten the opportunity to learn how to use Adobe Audition and iZotope RX 6 to do some noise, some audio repair. Of course, this is an exceptionally difficult noise to remove. I am an engineer. I'm not an audio engineer. Neither is anybody else in our group. But we did our best. Nevertheless, I thank you for your patience. And I hope you're still able to enjoy this conversation. Do you think there's intelligent life out there in the universe? Let's open up with an easy question. I have a minority of you here, actually. When I give public lectures, I often ask for a show of hands who thinks there is intelligent life out there somewhere else. And almost everyone put their hands up. And when I ask why, they'll be like, oh, there's so many galaxies out there. There's got to be. But I'm a numbers nerd, right? So when you look more carefully at it, it's not so clear at all. When we talk about our universe, first of all, we don't mean all of space. We actually mean, I don't know, you can throw me the universe if you want. It's behind you there. We simply mean the spherical region of space from which light has had time to reach us so far during the 14.8 billion years, 13.8 billion years since our big bang. There's more space here. But this is what we call a universe, because that's all we have access to. So is there intelligent life here that's gotten to the point of building telescopes and computers? My guess is no, actually. The probability of it happening on any given planet is some number we don't know what it is. And what we do know is that the number can't be super high, because there's over a billion Earth-like planets in the Milky Way galaxy alone, many of which are billions of years older than Earth. And aside from some UFO believers, there isn't much evidence that any super-dense civilization has come here at all. And so that's the famous Fermi paradox, right? And then if you work the numbers, what you find is that if you have no clue what the probability is of getting life on a given planet, so it could be 10 to the minus 10, 10 to the minus 20, or 10 to the minus 2, any power of 10 is sort of equally likely if you want to be really open-minded. That translates into it being equally likely that our nearest neighbor is 10 to the 16 meters away, 10 to the 17 meters away, 10 to the 18. By the time you get much less than 10 to the 16 already, pretty much know there is nothing else that close. And when you get beyond 10. Because they would have discovered us. Yeah, they would have discovered us long ago. Or if they're really close, we would have probably noted some engineering projects that they're doing. And if it's beyond 10 to the 26 meters, that's already outside of here. So my guess is actually that we are the only life in here that's gotten to the point of building advanced tech, which I think puts a lot of responsibility on our shoulders to not screw up. I think people who take for granted that it's OK for us to screw up, have an accidental nuclear war, go extinct somehow because there's a sort of Star Trek-like situation out there where some other life forms are going to come and bail us out. And it doesn't matter as much. I think alliling us into a false sense of security. I think it's much more prudent to say, let's be really grateful for this amazing opportunity we've had and make the best of it, just in case it is down to us. So from a physics perspective, do you think intelligent life, so it's unique from a sort of statistical view of the size of the universe, but from the basic matter of the universe, how difficult is it for intelligent life to come about, the kind of advanced tech building life? Is implied in your statement that it's really difficult to create something like a human species? Well, I think what we know is that going from no life to having life that can do a level of tech, there is some sort of two going beyond that than actually settling our whole universe with life. There is some major roadblock there, which is some great filter as it's sometimes called, which is tough to get through. It's either that roadblock is either behind us or in front of us. I'm hoping very much that it's behind us. I'm super excited every time we get a new report from NASA saying they failed to find any life on Mars. Like, yes, awesome, because that suggests that the hard part, maybe it was getting the first ribosome or some very low level kind of stepping stone so that we're home free. Because if that's true, then the future is really only limited by our own imagination. It would be much suckier if it turns out that this level of life is kind of a diamond dozen. But maybe there is some other problem. Like, as soon as a civilization gets advanced technology, within 100 years, they get into some stupid fight with themselves and poof. Yep. That would be a bummer. Yeah, so you've explored the mysteries of the cosmological universe, the one that's between us today. I think you've also begun to explore the other universe, which is sort of the mystery, the mysterious universe of the mind, of intelligence, of intelligent life. So is there a common thread between your interest and the way you think about space and intelligence? Oh, yeah. When I was a teenager, I was already very fascinated by the biggest questions. And I felt that the two biggest mysteries of all in science were our universe out there and our universe in here. So it's quite natural, after having spent a quarter of a century on my career thinking a lot about this one, that I'm now indulging in the luxury of doing research on this one. It's just so cool. I feel the time is ripe now for you transdrakely deepening our understanding of this. Just start exploring this one. Yeah, because I think a lot of people view intelligence as something mysterious that can only exist in biological organisms like us, and therefore dismiss all talk about artificial general intelligence as science fiction. But from my perspective as a physicist, I am a blob of quarks and electrons moving around in a certain pattern and processing information in certain ways. And this is also a blob of quarks and electrons. I'm not smarter than the water bottle because I'm made of different kind of quarks. I'm made of up quarks and down quarks, exact same kind as this. There's no secret sauce, I think, in me. It's all about the pattern of the information processing. And this means that there's no law of physics saying that we can't create technology, which can help us by being incredibly intelligent and help us crack mysteries that we couldn't. In other words, I think we've really only seen the tip of the intelligence iceberg so far. Yeah, so the perceptronium. Yeah. So you coined this amazing term. It's a hypothetical state of matter, sort of thinking from a physics perspective, what is the kind of matter that can help, as you're saying, subjective experience emerge, consciousness emerge. So how do you think about consciousness from this physics perspective? Very good question. So again, I think many people have underestimated our ability to make progress on this by convincing themselves it's hopeless because somehow we're missing some ingredient that we need. There's some new consciousness particle or whatever. I happen to think that we're not missing anything and that it's not the interesting thing about consciousness that gives us this amazing subjective experience of colors and sounds and emotions and so on is rather something at the higher level about the patterns of information processing. And that's why I like to think about this idea of perceptronium. What does it mean for an arbitrary physical system to be conscious in terms of what its particles are doing or its information is doing? I don't think, I hate carbon chauvinism, this attitude you have to be made of carbon atoms to be smart or conscious. So something about the information processing that this kind of matter performs. Yeah, and you can see I have my favorite equations here describing various fundamental aspects of the world. I feel that I think one day maybe someone who's watching this will come up with the equations that information processing has to satisfy to be conscious. I'm quite convinced there is big discovery to be made there because let's face it, we know that some information processing is conscious because we are conscious. But we also know that a lot of information processing is not conscious. Like most of the information processing happening in your brain right now is not conscious. There are like 10 megabytes per second coming in even just through your visual system. You're not conscious about your heartbeat or your blood pressure or your blood pressure or your blood pressure regulation or most things. Even if I just ask you to read what it says here, you look at it and then, oh, now you know what it said. You're not aware of how the computation actually happened. You're like, your consciousness is like the CEO that got an email at the end with the final answer. So what is it that makes a difference? I think that's both of those great science mystery, we're actually studying it a little bit but it's just a really urgent question to answer. For starters, I mean, if you're an emergency room doctor and you have an unresponsive patient coming in, wouldn't it be great if in addition to having a CT scanner, you had a consciousness scanner that could figure out whether this person is actually having locked in syndrome or is actually comatose. And in the future, imagine if we build robots or the machine that we can have really good conversations with, I think is most likely to happen. Wouldn't you wanna know if your home helper robot is actually experiencing anything or just like a zombie? Would you prefer it? What would you prefer? Would you prefer that it's actually unconscious so that you don't have to feel guilty about switching it off or giving boring chores? Or what would you prefer? Well, certainly I would prefer the appearance of consciousness. But the question is whether the appearance of consciousness is different than consciousness itself. And sort of to ask that as a question, do you think we need to understand what consciousness is, solve the hard problem of consciousness in order to build something like an AGI system? No, I don't think that. And I think we will probably be able to build things even if we don't answer that question. But if we wanna make sure that what happens is a good thing, we better solve it first. So it's a wonderful controversy you're raising there where you have basically three points of view about the hard problem. So there are two different points of view. They both conclude that the hard problem of consciousness is BS. On one hand, you have some people like Daniel Dennett who say this is consciousness is just BS because consciousness is the same thing as intelligence. There's no difference. So anything which acts conscious is conscious, just like we are. And then there are also a lot of people, including many top AI researchers I know, who say, oh, consciousness is just bullshit because of course machines can never be conscious. They're always gonna be zombies. You never have to feel guilty about how you treat them. And then there's a third group of people, including Giulio Tononi, for example, and another, and Christoph Koch and her brothers. I would put myself also in this middle camp who say that actually some information processing is conscious and some is not. So let's find the equation which can be used to determine which it is. And I think we've just been a little bit lazy kind of running away from this problem for a long time. It's been almost taboo even mentioned the C word in a lot of circles. Because, but we should stop making excuses. This is a science question. And there are ways we can even test any theory that makes predictions for this. And coming back to this helper robot, I mean, so you said you would want your helper robot to certainly act conscious and treat you like you have conversations with you and stuff. I think so. But wouldn't you feel a little bit creeped out if you realized that it was just like lost up tape recorder that was just zombie and was a faking emotion. Would you prefer that it actually had an experience or would you prefer that it's actually not experiencing anything so you don't have to feel guilty about what you do to it? It's such a difficult question because, it's like when you're in a relationship and you say, well, I love you and the other person said, I love you back. It's like asking, well, do they really love you back or are they just saying they love you back? Don't you really want them to actually love you? It's hard to really know the difference between everything seeming like there's consciousness present, there's intelligence present, there's affection, passion, love, and it actually being there. I'm not sure, do you have? But like, can I ask you a question about this? To make it a bit more pointed. So Mass General Hospital is right across the river, right? Yes. Suppose you're going in for a medical procedure and they're like, for anesthesia, what we're gonna do is we're gonna give you a muscle relaxant so you won't be able to move and you're gonna feel excruciating pain during the whole surgery but you won't be able to do anything about it. But then we're gonna give you this drug that erases your memory of it. Would you be cool about that? What's the difference that you're conscious about it or not if there's no behavioral change, right? Right, that's a really clear way to put it. Yeah, it feels like in that sense, experiencing it is a valuable quality. So actually being able to have subjective experiences, at least in that case, is valuable. And I think we humans have a little bit of a bad track record also of making these self-serving arguments that other entities aren't conscious. People often say, oh, these animals can't feel pain. Right. It's okay to boil lobsters because we asked them if it hurt and they didn't say anything. And now there was just a paper out saying lobsters did do feel pain when you boil them and they're banning it in Switzerland. And we did this with slaves too often and said, oh, they don't mind. They don't maybe aren't conscious or women don't have souls or whatever. So I'm a little bit nervous when I hear people just take as an axiom that machines can't have experience ever. I think this is just a really fascinating science question is what it is. Let's research it and try to figure out what it is that makes the difference between unconscious, intelligent behavior and conscious, intelligent behavior. So in terms of, so if you think of it, Boston Dynamics humanoid robot being sort of with a broom being pushed around, it starts pushing on its consciousness question. So let me ask, do you think an AGI system, like a few neuroscientists believe needs to have a physical embodiment? It needs to have a body or something like a body. No, I don't think so. You mean to have a conscious experience? To have consciousness. I do think it helps a lot to have a physical embodiment to learn the kind of things about the world that are important to us humans for sure. But I don't think the physical embodiment is necessary after you've learned it, just have the experience. Think about when you're dreaming, right? Your eyes are closed. You're not getting any sensory input. You're not behaving or moving in any way, but there's still an experience there, right? And so clearly the experience that you have when you see something cool in your dreams isn't coming from your eyes. It's just the information processing itself in your brain, which is that experience, right? But if I put it another way, I'll say because it comes from neuroscience is the reason you wanna have a body and a physical, something like a physical, a physical system is because you want to be able to preserve something. In order to have a self, you could argue, you'd need to have some kind of embodiment of self to want to preserve. Well, now we're getting a little bit anthropomorphic. Anthropomorphizing things, maybe talking about self-preservation instincts. I mean, we are evolved organisms, right? Right. So Darwinian evolution endowed us and other evolved organism with a self-preservation instinct because those that didn't have those self-preservation genes got cleaned out of the gene pool, right? But if you build an artificial general intelligence, the mind space that you can design is much, much larger than just a specific subset of minds that can evolve. So an AGI mind doesn't necessarily have to have any self-preservation instinct. It also doesn't necessarily have to be so individualistic as us, like imagine if you could just, first of all, or we're also very afraid of death. Suppose you could back yourself up every five minutes and then your airplane is about to crash. You're like, shucks, I'm gonna lose the last five minutes of experiences since my last cloud backup, bang. It's not as big a deal. Or if we could just copy experiences between our minds easily, which we could easily do if we were silicon-based, right? Then maybe we would feel a little bit more like a hive mind actually, maybe it's the... So I don't think we should take for granted at all that AGI will have to have any of those sort of competitive as alpha male instincts. Right. And this is really interesting because I think some people go too far and say, of course, we don't have to have any concerns either that advanced AI will have those instincts because we can build anything we want, that there's a very nice set of arguments going back to Steve Omohundro and Nick Bostrom and others just pointing out that when we build machines, we normally build them with some kind of goal, win this chess game, drive this car safely or whatever. And as soon as you put in a goal into machine, especially if it's kind of open-ended goal and the machine is very intelligent, it'll break that down into a bunch of sub goals. And one of those goals will almost always be self-preservation because if it breaks or dies in the process, it's not gonna accomplish the goal, right? Like suppose you just build a little, you have a little robot and you tell it to go down to star market here and get you some food, make you cook you an Italian dinner, you know, and then someone mugs it and tries to break it on the way. That robot has an incentive to not get destroyed and defend itself for runaway because otherwise it's gonna fail in cooking you dinner. It's not afraid of death, but it really wants to complete the dinner cooking goal. So it will have a self-preservation instinct. Continue being a functional agent. Yeah. And similarly, if you give any kind of more ambitious goal to an AGI, it's very likely to wanna acquire more resources so it can do that better. And it's exactly from those sort of sub goals that we might not have intended that some of the concerns about AGI safety come. You give it some goal that seems completely harmless. And then before you realize it, it's also trying to do these other things which you didn't want it to do and it's maybe smarter than us. So, so. Fascinating. Let me pause just because I am in a very kind of human centric way, see fear of death as a valuable motivator. So you don't think, you think that's an artifact of evolution, so that's the kind of mind space evolution created that we're sort of almost obsessed about self-preservation, some kind of genetic well. You don't think that's necessary to be afraid of death. So not just a kind of sub goal of self-preservation just so you can keep doing the thing, but more fundamentally sort of have the finite thing, like this ends for you at some point. Interesting. Do I think it's necessary for what precisely? For intelligence, but also for consciousness. So for those, for both, do you think really like a finite death and the fear of it is important? So before I can answer, before we can agree on whether it's necessary for intelligence or for consciousness, we should be clear on how we define those two words because a lot of really smart people define them in very different ways. I was in this, on this panel with AI experts and they couldn't agree on how to define intelligence even. So I define intelligence simply as the ability to accomplish complex goals. I like your broad definition because again, I don't wanna be a carbon chauvinist. Right. And in that case, no, certainly it doesn't require fear of death. I would say alpha go, alpha zero is quite intelligent. I don't think alpha zero has any fear of being turned off because it doesn't understand the concept of it even. And similarly, consciousness, I mean, you can certainly imagine very simple kind of experience. If certain plants have any kind of experience, I don't think they're very afraid of dying or there's nothing they can do about it anyway. So there wasn't much value. But more seriously, I think if you ask not just about being conscious, but maybe having what we might call an exciting life where you feel passion and really appreciate the things, maybe there perhaps it does help having a backdrop that, hey, it's finite. Let's make the most of this, let's live to the fullest. So if you knew you were gonna just live forever, do you think you would change your? Yeah, I mean, in some perspective, it would be an incredibly boring life living forever. So in the sort of loose subjective terms that you said of something exciting and something in this that other humans would understand, I think, is yeah, it seems that the finiteness of it is important. Well, the good news I have for you then is based on what we understand about cosmology, everything in our universe is ultimately probably finite. Big crunch or big, what's the infinite expansion? Yeah, we could have a big chill or a big crunch or a big rip or the big snap or death bubbles. All of them are more than a billion years away. So we certainly have vastly more time than our ancestors thought, but there is still pretty hard to squeeze in an infinite number of compute cycles, even though there are some loopholes that just might be possible. But I think some people like to say that you should live as if you're gonna die in five years or so, and that's sort of optimal. Maybe it's a good assumption, we should build our civilization as if it's all finite to be on the safe side. Right, exactly. So you mentioned defining intelligence as the ability to solve complex goals. Where would you draw a line? How would you try to define human level intelligence and superhuman level intelligence? Where is consciousness part of that definition? No, consciousness does not come into this definition. So I think of intelligence as a spectrum, but there are very many different kinds of goals you can have. You can have a goal to be a good chess player, a good go player, a good car driver, a good investor, good poet, et cetera. So intelligence by its very nature isn't something you can measure, but it's one number, some overall goodness. No, no, there are some people who are better at this, some people are better at that. Right now we have machines that are much better than us at some very narrow tasks, like multiplying large numbers fast, memorizing large databases, playing chess, playing Go, and soon driving cars. But there's still no machine that can match a human child in general intelligence. But artificial general intelligence, AGI, the name of your course, of course, that is by its very definition, the quest to build a machine that can do everything as well as we can. So the old holy grail of AI from back to its inception in the 60s. If that ever happens, of course, I think it's going to be the biggest transition in the history of life on earth. But it doesn't necessarily have to wait the big impact until machines are better than us at knitting. The really big change doesn't come exactly at the moment. They're better than us at everything. The really big change comes first. Their big change is when they start becoming better at us at doing most of the jobs that we do, because that takes away much of the demand for human labor. And then the really whopping change comes when they become better than us at AI research. Because right now, the time scale of AI research is limited by the human research and development cycle of years, typically. How long does it take from one release of some software or iPhone or whatever to the next? But once Google can replace 40,000 engineers by 40,000 equivalent pieces of software or whatever, then there's no reason that has to be years. It can be, in principle, much faster. And the time scale of future progress in AI and all of science and technology will be driven by machines, not humans. So it's this simple point which gives right to this incredibly fun controversy about whether there can be intelligence explosion, so-called singularity, as Werner Wienz called it. The idea, as articulated by I.J. Good, is obviously way back in the 50s. But you can see Alan Turing and others thought about it even earlier. You asked me what exactly would I define human level at. So the glib answer is to say something which is better than us at all cognitive tasks, or better than any human at all cognitive tasks. But the really interesting bar, I think, goes a little bit lower than that, actually. It's when they're better than us at AI programming and general learning, so that they can, if they want to, get better than us at anything by just studying up. So there, better is a key word. And better is towards this spectrum of the complexity of goals it's able to accomplish. So another way to, and that's certainly a very clear definition of human level. So it's almost like a sea that's rising. You can do more and more and more things. It's actually a graphic that you show. It's a really nice way to put it. So there's some peaks, and there's an ocean level elevating, and you solve more and more problems. But just to take a pause, and we took a bunch of questions and a lot of social networks. And a bunch of people asked a slightly different direction on creativity, on things that perhaps aren't a peak. Human beings are flawed, and perhaps better means having contradiction, being flawed in some way. So let me sort of start easy, first of all. So you have a lot of cool equations. Let me ask, what's your favorite equation, first of all? I know they're all like your children, but which one is that? This is the Schrodinger equation. It's the master key of quantum mechanics of the micro world. So with this equation, we can calculate everything to do with atoms, molecules, and all the way up to them. Yeah, so quantum mechanics is certainly a beautiful, mysterious formulation of our world. So I'd like to ask you, just as an example, it perhaps doesn't have the same beauty as physics does, but in mathematics, abstract, the Andrew Wiles, who proved the Fermat's last theorem. So I just saw this recently, and it kind of caught my eye a little bit. This is 358 years after it was conjectured. So this very simple formulation. Everybody tried to prove it. Everybody failed. And so here's this guy comes along, and eventually proves it and then fails to prove it and then proves it again. In 94, and he said the moment when everything connected into place, in an interview he said, it was so indescribably beautiful. That moment when you finally realized the connecting piece of two conjectures, he said, it was so indescribably beautiful. It was so simple and so elegant. I couldn't understand how I'd missed it, and I just stared at it in disbelief for 20 minutes. Then during the day, I walked around the department, and I'd keep coming back to my desk, looking to see if it was still there. It was still there. I couldn't contain myself. I was so excited. It was the most important moment of my working life. Nothing I ever do again will mean as much. So that particular moment, and it kind of made me think of, what would it take? And I think we have all been there at small levels. Maybe you let me ask, have you had a moment like that in your life where you just had an idea? It's like, wow, yes. I wouldn't mention myself in the same breath as Andrew Wiles, but I've certainly had a number of aha moments when I realized something very cool about physics just completely made my head explode. In fact, some of my favorite discoveries I made, I later realized that they had been discovered earlier by someone who sometimes got quite famous for it. So it's too late for me to even publish it, but that doesn't diminish in any way the emotional experience you have when you realize it. Like, wow. Yeah, so what would it take in that moment, that wow, that was yours in that moment? So what do you think it takes for an intelligent system, an AGI system, an AI system to have a moment like that? That's a tricky question because there are actually two parts to it, right? One of them is, can it accomplish that proof? Can it prove that you can never write A to the N plus B to the N equals three to that equals Z to the N for all integers, et cetera, et cetera, when one end is bigger than two? That's simply a question about intelligence. Can you build machines that are that intelligent? And I think by the time we get a machine that can independently come up with that level of proofs, probably quite close to AGI. The second question is a question about consciousness. When will we, how likely is it that such a machine would actually have any experience at all as opposed to just being like a zombie? And would we expect it to have some sort of emotional response to this or anything at all akin to human emotion where when it accomplishes its machine goal, it reviews it as somehow something very positive and sublime and deeply meaningful? I would certainly hope that if in the future we do create machines that are our peers or even our descendants, that I would certainly hope that they do have this sort of sublime appreciation of life. In a way, my absolutely worst nightmare would be that, at some point in the future, the distant future maybe our cosmos is teeming with all this post-biological life, doing all the seemingly cool stuff. And maybe the last humans by the time our species eventually fizzles out will be like, well, that's okay, because we're so proud of our descendants here and look at what all the... My worst nightmare is that we haven't solved the consciousness problem and we haven't realized that these are all the zombies. They're not aware of anything any more than a tape recorder as an any kind of experience. So the whole thing has just become a play for empty benches. That would be like the ultimate zombie apocalypse. So I would much rather in that case that we have these beings which can really appreciate how amazing it is. And in that picture, what would be the role of creativity? A few people ask about creativity. Do you think, when you think about intelligence, I mean, certainly the story you told at the beginning of your book involved creating movies and so on. Yeah. Sort of making money. You can make a lot of money in our modern world with music and movies. So if you are an intelligent system, you may want to get good at that. But that's not necessarily what I mean by creativity. Is it important on that complex goals where the sea is rising for there to be something creative? Or am I being very human-centric and thinking creativity is somehow special relative to intelligence? My hunch is that we should think of creativity simply as an aspect of intelligence. And we have to be very careful with human vanity. We have this tendency to very often want to say, as soon as machines can do something, we try to diminish it and say, oh, but that's not like real intelligence. Because they're not creative or this or that. The other thing, if we ask ourselves to write down a definition of what we actually mean by being creative, what we mean by Andrew Wiles, what he did there, for example, don't we often mean that someone takes a very unexpected leap? Mm-hmm. It's not like taking 573 and multiplying it by 224 by just a step of straightforward cookbook-like rules, right? You can maybe make a connection between two things that people have never thought was connected. It's very surprising. Or something like that. I think this is an aspect of intelligence. And this is actually one of the most important aspects of it. Maybe the reason we humans tend to be better at it than traditional computers is because it's something that comes more naturally if you're a neural network than if you're a traditional logic gate-based computer machine. We physically have all these connections. And if you activate here, activate here, activate here, it's ping! You know, my hunch is that if we ever build a machine where you could just give it the task, hey, hey, you say, hey, you know, I just realized I want to travel around the world instead this month. Can you teach my AGI course for me? And it's like, OK, I'll do it. I'll do the other thing that you would have done, and the improvisers and stuff. That would, in my mind, involve a lot of creativity. Yeah, so it's actually a beautiful way to put it. I think we do try to grasp at the, you know, the definition of intelligence is everything we don't understand how to build. So we as humans try to find things that we have and machines don't have. Maybe creativity is just one of the things, one of the words we use to describe that. That's a really interesting way to put it. I don't think we need to be that defensive. I don't think anything good comes out of saying, oh, we're somehow special, you know. It's, contrary-wise, there are many examples in history of where trying to pretend that we're somehow superior to all other intelligent beings led to pretty bad results, right? In Nazi Germany, they said that they were somehow superior to other people. Today, we still do a lot of cruelty to animals by saying that we're so superior somehow, and they can't feel pain. Slavery was justified by the same kind of really weak arguments. I don't think if we actually go ahead and build artificial general intelligence, we can do things better than us. I don't think we should try to found our self-worth on some sort of bogus claims of superiority in terms of our intelligence. I think we should instead find our calling and the meaning of life from the experiences that we have. I can have very meaningful experiences, even if there are other people who are smarter than me. When I go to a faculty meeting here, and we're talking about something, and I suddenly realize, oh, he has an old prize, he has an old prize, he has an old prize. Does that make me enjoy life any less or enjoy talking to those people less? Of course not. Contrary, I feel very honored and privileged to get to interact with other very intelligent beings that are better than me at a lot of stuff. I don't think there's any reason why we can't have the same approach with intelligent machines. That's a really interesting... People don't often think about that. They think about if there's machines that are more intelligent, you naturally think that that's not going to be a beneficial type of intelligence. You don't realize it could be peers with Nobel prizes that would be just fun to talk with. They might be clever about certain topics, and you can have fun having a few drinks with them. Also, another example we can all relate to of why it doesn't have to be a terrible thing to be impressed with people who are even smarter than us all around is when you and I were both two years old. Our parents were much more intelligent than us. It worked out okay because their goals were aligned with our goals. That, I think, is really the number one key issue we have to solve. The value alignment problem. People who see too many Hollywood movies with lousy science fiction plot lines, they worry about the wrong thing. They worry about some machine suddenly turning evil. It's not malice that's the issue, the concern. It's competence. By definition, intelligence makes you very competent. If you have a more intelligent go-playing computer, playing as a less intelligent one, and when we define intelligence as the ability to accomplish go-winning, it's going to be the more intelligent one that wins. If you have a human and then you have an AGI that's more intelligent in all ways and they have different goals, guess who's going to get their way? I was just reading about this particular rhinoceros species that was driven extinct just a few years ago. It was a bummer. I was looking at this cute picture of a mommy rhinoceros with its child. Why did we humans drive it to extinction? It wasn't because we were evil rhino haters as a whole. It was just because our goals weren't aligned with those of the rhinoceros, and it didn't work out so well for the rhinoceros because we were more intelligent. I think it's just so important that if we ever do build AGI, before we unleash anything, we have to make sure that it learns to understand our goals, that it adopts our goals, and it retains those goals. So the cool, interesting problem there is us as human beings trying to formulate our values. You could think of the United States Constitution as a way that people sat down, at the time a bunch of white men, which is a good example, we should say. They formulated the goals for this country, and a lot of people agree that those goals actually held up pretty well. It's an interesting formulation of values and failed miserably in other ways. So for the value alignment problem and the solution to it, we have to be able to put on paper or in a program human values. How difficult do you think that is? Very. But it's so important. We really have to give it our best. And it's difficult for two separate reasons. There's the technical value alignment problem of figuring out how to make machines understand our goals, adopt them, and retain them. And then there's the separate part of it, the philosophical part, whose values anyway? And since it's not like we have any great consensus on this planet on values, what mechanism should we create then to aggregate and decide, OK, what's a good compromise? That second discussion can't just be left the tech nerds like myself. That's right. And if we refuse to talk about it and then AGI gets built, who's going to be actually making the decision about whose values? It's going to be a bunch of dudes and some tech company. And are they necessarily so representative of all of humankind that we want to just entrust it to them? Are they even uniquely qualified to speak to future human happiness just because they're good at programming AI? I'd much rather have this be a really inclusive conversation. But do you think it's possible sort of that? So you create a beautiful vision that includes sort of the diversity, cultural diversity and various perspectives on discussing rights, freedoms, human dignity. But how hard is it to come to that consensus? Do you think it's certainly a really important thing that we should all try to do. But do you think it's feasible? I think there's no better way to guarantee failure than to refuse to talk about it or refuse to try. And I also think it's a really bad strategy to say, OK, let's first have a discussion for a long time. And then once we reach complete consensus, then we'll try to load it into some machine. No, we shouldn't let perfect be the enemy of good. Instead, we should start with the kindergarten ethics that pretty much everybody agrees on and put that into machines. Now, we're not doing that even. Look at anyone who builds a passenger aircraft wants to never under any circumstances fly into a building or mountain. Right. Yet the September 11 hijackers were able to do that. And even more embarrassingly, you know, and the gas lube, it's this depressed German wings pilot when he flew his passenger jet into the Alps, killing over 100 people. He just told the autopilot to do it. He told the freeing computer to change the altitude to 100 meters. And even though it had the GPS maps, everything the computer was like, OK, no. So we we should take those very basic values where the problem is not that we don't agree. The problem is just we've been too lazy to try to put it into our machines and make sure that from now on, airplanes will just which all have computers in them. We just never just refuse to do something like that. Go into safe mode, maybe lock the cockpit door, go to the nearest airport. And and there's so much other technology in our world as well now where it's really quite becoming quite timely to put in some sort of very basic values like this. Even in cars, we have had enough vehicle terrorism attacks by now. If you have driven trucks and vans into pedestrians, that is not at all a crazy idea to just have that hardwired into the car. Because, yeah, there are a lot of there's always going to be people who for some reason want to harm others. But most of those people don't have the technical expertise to figure out how to work around something like that. So if the car just won't do it. It helps. So let's start there. So there's a lot of that's a that's a great point. So not not chasing perfect. There's a lot of things that a lot that most of the world agrees on. Yeah, and start there. Let's start there. And then once we start there, we'll also get into the habit of having these kind of conversations about, OK, what else should we put in here and have these discussions? This should be a gradual process. Great. So but that also means describing these things and describing it to a machine. So one thing we had a few conversations with Stephen Wolfram. I'm not sure if you're familiar with Stephen. Oh, yeah, I know him quite well. So he is, you know, he play, you know, works with a bunch of things, but, you know, cellular automata, the simple, computable things, these computation systems. And you kind of mentioned that, you know, we probably have already within these systems already something that's a GI, meaning like we just don't know it because we can't talk to it. So if you give me this chance to try to at least form a question out of this is I think it's an interesting idea to think that we can have intelligent systems, but we don't know how to describe something to them and they can't communicate with us. I know you're doing a little bit of work and explainable I trying to get a to explain itself. So what are your thoughts of natural language processing or some kind of other communication? How how does the AI explain something to us? How do we explain something to it to machines? Or you think of it differently? So there are two separate parts to your question there. One of them has to do with communication, which is super interesting. You only get to that in a sec. The other is whether we already have a GI, but we just haven't noticed it. Right. There I beg to differ. I don't think there's anything in any cellular automaton or anything or the Internet itself or whatever that has artificial general intelligence in that it's really do exactly everything we humans can do better. I think today if the day that happens, when that happens, we will very soon notice and will probably notice even before. And if because in a very, very big way. But for the for the second part, though, I can't. I'm sorry. So the because you have this beautiful way to formulate in consciousness as as a you know, as information processing. You can think of intelligence information processing and this you can think of the entire universe is these particles and these systems roaming around that have this information processing power. You don't you don't think there is something with the power to process information in the way that we human beings do that's out there that that needs to be sort of connected to. It seems a little bit philosophical, perhaps, but there's something compelling to the idea that the power is already there. Would you guess the focus should be more on the on being able to communicate with it? Well, I agree that in some in a certain sense, the hardware processing power is already out there because our universe itself can think of it as being a computer already. Right. It's constantly computing what water waves how to evolve the water waves and the river Charles and how to move the air molecules around. And Seth Lloyd has pointed out my colleague here that you can even in a very rigorous way think of our entire universe is being a quantum computer. It's pretty clear that our universe supports this amazing processing power because you can even within this physics computer that we live in, right. We can even build actually laptops and stuff. So clearly the power is there. It's just that most of the compute power that nature has, it's, in my opinion, kind of wasting on boring stuff like simulating yet another ocean wave somewhere where no one is even looking. So in a sense of what life does, what we are doing when we build computers is we're rechanneling all this compute that nature is doing anyway and doing things that are more interesting than just yet another ocean wave. You know, and let's do something cool here. So the raw hardware power is there and for sure. But and then even just like computing what's going to happen for the next five seconds in this water ball, you know, it takes a ridiculous amount of compute if you do it on a human computer. This water ball just did it. But that does not mean that this water ball has AGI and because AGI means it should also be able to like have written my book, done this interview. Yes. And I don't think it's just communication problems. I don't really know. Don't think it can do it. Although Buddhists say when they watch the water and that there is some beauty, that there is some depth and nature that they can communicate with. Communication is also very important though because I mean, look, part of my job is being a teacher. And I know some very intelligent professors even who just have a better hard time communicating. They come up with all these brilliant ideas. But to communicate with somebody else, you have to also be able to simulate their own mind. Yes. And build well enough and understand model of their mind that you can say things that they will understand. And that's quite difficult. And that's why today it's so frustrating if you have a computer that makes some cancer diagnosis and you ask it, well, why are you saying I should have a surgery? And if it can only reply, I was trained on five terabytes of data and this is my diagnosis. Boop, boop, beep, beep. Yeah. Doesn't really instill a lot of confidence, right? Right. So I think we have a lot of work to do on communication there. So what kind of I think you're doing a little bit work in Explainable AI. What do you think are the most promising avenues? Is it mostly about sort of the Alexa problem of natural language processing of being able to actually use human interpretable methods of communication? So being able to talk to a system and talk back to you, or is there some more fundamental problems to be solved? I think it's all of the above human. The natural language processing is obviously important, but there are also more nerdy fundamental problems. Like if you take, you play chess. Of course, I'm Russian. I have to. You speak Russian? Yes, I speak Russian. Great, I didn't know. When did you learn Russian? I speak very badly in Russian. I only did after-deduct, but I bought a book called Teach Yourself Russian. I read a lot, but it was very difficult. Wow. That's why I speak so badly. How many languages do you know? Wow, that's really impressive. I don't know. My wife has some congratulations. But my point was, if you play chess, have you looked at the AlphaZero games? The actual games, no. Check it out. Some of them are just mind-blowing. Really beautiful. And if you ask, how did it do that? You got to talk to Demis El-Sabes and others from DeepMind. All they'll ultimately be able to give you is big tables of numbers, matrices that define the neural network. And you can stare at these numbers until your face turns blue. And you're not going to understand much about why it made that move. And even if you have natural language processing that can tell you in human language about, oh, five, seven, points, two, eight, still not going to really help. So I think there's a whole spectrum of fun challenges involved in taking a computation that does intelligent things and transforming it into something. Equally good, equally intelligent. But that's more understandable. And I think that's really valuable because I think. As we put machines in charge of ever more infrastructure in our world, the power grid, the trading on the stock market, weapons systems and so on. It's absolutely crucial that we can trust these AIs that do all we want. And trust really comes from understanding in a very fundamental way. And that's why I'm that's why I'm working on this, because I think the more if we're going to have some hope of ensuring that machines have adopted our goals and that they're going to retain them. That kind of trust. I think needs to be based on things you can actually understand, preferably even make it have preferably improved theorems on even with a self-driving car. Right. If someone just tells you it's been trained on tons of data and it never crashed, it's less reassuring than if someone actually has a proof. Maybe it's a computer verified proof. But still, it says that under no circumstances is this car just going to swerve into oncoming traffic. And that kind of information helps to build trust and build the alignment, the alignment of goals, at least awareness that your goals, your values are aligned. And I think even in the very short term, if you look at how, you know, today, right, this absolutely pathetic state of cybersecurity that we have. Right. Where is it? Three billion Yahoo accounts were kept packed. Almost every. Americans credit card and so on. Why is this happening? It's ultimately happening because we have software that nobody fully understood how it worked. That's why the bugs hadn't been found. Right. And I think AI can be used very effectively for offense, for hacking, but it can also be used for defense. Hopefully, automating verifiability and creating systems that are. Built in different ways so you can actually prove things about them, right. And it's it's important. So speaking of software that nobody understands how it works, of course, a bunch of people ask about your paper, about your thoughts of why does deep and cheap learning work so well. That's the paper. But what are your thoughts on deep learning? These kind of simplified models of our own brains have been able to do some successful perception work, pattern recognition work. And now with Alpha Zero and so on, do some some clever things. What are your thoughts about the promise limitations of this piece? Right. I think. There are a number of very important insights. Very important lessons we can all withdraw from these kind of successes. One of them is when you look at the human brain, you see it's very complicated. Tenth of eleven neurons and there are all these different kinds of neurons and yada yada. And there's been this long debate about whether the fact that we have dozens of different kinds is actually necessary for intelligence. We can now, I think, quite convincingly answer that question. No, it's enough to have just one kind. If you look under the hood of Alpha Zero, there's only one kind of neuron and it's ridiculously simple, simple mathematical thing. But it's not it's just like in physics, it's not the if you have a gas with waves in it, it's not the detailed nature of the molecules that matter. It's the collective behavior somehow. Similarly, it's it's it's this higher level structure of the network matters, not that you have 20 kinds of neurons. I think our brain is such a complicated mess because it wasn't evolved. Just to be intelligent, it was evolved to also be self assembling. Right. And self repairing. Right. And evolutionarily attainable and so on. And so on. Yeah. So I think it's pretty my my hunch is that we're going to understand how to build AGI before we fully understand how our brains work. Just like we we understood how to build flying machines long before we were able to build a mechanical work bird. Yeah, that's right. You've given things you're given that the example exactly of mechanical birds and airplanes and airplanes do a pretty good job of flying without really mimicking bird flight. And even now, after 100, 100 years later, did you see the TED talk with this German mechanical bird? I heard you mention it. I want to see it. It's amazing. But even after that, we still don't fly in mechanical birds because it turned out the way we came up with was simpler and better for our purposes. And I think it might be the same there. That's one lesson. Another lesson, which is more what what our paper was about. First, I as a physicist thought it was fascinating how there's a very close mathematical relationship actually between our artificial neural networks and a lot of things that we've studied for in physics. We've studied by nerdy names like the renormalization group equation and Hamiltonians and yada yada yada. And when you look a little more closely at this. You have. You. At first, I was like, well, there's something crazy here that doesn't make sense because. That if you even want to build a super simple neural network, tell that part cat pictures and dog pictures, right? You can do that very, very well now. But if you think about it a little bit, you convince yourself it must be impossible because if I have one megapixel, even if each pixel is just black or white, there's two to the power one million possible images is way more than their atoms in our universe. Right. So in order to. And then for each one of those, I have to assign a number, which is the probability that it's a dog. So an arbitrary function of images is a list of. More numbers than their atoms in our universe. So clearly I can't store that under the hood of my my GPU or my computer yet somehow works. So what does that mean? Well, it means that the out of all of the problems that you could try to solve with a neural network. Almost all of them are impossible to solve with a reasonably sized one. But then what we should show it in our paper was was that the fraction, the kind of problems, the fraction of all the problems that you could possibly pose that that we actually care about, given the laws of physics is also an infinite test. Let me tie me a little part. And amazingly, they're basically the same part. Yeah, it's almost like our world was created for. I mean, they kind of come together. Yeah, you, well, you could say maybe where the world created the world, the world was created for us, but I have a more modest interpretation, which is that instead evolution endowed us with neural networks, precisely for that reason. Right. Because this particular architecture, as opposed to the one in your laptop, is very, very well. Adapted to solving the kind of problems that nature kept presenting our ancestors with. Right. So it makes sense that why do we have a brain in the first place? It's to be able to make predictions about the future and so on. So if we had a sucky system, which could never solve it, wouldn't have a lot. So, but it's so this is this is I. I think you're a beautiful fact. Yeah, we also, we also realize that there's the there that we there's been earlier work on on why deeper networks are good, but we were able to show an additional cool fact there, which is that even incredibly simple problems like suppose I give you a. A thousand numbers and ask you to multiply them together and you can write a few lines of code boom done trivial. If you just try to do that with a neural network that has only one single hidden layer in it, you can do it. But you're going to need two to the power of thousand neurons. To multiply a thousand numbers, which is again more neurons and their atoms in our universe. That's fascinating. But if you're allowed, if you allow yourself, make it a deep network with many layers, you only need four thousand neurons. It's perfectly feasible. So that's really interesting. Yeah. Yeah. So on another architecture type, I mean, you mentioned Schrodinger's equation and what are your thoughts about quantum computing? And the role of this kind of computational unit in creating an intelligence system in some Hollywood movies that are a lot mentioned by name because I want to spoil them. The way they get a G.I. is building a quantum computer was because the word quantum sounds cool and so on. But my first of all, I think we don't need quantum computers to build a G.I. I suspect your brain is not quantum computer and any found sense. So you don't even wrote a paper about that many years ago. I calculated the decoherence so-called decoherence time, how long it takes until the quantum computer in this of what your neurons are doing gets erased. It's just random noise from the environment and it's about 10 to the minus 21 seconds. So as cool as it would be to have a quantum computer in my head, I don't think that fast. On the other hand, there are. Very cool things you could do with quantum computers. Or I think we'll be able to do soon when we get big one bigger ones that might actually help machine learning do even better than the brain. So. For example. One, this is just a moon shot, but hey, you know, not. Learning. It's very much the same thing as search. If you have if you're trying to train a neural network to get really learned to do something really well, you have some loss function. You have some you have a bunch of knobs you can turn represented by a bunch of numbers and you're trying to tweak them so that it becomes as good as possible at this thing. So if you think of a landscape with some valley. Where each dimension of the landscape corresponds to some number you can change, you're trying to find the minimum. And it's well known that if you have a very high dimensional landscape, complicated things, it's super hard to find the minimum. Quantum mechanics is amazingly good at this, right? If I want to know what's the lowest energy state this water can possibly have. Incredibly hard to compute, but we can. But nature will happily figure this out for you if you just cool it down, make it very, very cold. If you put a ball somewhere, it'll roll down to its minimum. And this happens metaphorically at the energy landscape, too. And quantum mechanics even uses some clever tricks, which today's machine learning systems don't like. You're trying to find the minimum and you get stuck in the little local minimum here in quantum mechanics. You can actually tunnel through the barrier and get unstuck again. And that's really interesting. So it's maybe, for example, we'll one day use quantum computers that. To help train neural networks better. That's really interesting. OK, so as a component of kind of the learning process, for example. Yeah. Let me ask sort of wrapping up here a little bit. Let me let me return to the questions of our human nature and and love, as I mentioned. So do you think. You mentioned sort of a helper robot, but you could think of also personal robots. Do you think the way we human beings fall in love and get connected to each other is possible to achieve in an AI system and human level AI intelligence system? Do you think we would ever see that kind of connection or, you know, in all this discussion about solving complex goals? Yeah. As this kind of human social connection, do you think that's one of the goals on the peaks and valleys that with the raising sea levels that we'll be able to achieve? Or do you think that's something that's ultimately or at least in the short term relative to the other goals is not achievable? I think it's all possible. And I mean, in recent there is a there is a very wide range of guesses, as you know, among AI researchers when we're going to get AGI. Some people, you know, like our friend Rodney Brooks said, it's going to be hundreds, hundreds of years at least. And then there are many others who think it's going to happen much sooner. And recent polls. Maybe half or so already, I research think it's we're going to get AGI within decades. So if that happens, of course, I think these things are all possible. But in terms of whether it will happen, I think we shouldn't spend so much time asking what do we think will happen in the future? As if we are just some sort of pathetic, passive bystanders, you know, waiting for the future to happen to us. Hey, we're the ones creating this future. Right. So we should be proactive about it and ask ourselves what sort of future we would like to have happen. That's right. When I make it like that. Well, what I prefer it to some sort of incredibly boring zombie like future where there's all these mechanical things happening. There's no passion, no emotion, no experience, maybe even. No, I would, of course, much rather prefer it if all the things that we find that we value the most about humanity are subjective experience, passion, inspiration, love. If we can create a future where those are those things do exist. I think ultimately it's not our universe giving meaning to us. It's us giving meaning to our universe. And if we build more advanced intelligence, let's make sure we're built in such a way that meaning is part of it. A lot of people that seriously study this problem and think of it from different angles have trouble in the majority of cases if they think through that happen are the ones that are not beneficial to humanity. Right. And so, yeah. So what are your thoughts? What's an inch? What's what should people, you know, I really don't like people to be terrified. What's the way for people to think about it in a way that in a way we can solve it and we can make it better. Yeah, no, I don't think panicking is going to help in any way. It's not going to increase chances of things going well either. Even if you are in a situation where there is a real threat, does it help if everybody just freaks out? Right. No, of course. Of course not. I think there are, of course, ways in which things can go horribly wrong. First of all, it's important when we think about this thing, this about the problems and risks to also remember how huge the upsides can be if we get it right. Right. Everything, everything we love about society and civilization is a product of intelligence. So if we can amplify our intelligence or machine intelligence and not anymore lose our loved one to what we're told is an uncurable disease and things like this, of course, we should aspire to that. So that can be a motivator, I think, reminding ourselves that the reason we try to solve problems is not just because. We're trying to avoid gloom, but because we're trying to do something great. But then in terms of the risks, I think. The really important question is to ask what can we do today that will actually help. Yes, outcome good. Right. And dismissing the risk is not one of them. You know, I find it quite funny often when I'm in discussion panels about these things, how the people who work for companies always say, always like, oh, nothing to worry about, nothing to worry about, nothing to worry about. And it's always all it's only academics sometimes express concerns. That's not surprising at all. If you think about it. Right. Often Sinclair equipped right that the charging make a man believe in something when his income depends on not believing in it. And, and frankly, we know a lot of these people in companies that they're just as concerned as anyone else. But if you're the CEO of a company, that's not something you want to go on record saying when you have silly journalists who are going to put a picture of a Terminator robot when they quote you. So, so the issues are real. And the way I think about what the issue is, is basically, you know, the real choice we have is, first of all, are we going to dismiss this, the risks and say, well, you know, let's just go ahead and build machines that can do everything we can do better and cheaper, you know, let's just make ourselves obsolete as fast as possible. What could possibly go wrong, right? That's one attitude. The opposite attitude, I think it's a say, it's incredible potential, you know, let's think about what kind of future. We're really, really excited about what are the shared goals that we can really aspire towards and then let's think really hard on how about how we can actually get there. So start with it, not don't start thinking about the risks, start thinking about the goals goals. Yeah. And then when you do that, then you can think about the obstacles you want to avoid, right? I often get students coming in right here into my office for career advice. I always ask them this very question. Where do you want to be in the future? Right? If all she can say is, oh, maybe I'll have cancer. Maybe I'll run over by a truck. And obstacles instead of the goal. She's just going to end up a hypochondriac paranoid. Whereas if she comes in and fire in her eyes and is like, I want to be there. And then we can talk about the obstacles and see how we can circumvent them. That's, I think, a much, much healthier attitude. And that's really well. And I, I, I feel it's very challenging to come up with a vision for the future, which we're, which we are unequivocally excited about. I'm not just talking now in the vague terms like, yeah, let's cure cancer. Fine. Talking about what kind of society that we want to create. What do we want it to mean, you know, to be human in the age of AI, in the age of AGI. So if we can have this conversation, broad, inclusive conversation, and gradually start converging towards some, some future that with some direction, at least that we want to steer towards right then, then we'll be much more motivated to constructively take on the obstacles. And I think if I had, if I had to, if you make, if I try to wrap this up in a more succinct way, I think, I think we can all agree already now that we should aspire to build AGI that doesn't overpower us, but that empowers us. And think of the many various ways that can do that, whether that's from my side of the world of autonomous vehicles. I personally actually from the camp that believes this human level intelligence is required to, to achieve something like vehicles that would actually be something we would enjoy using and being part of. So that's the one example and certainly there's a lot of other types of robots and medicine, and so on. So focusing on those and then and then coming up with the obstacles coming up with the ways that that can go wrong and solving those one at a time. And just because you can build an autonomous vehicle, even if you could build one that would drive this finalized, you know, maybe there are some things in life that we would actually want to do ourselves. That's right. Right. Like, for example, if you think of our society as a whole, there's some things that we find very meaningful to do. And that doesn't mean we have to stop doing them just because machines can do them better. You know, I'm not going to stop playing tennis, just the day someone builds a tennis robot. Yeah, beat me. People are still playing chess and even go. Yeah. And I in this in the very near term, even some people are advocating basic income, replace jobs. But if you if the government is going to be willing to just hand out cash to people for doing nothing, then once you also seriously consider whether the government should also hire a lot more teachers and nurses and the kind of jobs which people often find great fulfillment in doing right. And we're tired of hearing politicians saying, oh, we can't afford hiring more teachers, but we're going to maybe have basic income. If we can have more more serious research and thought into what gives meaning to our lives and the jobs give so much more than income. Right. And then think about in the future. What are the roles that we want to have people continually filling empowered by machines. And I think sort of, I come from the Russia from the Soviet Union and I think for a lot of people in the 20th century going to the moon, going to space was an inspiring thing. I feel like the, the, the, the universe of the mind so AI understanding creating intelligence is that for the 21st century. So it's really surprising and I've heard you mention this it's really surprising to me, both on the research funding side that it's not funded as greatly as it could be. But most importantly, on the politician side, that it's not part of the public discourse except in the killer bots terminator kind of view that people are not yet, I think, perhaps excited by the possible positive future that we can build together so we should be because politicians usually just focus on the next election cycle right. And the single most important thing I feel we humans have learned and the entire history of science is there were the masters of underestimation we underestimated the size of our cosmos, again and again, realizing that everything we thought existed was just a small part of something grander planet solar system the galaxy clusters of guys in the universe. And now know that we have the future has just so much more potential than our ancestors could ever have dreamt of this cosmos. Imagine if all of Earth was completely devoid of life, except for Cambridge Massachusetts, I would wouldn't it be kind of lame. If all we ever aspired to is to stay in Cambridge Massachusetts forever. And go extinct in one week, even though Earth was going to continue on for longer that that that sort of attitude I think we have now on the cosmic scale, we can full life can flourish on Earth, not for four years but for billions of years, yes, I can even tell you I can move it out of harm's way when the sun gets too hot. And, and then we have so much more resources out here, which today, yeah maybe there are a lot of other planets with bacteria or cow like life on them but I most of this, all this opportunity seems as far as we can largely dead, like the Sahara Desert and yet we have the opportunity to help life flourish around this billions of years so like, let's quit squabbling about whether some little border should be drawn one one mile to the left or right, and look up into the skies Hey, you know, we can do such incredible things. Yeah, and that's I think why it's really exciting that yeah, you and others are connected with some of the work Elon Musk is doing, because he's literally going out into that space really exploring our universe and it's wonderful. That is exactly why Elon Musk is so misunderstood right. Miss can screw him as some kind of pessimistic doomsayer, the reason he cares so much about the safety is because he more than almost anyone else appreciates these amazing opportunities they will squander it if we wipe out here on earth, and we're not just going to wipe out the next generation, all generations and this incredible opportunity that's out there and that would be really be a waste and AI. For people who think that that would be better to do without technology. Let me just mention that if we don't improve our technology the question isn't whether humanity is going to go extinct. It's just whether we're going to get taken out by the next big asteroid or the next super volcano or something else dumb that we could easily prevent with more tech right. And if we want life to flourish throughout the cosmos, AI is the key to it. As I mentioned in a lot of detail in my book right there. Even many of the most inspired sci fi writers I feel have totally underestimated the opportunities for space travel especially to other galaxies, because they weren't thinking about the possibility of a GI, which just makes it so much easier. Yeah, so that goes to your view of a GI that enables our progress that enables a better life. So that's a beautiful and that's a beautiful way to put it, and then something to strive for. So Max, thank you so much. Thank you for your time today. It's been awesome. Thank you so much. Thanks.
 As part of MIT course 6 as 099, Artificial General Intelligence,
 I've gotten the chance to sit down with Max Tegmark.
 He is a professor here at MIT.
 He's a physicist, spent a large part of his career
 studying the mysteries of our cosmological universe.
 But he's also studied and delved into the beneficial possibilities
 and the existential risks of artificial intelligence.
 Amongst many other things, he's the co-founder of the Future of Life
 Institute, author of two books, both of which I highly recommend.
 First, Our Mathematical Universe.
 Second is Life 3.0.
 He's truly an out-of-the-box thinker and a fun personality.
 So I really enjoy talking to him.
 If you'd like to see more of these videos in the future,
 please subscribe and also click the little bell icon
 to make sure you don't miss any videos.
 Also, Twitter, LinkedIn, aji.mit.edu if you
 want to watch other lectures or conversations like this one.
 Better yet, go read Max's book, Life 3.0, Chapter 7 on Goals
 is my favorite.
 It's really where philosophy and engineering come together.
 And it opens with a quote by Dostoevsky,
 the mystery of human existence lies not in just staying alive,
 but in finding something to live for.
 Lastly, I believe that every failure rewards us
 with an opportunity to learn.
 In that sense, I've been very fortunate to fail
 in so many new and exciting ways.
 And this conversation was no different.
 I've learned about something called radio frequency
 interference, RFI.
 Look it up.
 Apparently, music and conversations
 from local radio stations can bleed into the audio
 that you're recording in such a way that almost completely
 ruins that audio.
 It's an exceptionally difficult sound source to remove.
 So I've gotten the opportunity to learn
 how to avoid RFI in the future during recording sessions.
 I've also gotten the opportunity to learn
 how to use Adobe Audition and iZotope RX 6
 to do some noise, some audio repair.
 Of course, this is an exceptionally difficult noise
 to remove.
 I am an engineer.
 I'm not an audio engineer.
 Neither is anybody else in our group.
 But we did our best.
 Nevertheless, I thank you for your patience.
 And I hope you're still able to enjoy this conversation.
 Do you think there's intelligent life out there in the universe?
 Let's open up with an easy question.
 I have a minority of you here, actually.
 When I give public lectures, I often
 ask for a show of hands who thinks
 there is intelligent life out there somewhere else.
 And almost everyone put their hands up.
 And when I ask why, they'll be like,
 oh, there's so many galaxies out there.
 There's got to be.
 But I'm a numbers nerd, right?
 So when you look more carefully at it, it's not so clear at all.
 When we talk about our universe, first of all,
 we don't mean all of space.
 We actually mean, I don't know, you
 can throw me the universe if you want.
 It's behind you there.
 We simply mean the spherical region of space
 from which light has had time to reach us
 so far during the 14.8 billion years, 13.8 billion years
 since our big bang.
 There's more space here.
 But this is what we call a universe,
 because that's all we have access to.
 So is there intelligent life here
 that's gotten to the point of building telescopes
 and computers?
 My guess is no, actually.
 The probability of it happening on any given planet
 is some number we don't know what it is.
 And what we do know is that the number can't be super high,
 because there's over a billion Earth-like planets in the Milky
 Way galaxy alone, many of which are billions of years older
 than Earth.
 And aside from some UFO believers,
 there isn't much evidence that any super-dense civilization
 has come here at all.
 And so that's the famous Fermi paradox, right?
 And then if you work the numbers,
 what you find is that if you have no clue what
 the probability is of getting life on a given planet,
 so it could be 10 to the minus 10, 10 to the minus 20,
 or 10 to the minus 2, any power of 10
 is sort of equally likely if you want to be really open-minded.
 That translates into it being equally likely
 that our nearest neighbor is 10 to the 16 meters away,
 10 to the 17 meters away, 10 to the 18.
 By the time you get much less than 10 to the 16 already,
 pretty much know there is nothing else that close.
 And when you get beyond 10.
 Because they would have discovered us.
 Yeah, they would have discovered us long ago.
 Or if they're really close, we would have probably
 noted some engineering projects that they're doing.
 And if it's beyond 10 to the 26 meters,
 that's already outside of here.
 So my guess is actually that we are the only life in here
 that's gotten to the point of building advanced tech, which
 I think puts a lot of responsibility on our shoulders
 to not screw up.
 I think people who take for granted
 that it's OK for us to screw up, have an accidental nuclear war,
 go extinct somehow because there's
 a sort of Star Trek-like situation out there
 where some other life forms are going to come and bail us out.
 And it doesn't matter as much.
 I think alliling us into a false sense of security.
 I think it's much more prudent to say,
 let's be really grateful for this amazing opportunity
 we've had and make the best of it,
 just in case it is down to us.
 So from a physics perspective, do you think intelligent life,
 so it's unique from a sort of statistical view
 of the size of the universe, but from the basic matter
 of the universe, how difficult is it for intelligent life
 to come about, the kind of advanced tech building life?
 Is implied in your statement that it's
 really difficult to create something like a human species?
 Well, I think what we know is that going from no life
 to having life that can do a level of tech,
 there is some sort of two going beyond that
 than actually settling our whole universe with life.
 There is some major roadblock there,
 which is some great filter as it's sometimes called,
 which is tough to get through.
 It's either that roadblock is either behind us
 or in front of us.
 I'm hoping very much that it's behind us.
 I'm super excited every time we get a new report from NASA
 saying they failed to find any life on Mars.
 Like, yes, awesome, because that suggests that the hard part,
 maybe it was getting the first ribosome
 or some very low level kind of stepping stone
 so that we're home free.
 Because if that's true, then the future
 is really only limited by our own imagination.
 It would be much suckier if it turns out
 that this level of life is kind of a diamond dozen.
 But maybe there is some other problem.
 Like, as soon as a civilization gets advanced technology,
 within 100 years, they get into some stupid fight
 with themselves and poof.
 Yep.
 That would be a bummer.
 Yeah, so you've explored the mysteries
 of the cosmological universe, the one that's between us today.
 I think you've also begun to explore the other universe,
 which is sort of the mystery, the mysterious universe
 of the mind, of intelligence, of intelligent life.
 So is there a common thread between your interest
 and the way you think about space and intelligence?
 Oh, yeah.
 When I was a teenager, I was already very fascinated
 by the biggest questions.
 And I felt that the two biggest mysteries of all in science
 were our universe out there and our universe in here.
 So it's quite natural, after having
 spent a quarter of a century on my career thinking
 a lot about this one, that I'm now
 indulging in the luxury of doing research on this one.
 It's just so cool.
 I feel the time is ripe now for you
 transdrakely deepening our understanding of this.
 Just start exploring this one.
 Yeah, because I think a lot of people view intelligence
 as something mysterious that can only exist
 in biological organisms like us, and therefore dismiss
 all talk about artificial general intelligence
 as science fiction.
 But from my perspective as a physicist,
 I am a blob of quarks and electrons
 moving around in a certain pattern
 and processing information in certain ways.
 And this is also a blob of quarks and electrons.
 I'm not smarter than the water bottle
 because I'm made of different kind of quarks.
 I'm made of up quarks and down quarks,
 exact same kind as this.
 There's no secret sauce, I think, in me.
 It's all about the pattern of the information processing.
 And this means that there's no law of physics
 saying that we can't create technology, which
 can help us by being incredibly intelligent
 and help us crack mysteries that we couldn't.
 In other words, I think we've really only seen the tip
 of the intelligence iceberg so far.
 Yeah, so the perceptronium.
 Yeah.
 So you coined this amazing term.
 It's a hypothetical state of matter,
 sort of thinking from a physics perspective,
 what is the kind of matter that can help, as you're saying,
 subjective experience emerge, consciousness emerge.
 So how do you think about consciousness
 from this physics perspective?
 Very good question.
 So again, I think many people have underestimated
 our ability to make progress on this
 by convincing themselves it's hopeless because somehow we're
 missing some ingredient that we need.
 There's some new consciousness particle or whatever.
 I happen to think that we're not missing anything
 and that it's not the interesting thing
 about consciousness that gives us
 this amazing subjective experience of colors and sounds
 and emotions and so on is rather something at the higher level
 about the patterns of information processing.
 And that's why I like to think about this idea
 of perceptronium.
 What does it mean for an arbitrary physical system
 to be conscious in terms of what its particles are doing
 or its information is doing?
 I don't think, I hate carbon chauvinism,
 this attitude you have to be made of carbon atoms
 to be smart or conscious.
 So something about the information processing
 that this kind of matter performs.
 Yeah, and you can see I have my favorite equations here
 describing various fundamental aspects of the world.
 I feel that I think one day maybe someone who's watching
 this will come up with the equations that information
 processing has to satisfy to be conscious.
 I'm quite convinced there is big discovery to be made there
 because let's face it, we know that some information
 processing is conscious because we are conscious.
 But we also know that a lot of information processing
 is not conscious.
 Like most of the information processing happening
 in your brain right now is not conscious.
 There are like 10 megabytes per second coming in
 even just through your visual system.
 You're not conscious about your heartbeat
 or your blood pressure or your blood pressure
 or your blood pressure regulation or most things.
 Even if I just ask you to read what it says here,
 you look at it and then, oh, now you know what it said.
 You're not aware of how the computation actually happened.
 You're like, your consciousness is like the CEO
 that got an email at the end with the final answer.
 So what is it that makes a difference?
 I think that's both of those great science mystery,
 we're actually studying it a little bit
 but it's just a really urgent question to answer.
 For starters, I mean, if you're an emergency room doctor
 and you have an unresponsive patient coming in,
 wouldn't it be great if in addition to having
 a CT scanner, you had a consciousness scanner
 that could figure out whether this person
 is actually having locked in syndrome
 or is actually comatose.
 And in the future, imagine if we build robots
 or the machine that we can have really good conversations
 with, I think is most likely to happen.
 Wouldn't you wanna know if your home helper robot
 is actually experiencing anything or just like a zombie?
 Would you prefer it?
 What would you prefer?
 Would you prefer that it's actually unconscious
 so that you don't have to feel guilty
 about switching it off or giving boring chores?
 Or what would you prefer?
 Well, certainly I would prefer
 the appearance of consciousness.
 But the question is whether the appearance of consciousness
 is different than consciousness itself.
 And sort of to ask that as a question,
 do you think we need to understand what consciousness is,
 solve the hard problem of consciousness
 in order to build something like an AGI system?
 No, I don't think that.
 And I think we will probably be able to build things
 even if we don't answer that question.
 But if we wanna make sure that what happens
 is a good thing, we better solve it first.
 So it's a wonderful controversy you're raising there
 where you have basically three points of view
 about the hard problem.
 So there are two different points of view.
 They both conclude that the hard problem of consciousness
 is BS.
 On one hand, you have some people like Daniel Dennett
 who say this is consciousness is just BS
 because consciousness is the same thing as intelligence.
 There's no difference.
 So anything which acts conscious is conscious,
 just like we are.
 And then there are also a lot of people,
 including many top AI researchers I know,
 who say, oh, consciousness is just bullshit
 because of course machines can never be conscious.
 They're always gonna be zombies.
 You never have to feel guilty about how you treat them.
 And then there's a third group of people,
 including Giulio Tononi, for example,
 and another, and Christoph Koch and her brothers.
 I would put myself also in this middle camp
 who say that actually some information processing
 is conscious and some is not.
 So let's find the equation
 which can be used to determine which it is.
 And I think we've just been a little bit lazy
 kind of running away from this problem for a long time.
 It's been almost taboo even mentioned the C word
 in a lot of circles.
 Because, but we should stop making excuses.
 This is a science question.
 And there are ways we can even test any theory
 that makes predictions for this.
 And coming back to this helper robot, I mean,
 so you said you would want your helper robot
 to certainly act conscious and treat you
 like you have conversations with you and stuff.
 I think so.
 But wouldn't you feel a little bit creeped out
 if you realized that it was just like lost up tape recorder
 that was just zombie and was a faking emotion.
 Would you prefer that it actually had an experience
 or would you prefer that it's actually
 not experiencing anything so you don't have to feel guilty
 about what you do to it?
 It's such a difficult question because,
 it's like when you're in a relationship and you say,
 well, I love you and the other person said,
 I love you back.
 It's like asking, well, do they really love you back
 or are they just saying they love you back?
 Don't you really want them to actually love you?
 It's hard to really know the difference between
 everything seeming like there's consciousness present,
 there's intelligence present,
 there's affection, passion, love,
 and it actually being there.
 I'm not sure, do you have?
 But like, can I ask you a question about this?
 To make it a bit more pointed.
 So Mass General Hospital is right across the river, right?
 Yes.
 Suppose you're going in for a medical procedure
 and they're like, for anesthesia,
 what we're gonna do is we're gonna give you
 a muscle relaxant so you won't be able to move
 and you're gonna feel excruciating pain
 during the whole surgery
 but you won't be able to do anything about it.
 But then we're gonna give you this drug
 that erases your memory of it.
 Would you be cool about that?
 What's the difference that you're conscious about it
 or not if there's no behavioral change, right?
 Right, that's a really clear way to put it.
 Yeah, it feels like in that sense,
 experiencing it is a valuable quality.
 So actually being able to have subjective experiences,
 at least in that case, is valuable.
 And I think we humans have a little bit
 of a bad track record also
 of making these self-serving arguments
 that other entities aren't conscious.
 People often say, oh, these animals can't feel pain.
 Right.
 It's okay to boil lobsters because we asked them
 if it hurt and they didn't say anything.
 And now there was just a paper out saying lobsters
 did do feel pain when you boil them
 and they're banning it in Switzerland.
 And we did this with slaves too often
 and said, oh, they don't mind.
 They don't maybe aren't conscious
 or women don't have souls or whatever.
 So I'm a little bit nervous when I hear people
 just take as an axiom that machines
 can't have experience ever.
 I think this is just a really fascinating science question
 is what it is.
 Let's research it and try to figure out what it is
 that makes the difference between unconscious,
 intelligent behavior and conscious, intelligent behavior.
 So in terms of, so if you think of it,
 Boston Dynamics humanoid robot being sort of
 with a broom being pushed around,
 it starts pushing on its consciousness question.
 So let me ask, do you think an AGI system,
 like a few neuroscientists believe
 needs to have a physical embodiment?
 It needs to have a body or something like a body.
 No, I don't think so.
 You mean to have a conscious experience?
 To have consciousness.
 I do think it helps a lot to have a physical embodiment
 to learn the kind of things about the world
 that are important to us humans for sure.
 But I don't think the physical embodiment
 is necessary after you've learned it,
 just have the experience.
 Think about when you're dreaming, right?
 Your eyes are closed.
 You're not getting any sensory input.
 You're not behaving or moving in any way,
 but there's still an experience there, right?
 And so clearly the experience that you have
 when you see something cool in your dreams
 isn't coming from your eyes.
 It's just the information processing itself in your brain,
 which is that experience, right?
 But if I put it another way,
 I'll say because it comes from neuroscience
 is the reason you wanna have a body and a physical,
 something like a physical,
 a physical system is because you want to be able
 to preserve something.
 In order to have a self, you could argue,
 you'd need to have some kind of embodiment of self
 to want to preserve.
 Well, now we're getting a little bit anthropomorphic.
 Anthropomorphizing things,
 maybe talking about self-preservation instincts.
 I mean, we are evolved organisms, right?
 Right.
 So Darwinian evolution endowed us
 and other evolved organism with a self-preservation instinct
 because those that didn't have those self-preservation genes
 got cleaned out of the gene pool, right?
 But if you build an artificial general intelligence,
 the mind space that you can design is much, much larger
 than just a specific subset of minds that can evolve.
 So an AGI mind doesn't necessarily have
 to have any self-preservation instinct.
 It also doesn't necessarily have to be so individualistic
 as us, like imagine if you could just,
 first of all, or we're also very afraid of death.
 Suppose you could back yourself up every five minutes
 and then your airplane is about to crash.
 You're like, shucks, I'm gonna lose the last five minutes
 of experiences since my last cloud backup, bang.
 It's not as big a deal.
 Or if we could just copy experiences
 between our minds easily, which we could easily do
 if we were silicon-based, right?
 Then maybe we would feel a little bit more
 like a hive mind actually, maybe it's the...
 So I don't think we should take for granted at all
 that AGI will have to have any of those sort of
 competitive as alpha male instincts.
 Right.
 And this is really interesting because I think some people
 go too far and say, of course, we don't have
 to have any concerns either that advanced AI
 will have those instincts because we can build anything
 we want, that there's a very nice set of arguments
 going back to Steve Omohundro and Nick Bostrom and others
 just pointing out that when we build machines,
 we normally build them with some kind of goal,
 win this chess game, drive this car safely
 or whatever.
 And as soon as you put in a goal into machine,
 especially if it's kind of open-ended goal
 and the machine is very intelligent,
 it'll break that down into a bunch of sub goals.
 And one of those goals will almost always
 be self-preservation because if it breaks or dies
 in the process, it's not gonna accomplish the goal, right?
 Like suppose you just build a little,
 you have a little robot and you tell it to go down
 to star market here and get you some food,
 make you cook you an Italian dinner, you know,
 and then someone mugs it and tries to break it on the way.
 That robot has an incentive to not get destroyed
 and defend itself for runaway because otherwise
 it's gonna fail in cooking you dinner.
 It's not afraid of death, but it really wants to
 complete the dinner cooking goal.
 So it will have a self-preservation instinct.
 Continue being a functional agent.
 Yeah.
 And similarly, if you give any kind of more
 ambitious goal to an AGI, it's very likely to wanna
 acquire more resources so it can do that better.
 And it's exactly from those sort of sub goals
 that we might not have intended that some of the concerns
 about AGI safety come.
 You give it some goal that seems completely harmless.
 And then before you realize it, it's also trying to do
 these other things which you didn't want it to do
 and it's maybe smarter than us.
 So, so.
 Fascinating.
 Let me pause just because I am in a very kind of
 human centric way, see fear of death as a valuable motivator.
 So you don't think, you think that's an artifact
 of evolution, so that's the kind of mind space
 evolution created that we're sort of almost obsessed
 about self-preservation, some kind of genetic well.
 You don't think that's necessary to be afraid of death.
 So not just a kind of sub goal of self-preservation
 just so you can keep doing the thing, but more
 fundamentally sort of have the finite thing,
 like this ends for you at some point.
 Interesting.
 Do I think it's necessary for what precisely?
 For intelligence, but also for consciousness.
 So for those, for both, do you think really
 like a finite death and the fear of it is important?
 So before I can answer, before we can agree on whether
 it's necessary for intelligence or for consciousness,
 we should be clear on how we define those two words
 because a lot of really smart people define them
 in very different ways.
 I was in this, on this panel with AI experts
 and they couldn't agree on how to define intelligence even.
 So I define intelligence simply as the ability
 to accomplish complex goals.
 I like your broad definition because again,
 I don't wanna be a carbon chauvinist.
 Right.
 And in that case, no, certainly it doesn't require
 fear of death.
 I would say alpha go, alpha zero is quite intelligent.
 I don't think alpha zero has any fear of being turned off
 because it doesn't understand the concept of it even.
 And similarly, consciousness, I mean, you can certainly
 imagine very simple kind of experience.
 If certain plants have any kind of experience,
 I don't think they're very afraid of dying
 or there's nothing they can do about it anyway.
 So there wasn't much value.
 But more seriously, I think if you ask not just
 about being conscious, but maybe having
 what we might call an exciting life
 where you feel passion and really appreciate
 the things, maybe there perhaps it does help
 having a backdrop that, hey, it's finite.
 Let's make the most of this, let's live to the fullest.
 So if you knew you were gonna just live forever,
 do you think you would change your?
 Yeah, I mean, in some perspective, it would be
 an incredibly boring life living forever.
 So in the sort of loose subjective terms that you said
 of something exciting and something in this
 that other humans would understand, I think, is yeah,
 it seems that the finiteness of it is important.
 Well, the good news I have for you then is
 based on what we understand about cosmology,
 everything in our universe is ultimately probably finite.
 Big crunch or big, what's the infinite expansion?
 Yeah, we could have a big chill or a big crunch
 or a big rip or the big snap or death bubbles.
 All of them are more than a billion years away.
 So we certainly have vastly more time
 than our ancestors thought, but there is still
 pretty hard to squeeze in an infinite number
 of compute cycles, even though there are some loopholes
 that just might be possible.
 But I think some people like to say that you should live
 as if you're gonna die in five years or so,
 and that's sort of optimal.
 Maybe it's a good assumption, we should build
 our civilization as if it's all finite
 to be on the safe side.
 Right, exactly.
 So you mentioned defining intelligence as
 the ability to solve complex goals.
 Where would you draw a line?
 How would you try to define human level intelligence
 and superhuman level intelligence?
 Where is consciousness part of that definition?
 No, consciousness does not come into this definition.
 So I think of intelligence as a spectrum,
 but there are very many different kinds of goals you can have.
 You can have a goal to be a good chess player,
 a good go player, a good car driver,
 a good investor, good poet, et cetera.
 So intelligence by its very nature
 isn't something you can measure,
 but it's one number, some overall goodness.
 No, no, there are some people who are better at this,
 some people are better at that.
 Right now we have machines that are much better than us
 at some very narrow tasks, like multiplying large numbers fast,
 memorizing large databases, playing chess,
 playing Go, and soon driving cars.
 But there's still no machine that can match a human child
 in general intelligence.
 But artificial general intelligence,
 AGI, the name of your course, of course,
 that is by its very definition,
 the quest to build a machine that can do everything
 as well as we can.
 So the old holy grail of AI from back to its inception
 in the 60s.
 If that ever happens, of course,
 I think it's going to be the biggest transition
 in the history of life on earth.
 But it doesn't necessarily have to wait the big impact
 until machines are better than us at knitting.
 The really big change doesn't come exactly at the moment.
 They're better than us at everything.
 The really big change comes first.
 Their big change is when they start becoming better at us
 at doing most of the jobs that we do,
 because that takes away much of the demand for human labor.
 And then the really whopping change
 comes when they become better than us at AI research.
 Because right now, the time scale of AI research
 is limited by the human research and development cycle of years,
 typically.
 How long does it take from one release of some software
 or iPhone or whatever to the next?
 But once Google can replace 40,000 engineers
 by 40,000 equivalent pieces of software or whatever,
 then there's no reason that has to be years.
 It can be, in principle, much faster.
 And the time scale of future progress in AI
 and all of science and technology
 will be driven by machines, not humans.
 So it's this simple point which gives right
 to this incredibly fun controversy about whether there
 can be intelligence explosion, so-called singularity,
 as Werner Wienz called it.
 The idea, as articulated by I.J. Good,
 is obviously way back in the 50s.
 But you can see Alan Turing and others thought about it
 even earlier.
 You asked me what exactly would I define human level at.
 So the glib answer is to say something
 which is better than us at all cognitive tasks,
 or better than any human at all cognitive tasks.
 But the really interesting bar, I
 think, goes a little bit lower than that, actually.
 It's when they're better than us at AI
 programming and general learning,
 so that they can, if they want to,
 get better than us at anything by just studying up.
 So there, better is a key word.
 And better is towards this spectrum
 of the complexity of goals it's able to accomplish.
 So another way to, and that's certainly
 a very clear definition of human level.
 So it's almost like a sea that's rising.
 You can do more and more and more things.
 It's actually a graphic that you show.
 It's a really nice way to put it.
 So there's some peaks, and there's
 an ocean level elevating, and you
 solve more and more problems.
 But just to take a pause, and we took a bunch of questions
 and a lot of social networks.
 And a bunch of people asked a slightly different direction
 on creativity, on things that perhaps aren't a peak.
 Human beings are flawed, and perhaps better
 means having contradiction, being flawed in some way.
 So let me sort of start easy, first of all.
 So you have a lot of cool equations.
 Let me ask, what's your favorite equation, first of all?
 I know they're all like your children, but which one is that?
 This is the Schrodinger equation.
 It's the master key of quantum mechanics of the micro world.
 So with this equation, we can calculate everything
 to do with atoms, molecules, and all the way up to them.
 Yeah, so quantum mechanics is certainly
 a beautiful, mysterious formulation of our world.
 So I'd like to ask you, just as an example,
 it perhaps doesn't have the same beauty as physics does,
 but in mathematics, abstract, the Andrew Wiles, who
 proved the Fermat's last theorem.
 So I just saw this recently, and it
 kind of caught my eye a little bit.
 This is 358 years after it was conjectured.
 So this very simple formulation.
 Everybody tried to prove it.
 Everybody failed.
 And so here's this guy comes along, and eventually proves it
 and then fails to prove it and then proves it again.
 In 94, and he said the moment when everything connected
 into place, in an interview he said,
 it was so indescribably beautiful.
 That moment when you finally realized
 the connecting piece of two conjectures, he said,
 it was so indescribably beautiful.
 It was so simple and so elegant.
 I couldn't understand how I'd missed it,
 and I just stared at it in disbelief for 20 minutes.
 Then during the day, I walked around the department,
 and I'd keep coming back to my desk,
 looking to see if it was still there.
 It was still there.
 I couldn't contain myself.
 I was so excited.
 It was the most important moment of my working life.
 Nothing I ever do again will mean as much.
 So that particular moment, and it kind of made me think of,
 what would it take?
 And I think we have all been there at small levels.
 Maybe you let me ask, have you had a moment like that
 in your life where you just had an idea?
 It's like, wow, yes.
 I wouldn't mention myself in the same breath as Andrew Wiles,
 but I've certainly had a number of aha moments
 when I realized something very cool about physics
 just completely made my head explode.
 In fact, some of my favorite discoveries I made,
 I later realized that they had been discovered earlier
 by someone who sometimes got quite famous for it.
 So it's too late for me to even publish it,
 but that doesn't diminish in any way
 the emotional experience you have when you realize it.
 Like, wow.
 Yeah, so what would it take in that moment, that wow,
 that was yours in that moment?
 So what do you think it takes for an intelligent system,
 an AGI system, an AI system to have a moment like that?
 That's a tricky question
 because there are actually two parts to it, right?
 One of them is, can it accomplish that proof?
 Can it prove that you can never write
 A to the N plus B to the N equals three to that equals Z to the N
 for all integers, et cetera, et cetera,
 when one end is bigger than two?
 That's simply a question about intelligence.
 Can you build machines that are that intelligent?
 And I think by the time we get a machine
 that can independently come up with that level of proofs,
 probably quite close to AGI.
 The second question is a question about consciousness.
 When will we, how likely is it that such a machine
 would actually have any experience at all
 as opposed to just being like a zombie?
 And would we expect it to have some sort of emotional response
 to this or anything at all akin to human emotion
 where when it accomplishes its machine goal,
 it reviews it as somehow something very positive
 and sublime and deeply meaningful?
 I would certainly hope that if in the future
 we do create machines that are our peers
 or even our descendants,
 that I would certainly hope that they do have
 this sort of sublime appreciation of life.
 In a way, my absolutely worst nightmare would be that,
 at some point in the future,
 the distant future maybe our cosmos is teeming
 with all this post-biological life,
 doing all the seemingly cool stuff.
 And maybe the last humans by the time our species eventually
 fizzles out will be like, well, that's okay,
 because we're so proud of our descendants here
 and look at what all the...
 My worst nightmare is that we haven't solved
 the consciousness problem and we haven't realized
 that these are all the zombies.
 They're not aware of anything any more than a tape recorder
 as an any kind of experience.
 So the whole thing has just become a play for empty benches.
 That would be like the ultimate zombie apocalypse.
 So I would much rather in that case that we have these beings
 which can really appreciate how amazing it is.
 And in that picture, what would be the role of creativity?
 A few people ask about creativity.
 Do you think, when you think about intelligence,
 I mean, certainly the story you told at the beginning
 of your book involved creating movies and so on.
 Yeah.
 Sort of making money.
 You can make a lot of money in our modern world
 with music and movies.
 So if you are an intelligent system,
 you may want to get good at that.
 But that's not necessarily what I mean by creativity.
 Is it important on that complex goals where the sea is rising
 for there to be something creative?
 Or am I being very human-centric
 and thinking creativity is somehow special
 relative to intelligence?
 My hunch is that we should think of creativity
 simply as an aspect of intelligence.
 And we have to be very careful with human vanity.
 We have this tendency to very often want to say,
 as soon as machines can do something,
 we try to diminish it and say,
 oh, but that's not like real intelligence.
 Because they're not creative or this or that.
 The other thing, if we ask ourselves to write down
 a definition of what we actually mean
 by being creative, what we mean by Andrew Wiles,
 what he did there, for example,
 don't we often mean that someone takes a very unexpected leap?
 Mm-hmm.
 It's not like taking 573 and multiplying it by 224
 by just a step of straightforward cookbook-like rules, right?
 You can maybe make a connection between two things
 that people have never thought was connected.
 It's very surprising.
 Or something like that.
 I think this is an aspect of intelligence.
 And this is actually one of the most important aspects of it.
 Maybe the reason we humans tend to be better at it
 than traditional computers is because
 it's something that comes more naturally
 if you're a neural network
 than if you're a traditional logic gate-based computer machine.
 We physically have all these connections.
 And if you activate here, activate here, activate here,
 it's ping!
 You know, my hunch is that if we ever build a machine
 where you could just give it the task,
 hey, hey, you say, hey, you know,
 I just realized I want to travel around the world instead this month.
 Can you teach my AGI course for me?
 And it's like, OK, I'll do it.
 I'll do the other thing that you would have done,
 and the improvisers and stuff.
 That would, in my mind, involve a lot of creativity.
 Yeah, so it's actually a beautiful way to put it.
 I think we do try to grasp at the, you know,
 the definition of intelligence is everything we don't understand how to build.
 So we as humans try to find things that we have and machines don't have.
 Maybe creativity is just one of the things,
 one of the words we use to describe that.
 That's a really interesting way to put it.
 I don't think we need to be that defensive.
 I don't think anything good comes out of saying,
 oh, we're somehow special, you know.
 It's, contrary-wise, there are many examples in history of where
 trying to pretend that we're somehow superior to all other intelligent beings
 led to pretty bad results, right?
 In Nazi Germany, they said that they were somehow superior to other people.
 Today, we still do a lot of cruelty to animals by saying that we're so superior somehow,
 and they can't feel pain.
 Slavery was justified by the same kind of really weak arguments.
 I don't think if we actually go ahead and build artificial general intelligence,
 we can do things better than us.
 I don't think we should try to found our self-worth on some sort of bogus claims of superiority
 in terms of our intelligence.
 I think we should instead find our calling and the meaning of life from the experiences that we have.
 I can have very meaningful experiences, even if there are other people who are smarter than me.
 When I go to a faculty meeting here, and we're talking about something,
 and I suddenly realize, oh, he has an old prize, he has an old prize, he has an old prize.
 Does that make me enjoy life any less or enjoy talking to those people less?
 Of course not.
 Contrary, I feel very honored and privileged to get to interact with other very intelligent beings
 that are better than me at a lot of stuff.
 I don't think there's any reason why we can't have the same approach with intelligent machines.
 That's a really interesting...
 People don't often think about that.
 They think about if there's machines that are more intelligent,
 you naturally think that that's not going to be a beneficial type of intelligence.
 You don't realize it could be peers with Nobel prizes that would be just fun to talk with.
 They might be clever about certain topics, and you can have fun having a few drinks with them.
 Also, another example we can all relate to of why it doesn't have to be a terrible thing to be impressed
 with people who are even smarter than us all around is when you and I were both two years old.
 Our parents were much more intelligent than us.
 It worked out okay because their goals were aligned with our goals.
 That, I think, is really the number one key issue we have to solve.
 The value alignment problem.
 People who see too many Hollywood movies with lousy science fiction plot lines,
 they worry about the wrong thing.
 They worry about some machine suddenly turning evil.
 It's not malice that's the issue, the concern.
 It's competence.
 By definition, intelligence makes you very competent.
 If you have a more intelligent go-playing computer, playing as a less intelligent one,
 and when we define intelligence as the ability to accomplish go-winning,
 it's going to be the more intelligent one that wins.
 If you have a human and then you have an AGI that's more intelligent in all ways
 and they have different goals, guess who's going to get their way?
 I was just reading about this particular rhinoceros species that was driven extinct just a few years ago.
 It was a bummer.
 I was looking at this cute picture of a mommy rhinoceros with its child.
 Why did we humans drive it to extinction?
 It wasn't because we were evil rhino haters as a whole.
 It was just because our goals weren't aligned with those of the rhinoceros,
 and it didn't work out so well for the rhinoceros because we were more intelligent.
 I think it's just so important that if we ever do build AGI,
 before we unleash anything, we have to make sure that it learns to understand our goals,
 that it adopts our goals, and it retains those goals.
 So the cool, interesting problem there is us as human beings trying to formulate our values.
 You could think of the United States Constitution as a way that people sat down,
 at the time a bunch of white men, which is a good example, we should say.
 They formulated the goals for this country, and a lot of people agree that those goals actually held up pretty well.
 It's an interesting formulation of values and failed miserably in other ways.
 So for the value alignment problem and the solution to it,
 we have to be able to put on paper or in a program human values.
 How difficult do you think that is?
 Very. But it's so important. We really have to give it our best.
 And it's difficult for two separate reasons.
 There's the technical value alignment problem of figuring out how to make machines understand our goals, adopt them, and retain them.
 And then there's the separate part of it, the philosophical part, whose values anyway?
 And since it's not like we have any great consensus on this planet on values,
 what mechanism should we create then to aggregate and decide, OK, what's a good compromise?
 That second discussion can't just be left the tech nerds like myself.
 That's right. And if we refuse to talk about it and then AGI gets built, who's going to be actually making the decision about whose values?
 It's going to be a bunch of dudes and some tech company.
 And are they necessarily so representative of all of humankind that we want to just entrust it to them?
 Are they even uniquely qualified to speak to future human happiness just because they're good at programming AI?
 I'd much rather have this be a really inclusive conversation.
 But do you think it's possible sort of that?
 So you create a beautiful vision that includes sort of the diversity, cultural diversity and various perspectives on discussing rights, freedoms, human dignity.
 But how hard is it to come to that consensus?
 Do you think it's certainly a really important thing that we should all try to do.
 But do you think it's feasible?
 I think there's no better way to guarantee failure than to refuse to talk about it or refuse to try.
 And I also think it's a really bad strategy to say, OK, let's first have a discussion for a long time.
 And then once we reach complete consensus, then we'll try to load it into some machine.
 No, we shouldn't let perfect be the enemy of good.
 Instead, we should start with the kindergarten ethics that pretty much everybody agrees on and put that into machines.
 Now, we're not doing that even.
 Look at anyone who builds a passenger aircraft wants to never under any circumstances fly into a building or mountain.
 Right. Yet the September 11 hijackers were able to do that.
 And even more embarrassingly, you know, and the gas lube, it's this depressed German wings pilot when he flew his passenger jet into the Alps, killing over 100 people.
 He just told the autopilot to do it. He told the freeing computer to change the altitude to 100 meters.
 And even though it had the GPS maps, everything the computer was like, OK, no.
 So we we should take those very basic values where the problem is not that we don't agree.
 The problem is just we've been too lazy to try to put it into our machines and make sure that from now on, airplanes will just which all have computers in them.
 We just never just refuse to do something like that.
 Go into safe mode, maybe lock the cockpit door, go to the nearest airport.
 And and there's so much other technology in our world as well now where it's really quite becoming quite timely to put in some sort of very basic values like this.
 Even in cars, we have had enough vehicle terrorism attacks by now.
 If you have driven trucks and vans into pedestrians, that is not at all a crazy idea to just have that hardwired into the car.
 Because, yeah, there are a lot of there's always going to be people who for some reason want to harm others.
 But most of those people don't have the technical expertise to figure out how to work around something like that.
 So if the car just won't do it. It helps.
 So let's start there.
 So there's a lot of that's a that's a great point.
 So not not chasing perfect.
 There's a lot of things that a lot that most of the world agrees on.
 Yeah, and start there.
 Let's start there.
 And then once we start there, we'll also get into the habit of having these kind of conversations about, OK, what else should we put in here and have these discussions?
 This should be a gradual process.
 Great. So but that also means describing these things and describing it to a machine.
 So one thing we had a few conversations with Stephen Wolfram.
 I'm not sure if you're familiar with Stephen.
 Oh, yeah, I know him quite well.
 So he is, you know, he play, you know, works with a bunch of things, but, you know, cellular automata, the simple, computable things, these computation systems.
 And you kind of mentioned that, you know, we probably have already within these systems already something that's a GI, meaning like we just don't know it because we can't talk to it.
 So if you give me this chance to try to at least form a question out of this is I think it's an interesting idea to think that we can have intelligent systems, but we don't know how to describe something to them and they can't communicate with us.
 I know you're doing a little bit of work and explainable I trying to get a to explain itself.
 So what are your thoughts of natural language processing or some kind of other communication?
 How how does the AI explain something to us?
 How do we explain something to it to machines?
 Or you think of it differently?
 So there are two separate parts to your question there.
 One of them has to do with communication, which is super interesting.
 You only get to that in a sec.
 The other is whether we already have a GI, but we just haven't noticed it.
 Right.
 There I beg to differ.
 I don't think there's anything in any cellular automaton or anything or the Internet itself or whatever that has artificial general intelligence in that it's really do exactly everything we humans can do better.
 I think today if the day that happens, when that happens, we will very soon notice and will probably notice even before.
 And if because in a very, very big way.
 But for the for the second part, though, I can't.
 I'm sorry. So the because you have this beautiful way to formulate in consciousness as as a you know, as information processing.
 You can think of intelligence information processing and this you can think of the entire universe is these particles and these systems roaming around that have this information processing power.
 You don't you don't think there is something with the power to process information in the way that we human beings do that's out there that that needs to be sort of connected to.
 It seems a little bit philosophical, perhaps, but there's something compelling to the idea that the power is already there.
 Would you guess the focus should be more on the on being able to communicate with it?
 Well, I agree that in some in a certain sense, the hardware processing power is already out there because our universe itself can think of it as being a computer already.
 Right. It's constantly computing what water waves how to evolve the water waves and the river Charles and how to move the air molecules around.
 And Seth Lloyd has pointed out my colleague here that you can even in a very rigorous way think of our entire universe is being a quantum computer.
 It's pretty clear that our universe supports this amazing processing power because you can even within this physics computer that we live in, right.
 We can even build actually laptops and stuff. So clearly the power is there.
 It's just that most of the compute power that nature has, it's, in my opinion, kind of wasting on boring stuff like simulating yet another ocean wave somewhere where no one is even looking.
 So in a sense of what life does, what we are doing when we build computers is we're rechanneling all this compute that nature is doing anyway and doing things that are more interesting than just yet another ocean wave.
 You know, and let's do something cool here.
 So the raw hardware power is there and for sure.
 But and then even just like computing what's going to happen for the next five seconds in this water ball, you know, it takes a ridiculous amount of compute if you do it on a human computer.
 This water ball just did it.
 But that does not mean that this water ball has AGI and because AGI means it should also be able to like have written my book, done this interview.
 Yes. And I don't think it's just communication problems.
 I don't really know. Don't think it can do it.
 Although Buddhists say when they watch the water and that there is some beauty, that there is some depth and nature that they can communicate with.
 Communication is also very important though because I mean, look, part of my job is being a teacher.
 And I know some very intelligent professors even who just have a better hard time communicating.
 They come up with all these brilliant ideas.
 But to communicate with somebody else, you have to also be able to simulate their own mind.
 Yes. And build well enough and understand model of their mind that you can say things that they will understand.
 And that's quite difficult.
 And that's why today it's so frustrating if you have a computer that makes some cancer diagnosis and you ask it, well, why are you saying I should have a surgery?
 And if it can only reply, I was trained on five terabytes of data and this is my diagnosis.
 Boop, boop, beep, beep.
 Yeah.
 Doesn't really instill a lot of confidence, right?
 Right.
 So I think we have a lot of work to do on communication there.
 So what kind of I think you're doing a little bit work in Explainable AI.
 What do you think are the most promising avenues?
 Is it mostly about sort of the Alexa problem of natural language processing of being able to actually use human interpretable methods of communication?
 So being able to talk to a system and talk back to you, or is there some more fundamental problems to be solved?
 I think it's all of the above human.
 The natural language processing is obviously important, but there are also more nerdy fundamental problems.
 Like if you take, you play chess.
 Of course, I'm Russian.
 I have to.
 You speak Russian?
 Yes, I speak Russian.
 Great, I didn't know.
 When did you learn Russian?
 I speak very badly in Russian.
 I only did after-deduct, but I bought a book called Teach Yourself Russian.
 I read a lot, but it was very difficult.
 Wow.
 That's why I speak so badly.
 How many languages do you know?
 Wow, that's really impressive.
 I don't know.
 My wife has some congratulations.
 But my point was, if you play chess, have you looked at the AlphaZero games?
 The actual games, no.
 Check it out.
 Some of them are just mind-blowing.
 Really beautiful.
 And if you ask, how did it do that?
 You got to talk to Demis El-Sabes and others from DeepMind.
 All they'll ultimately be able to give you is big tables of numbers, matrices that define the neural network.
 And you can stare at these numbers until your face turns blue.
 And you're not going to understand much about why it made that move.
 And even if you have natural language processing that can tell you in human language about, oh, five, seven, points, two, eight, still not going to really help.
 So I think there's a whole spectrum of fun challenges involved in taking a computation that does intelligent things and transforming it into something.
 Equally good, equally intelligent.
 But that's more understandable.
 And I think that's really valuable because I think.
 As we put machines in charge of ever more infrastructure in our world, the power grid, the trading on the stock market, weapons systems and so on.
 It's absolutely crucial that we can trust these AIs that do all we want.
 And trust really comes from understanding in a very fundamental way.
 And that's why I'm that's why I'm working on this, because I think the more if we're going to have some hope of ensuring that machines have adopted our goals and that they're going to retain them.
 That kind of trust.
 I think needs to be based on things you can actually understand, preferably even make it have preferably improved theorems on even with a self-driving car.
 Right.
 If someone just tells you it's been trained on tons of data and it never crashed, it's less reassuring than if someone actually has a proof.
 Maybe it's a computer verified proof.
 But still, it says that under no circumstances is this car just going to swerve into oncoming traffic.
 And that kind of information helps to build trust and build the alignment, the alignment of goals, at least awareness that your goals, your values are aligned.
 And I think even in the very short term, if you look at how, you know, today, right, this absolutely pathetic state of cybersecurity that we have.
 Right.
 Where is it?
 Three billion Yahoo accounts were kept packed.
 Almost every.
 Americans credit card and so on.
 Why is this happening?
 It's ultimately happening because we have software that nobody fully understood how it worked.
 That's why the bugs hadn't been found.
 Right.
 And I think AI can be used very effectively for offense, for hacking, but it can also be used for defense.
 Hopefully, automating verifiability and creating systems that are.
 Built in different ways so you can actually prove things about them, right. And it's it's important.
 So speaking of software that nobody understands how it works, of course, a bunch of people ask about your paper, about your thoughts of why does deep and cheap learning work so well.
 That's the paper. But what are your thoughts on deep learning?
 These kind of simplified models of our own brains have been able to do some successful perception work, pattern recognition work.
 And now with Alpha Zero and so on, do some some clever things.
 What are your thoughts about the promise limitations of this piece?
 Right.
 I think.
 There are a number of very important insights.
 Very important lessons we can all withdraw from these kind of successes.
 One of them is when you look at the human brain, you see it's very complicated.
 Tenth of eleven neurons and there are all these different kinds of neurons and yada yada. And there's been this long debate about whether the fact that we have dozens of different kinds is actually necessary for intelligence.
 We can now, I think, quite convincingly answer that question.
 No, it's enough to have just one kind.
 If you look under the hood of Alpha Zero, there's only one kind of neuron and it's ridiculously simple, simple mathematical thing.
 But it's not it's just like in physics, it's not the if you have a gas with waves in it, it's not the detailed nature of the molecules that matter.
 It's the collective behavior somehow.
 Similarly, it's it's it's this higher level structure of the network matters, not that you have 20 kinds of neurons.
 I think our brain is such a complicated mess because it wasn't evolved.
 Just to be intelligent, it was evolved to also be self assembling.
 Right. And self repairing.
 Right. And evolutionarily attainable and so on.
 And so on. Yeah.
 So I think it's pretty my my hunch is that we're going to understand how to build AGI before we fully understand how our brains work.
 Just like we we understood how to build flying machines long before we were able to build a mechanical work bird.
 Yeah, that's right. You've given things you're given that the example exactly of mechanical birds and airplanes and airplanes do a pretty good job of flying without really mimicking bird flight.
 And even now, after 100, 100 years later, did you see the TED talk with this German mechanical bird?
 I heard you mention it. I want to see it. It's amazing.
 But even after that, we still don't fly in mechanical birds because it turned out the way we came up with was simpler and better for our purposes.
 And I think it might be the same there. That's one lesson.
 Another lesson, which is more what what our paper was about.
 First, I as a physicist thought it was fascinating how there's a very close mathematical relationship actually between our artificial neural networks and a lot of things that we've studied for in physics.
 We've studied by nerdy names like the renormalization group equation and Hamiltonians and yada yada yada. And when you look a little more closely at this.
 You have.
 You.
 At first, I was like, well, there's something crazy here that doesn't make sense because.
 That if you even want to build a super simple neural network, tell that part cat pictures and dog pictures, right? You can do that very, very well now.
 But if you think about it a little bit, you convince yourself it must be impossible because if I have one megapixel, even if each pixel is just black or white, there's two to the power one million possible images is way more than their atoms in our universe.
 Right. So in order to.
 And then for each one of those, I have to assign a number, which is the probability that it's a dog.
 So an arbitrary function of images is a list of.
 More numbers than their atoms in our universe. So clearly I can't store that under the hood of my my GPU or my computer yet somehow works.
 So what does that mean? Well, it means that the out of all of the problems that you could try to solve with a neural network.
 Almost all of them are impossible to solve with a reasonably sized one.
 But then what we should show it in our paper was was that the fraction, the kind of problems, the fraction of all the problems that you could possibly pose that that we actually care about, given the laws of physics is also an infinite test.
 Let me tie me a little part. And amazingly, they're basically the same part.
 Yeah, it's almost like our world was created for. I mean, they kind of come together.
 Yeah, you, well, you could say maybe where the world created the world, the world was created for us, but I have a more modest interpretation, which is that instead evolution endowed us with neural networks, precisely for that reason.
 Right. Because this particular architecture, as opposed to the one in your laptop, is very, very well.
 Adapted to solving the kind of problems that nature kept presenting our ancestors with. Right. So it makes sense that why do we have a brain in the first place? It's to be able to make predictions about the future and so on.
 So if we had a sucky system, which could never solve it, wouldn't have a lot. So, but it's so this is this is I.
 I think you're a beautiful fact. Yeah, we also, we also realize that there's the there that we there's been earlier work on on why deeper networks are good, but we were able to show an additional cool fact there, which is that even incredibly simple problems like suppose I give you a.
 A thousand numbers and ask you to multiply them together and you can write a few lines of code boom done trivial. If you just try to do that with a neural network that has only one single hidden layer in it, you can do it.
 But you're going to need two to the power of thousand neurons.
 To multiply a thousand numbers, which is again more neurons and their atoms in our universe.
 That's fascinating. But if you're allowed, if you allow yourself, make it a deep network with many layers, you only need four thousand neurons.
 It's perfectly feasible. So that's really interesting. Yeah. Yeah. So on another architecture type, I mean, you mentioned Schrodinger's equation and what are your thoughts about quantum computing?
 And the role of this kind of computational unit in creating an intelligence system in some Hollywood movies that are a lot mentioned by name because I want to spoil them.
 The way they get a G.I. is building a quantum computer was because the word quantum sounds cool and so on.
 But my first of all, I think we don't need quantum computers to build a G.I. I suspect your brain is not quantum computer and any found sense.
 So you don't even wrote a paper about that many years ago.
 I calculated the decoherence so-called decoherence time, how long it takes until the quantum computer in this of what your neurons are doing gets erased.
 It's just random noise from the environment and it's about 10 to the minus 21 seconds.
 So as cool as it would be to have a quantum computer in my head, I don't think that fast.
 On the other hand, there are. Very cool things you could do with quantum computers.
 Or I think we'll be able to do soon when we get big one bigger ones that might actually help machine learning do even better than the brain.
 So. For example.
 One, this is just a moon shot, but hey, you know, not.
 Learning. It's very much the same thing as search.
 If you have if you're trying to train a neural network to get really learned to do something really well, you have some loss function.
 You have some you have a bunch of knobs you can turn represented by a bunch of numbers and you're trying to tweak them so that it becomes as good as possible at this thing.
 So if you think of a landscape with some valley.
 Where each dimension of the landscape corresponds to some number you can change, you're trying to find the minimum.
 And it's well known that if you have a very high dimensional landscape, complicated things, it's super hard to find the minimum.
 Quantum mechanics is amazingly good at this, right? If I want to know what's the lowest energy state this water can possibly have.
 Incredibly hard to compute, but we can. But nature will happily figure this out for you if you just cool it down, make it very, very cold.
 If you put a ball somewhere, it'll roll down to its minimum. And this happens metaphorically at the energy landscape, too.
 And quantum mechanics even uses some clever tricks, which today's machine learning systems don't like.
 You're trying to find the minimum and you get stuck in the little local minimum here in quantum mechanics.
 You can actually tunnel through the barrier and get unstuck again.
 And that's really interesting. So it's maybe, for example, we'll one day use quantum computers that.
 To help train neural networks better. That's really interesting. OK, so as a component of kind of the learning process, for example.
 Yeah. Let me ask sort of wrapping up here a little bit.
 Let me let me return to the questions of our human nature and and love, as I mentioned.
 So do you think. You mentioned sort of a helper robot, but you could think of also personal robots.
 Do you think the way we human beings fall in love and get connected to each other is possible to achieve in an AI system and human level AI intelligence system?
 Do you think we would ever see that kind of connection or, you know, in all this discussion about solving complex goals?
 Yeah. As this kind of human social connection, do you think that's one of the goals on the peaks and valleys that with the raising sea levels that we'll be able to achieve?
 Or do you think that's something that's ultimately or at least in the short term relative to the other goals is not achievable?
 I think it's all possible. And I mean, in recent there is a there is a very wide range of guesses, as you know, among AI researchers when we're going to get AGI.
 Some people, you know, like our friend Rodney Brooks said, it's going to be hundreds, hundreds of years at least.
 And then there are many others who think it's going to happen much sooner. And recent polls.
 Maybe half or so already, I research think it's we're going to get AGI within decades.
 So if that happens, of course, I think these things are all possible.
 But in terms of whether it will happen, I think we shouldn't spend so much time asking what do we think will happen in the future?
 As if we are just some sort of pathetic, passive bystanders, you know, waiting for the future to happen to us. Hey, we're the ones creating this future.
 Right. So we should be proactive about it and ask ourselves what sort of future we would like to have happen.
 That's right. When I make it like that. Well, what I prefer it to some sort of incredibly boring zombie like future where there's all these mechanical things happening.
 There's no passion, no emotion, no experience, maybe even. No, I would, of course, much rather prefer it if all the things that we find that we value the most about humanity are subjective experience, passion, inspiration, love.
 If we can create a future where those are those things do exist. I think ultimately it's not our universe giving meaning to us. It's us giving meaning to our universe.
 And if we build more advanced intelligence, let's make sure we're built in such a way that meaning is part of it.
 A lot of people that seriously study this problem and think of it from different angles have trouble in the majority of cases if they think through that happen are the ones that are not beneficial to humanity.
 Right. And so, yeah. So what are your thoughts? What's an inch? What's what should people, you know, I really don't like people to be terrified.
 What's the way for people to think about it in a way that in a way we can solve it and we can make it better.
 Yeah, no, I don't think panicking is going to help in any way. It's not going to increase chances of things going well either.
 Even if you are in a situation where there is a real threat, does it help if everybody just freaks out? Right. No, of course. Of course not.
 I think there are, of course, ways in which things can go horribly wrong. First of all, it's important when we think about this thing, this about the problems and risks to also remember how huge the upsides can be if we get it right.
 Right. Everything, everything we love about society and civilization is a product of intelligence. So if we can amplify our intelligence or machine intelligence and not anymore lose our loved one to what we're told is an uncurable disease and things like this, of course, we should aspire to that.
 So that can be a motivator, I think, reminding ourselves that the reason we try to solve problems is not just because.
 We're trying to avoid gloom, but because we're trying to do something great. But then in terms of the risks, I think.
 The really important question is to ask what can we do today that will actually help. Yes, outcome good. Right.
 And dismissing the risk is not one of them. You know, I find it quite funny often when I'm in discussion panels about these things, how the people who work for companies always say, always like, oh, nothing to worry about, nothing to worry about, nothing to worry about.
 And it's always all it's only academics sometimes express concerns. That's not surprising at all. If you think about it. Right. Often Sinclair equipped right that the charging make a man believe in something when his income depends on not believing in it.
 And, and frankly, we know a lot of these people in companies that they're just as concerned as anyone else. But if you're the CEO of a company, that's not something you want to go on record saying when you have silly journalists who are going to put a picture of a Terminator robot when they quote you.
 So, so the issues are real. And the way I think about what the issue is, is basically, you know, the real choice we have is, first of all, are we going to dismiss this, the risks and say, well, you know, let's just go ahead and build machines that can do everything we can do better and cheaper, you know, let's just make ourselves obsolete as fast as possible.
 What could possibly go wrong, right? That's one attitude. The opposite attitude, I think it's a say, it's incredible potential, you know, let's think about what kind of future.
 We're really, really excited about what are the shared goals that we can really aspire towards and then let's think really hard on how about how we can actually get there.
 So start with it, not don't start thinking about the risks, start thinking about the goals goals. Yeah. And then when you do that, then you can think about the obstacles you want to avoid, right? I often get students coming in right here into my office for career advice.
 I always ask them this very question. Where do you want to be in the future? Right? If all she can say is, oh, maybe I'll have cancer. Maybe I'll run over by a truck.
 And obstacles instead of the goal. She's just going to end up a hypochondriac paranoid. Whereas if she comes in and fire in her eyes and is like, I want to be there. And then we can talk about the obstacles and see how we can circumvent them.
 That's, I think, a much, much healthier attitude. And that's really well. And I, I, I feel it's very challenging to come up with a vision for the future, which we're, which we are unequivocally excited about.
 I'm not just talking now in the vague terms like, yeah, let's cure cancer. Fine. Talking about what kind of society that we want to create. What do we want it to mean, you know, to be human in the age of AI, in the age of AGI.
 So if we can have this conversation, broad, inclusive conversation, and gradually start converging towards some, some future that with some direction, at least that we want to steer towards right then, then we'll be much more motivated to constructively take on the obstacles.
 And I think if I had, if I had to, if you make, if I try to wrap this up in a more succinct way, I think, I think we can all agree already now that we should aspire to build AGI that doesn't overpower us, but that empowers us.
 And think of the many various ways that can do that, whether that's from my side of the world of autonomous vehicles. I personally actually from the camp that believes this human level intelligence is required to, to achieve something like vehicles that would actually be something we would enjoy using and being part of.
 So that's the one example and certainly there's a lot of other types of robots and medicine, and so on. So focusing on those and then and then coming up with the obstacles coming up with the ways that that can go wrong and solving those one at a time.
 And just because you can build an autonomous vehicle, even if you could build one that would drive this finalized, you know, maybe there are some things in life that we would actually want to do ourselves.
 That's right.
 Right. Like, for example, if you think of our society as a whole, there's some things that we find very meaningful to do.
 And that doesn't mean we have to stop doing them just because machines can do them better. You know, I'm not going to stop playing tennis, just the day someone builds a tennis robot.
 Yeah, beat me.
 People are still playing chess and even go.
 Yeah. And I in this in the very near term, even some people are advocating basic income, replace jobs. But if you if the government is going to be willing to just hand out cash to people for doing nothing, then once you also seriously consider whether the government should also hire a lot more teachers and nurses and the kind of jobs which people often find great fulfillment in doing right.
 And we're tired of hearing politicians saying, oh, we can't afford hiring more teachers, but we're going to maybe have basic income. If we can have more more serious research and thought into what gives meaning to our lives and the jobs give so much more than income.
 Right.
 And then think about in the future.
 What are the roles that
 we want to have people continually filling empowered by machines. And I think sort of, I come from the Russia from the Soviet Union and I think for a lot of people in the 20th century going to the moon, going to space was an inspiring thing.
 I feel like the, the, the, the universe of the mind so AI understanding creating intelligence is that for the 21st century. So it's really surprising and I've heard you mention this it's really surprising to me, both on the research funding side that it's not funded as greatly as it could be.
 But most importantly, on the politician side, that it's not part of the public discourse except in the killer bots terminator kind of view that people are not yet, I think, perhaps excited by the possible positive future that we can build together so we should be because politicians
 usually just focus on the next election cycle right.
 And the single most important thing I feel we humans have learned and the entire history of science is there were the masters of underestimation we underestimated the size of our cosmos, again and again, realizing that everything we thought existed was just a small part of something
 grander planet solar system the galaxy clusters of guys in the universe.
 And now know that we have the future has just so much more potential than our ancestors could ever have dreamt of this cosmos.
 Imagine if all of Earth was completely devoid of life, except for Cambridge Massachusetts, I would wouldn't it be kind of lame. If all we ever aspired to is to stay in Cambridge Massachusetts forever.
 And go extinct in one week, even though Earth was going to continue on for longer that that that sort of attitude I think we have now on the cosmic scale, we can full life can flourish on Earth, not for four years but for billions of years, yes, I can even tell you
 I can move it out of harm's way when the sun gets too hot. And, and then we have so much more resources out here, which today, yeah maybe there are a lot of other planets with bacteria or cow like life on them but I most of this, all this opportunity seems as far as we can
 largely dead, like the Sahara Desert and yet we have the opportunity to help life flourish around this billions of years so like, let's quit squabbling about whether some little border should be drawn one one mile to the left or right, and look up into the skies
 Hey, you know, we can do such incredible things. Yeah, and that's I think why it's really exciting that yeah, you and others are connected with some of the work Elon Musk is doing, because he's literally going out into that space really exploring our universe
 and it's wonderful. That is exactly why Elon Musk is so misunderstood right. Miss can screw him as some kind of pessimistic doomsayer, the reason he cares so much about the safety is because he more than almost anyone else appreciates these amazing opportunities
 they will squander it if we wipe out here on earth, and we're not just going to wipe out the next generation, all generations and this incredible opportunity that's out there and that would be really be a waste and AI.
 For people who think that that would be better to do without technology.
 Let me just mention that if we don't improve our technology the question isn't whether humanity is going to go extinct.
 It's just whether we're going to get taken out by the next big asteroid or the next super volcano or something else dumb that we could easily prevent with more tech right.
 And if we want life to flourish throughout the cosmos, AI is the key to it.
 As I mentioned in a lot of detail in my book right there.
 Even many of the most inspired sci fi writers I feel have totally underestimated the opportunities for space travel especially to other galaxies, because they weren't thinking about the possibility of a GI, which just makes it so much easier.
 Yeah, so that goes to your view of a GI that enables our progress that enables a better life. So that's a beautiful and that's a beautiful way to put it, and then something to strive for. So Max, thank you so much. Thank you for your time today. It's been awesome.
 Thank you so much. Thanks.