Max Tegmark: AI and Physics | Lex Fridman Podcast #155
max tegmark artificial intelligence agi ai ai podcast artificial intelligence podcast lex fridman lex podcast lex mit lex ai lex jre mit ai
Max Tegmark is a physicist and AI researcher at MIT. Please support this podcast by checking out our sponsors:
- The Jordan Harbinger Show: https://www.jordanharbinger.com/lex/
- Four Sigmatic: https://foursigmatic.com/lex and use code LexPod to get up to 60% off
- BetterHelp: https://betterhelp.com/lex to get 10% off
- ExpressVPN: https://expressvpn.com/lexpod and use code LexPod to get 3 months free

EPISODE LINKS:
News Project Explainer Video: https://www.youtube.com/watch?v=PRLF17Pb6vo
News Project Website: https://www.improvethenews.org/
Max's Twitter: https://twitter.com/tegmark
Max's Website: https://space.mit.edu/home/tegmark/
Future of Life Institute: https://futureoflife.org/
Lex Fridman Podcast #1: https://www.youtube.com/watch?v=Gi8LUnhP5yU

PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
Full episodes playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
2:49 - AI and physics
16:07 - Can AI discover new laws of physics?
24:57 - AI safety
42:33 - Extinction of human species
53:31 - How to fix fake news and misinformation
1:15:05 - Autonomous weapons
1:30:28 - The man who prevented nuclear war
1:40:36 - Elon Musk and AI
1:54:14 - AI alignment
2:00:16 - Consciousness
2:09:20 - Richard Feynman
2:13:30 - Machine learning and computational physics
2:24:28 - AI and creativity
2:35:42 - Aliens
2:51:25 - Mortality

CONNECT:
- Subscribe to this YouTube channel
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/LexFridmanPage
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman
 The following is a conversation with Max Tegmark, his second time on the podcast. In fact, the previous conversation was episode number one of this very podcast. He is a physicist and artificial intelligence researcher at MIT, co-founder of the Future of Life Institute, and author of Life 3.0, Being Human in the Age of Artificial Intelligence. He's also the head of a bunch of other huge, fascinating projects, and has written a lot of different things that you should definitely check out. He has been one of the key humans who has been outspoken about long-term existential risks of AI and also its exciting possibilities and solutions to real-world problems, most recently at the intersection of AI and physics, and also in re-engineering the algorithms that divide us by controlling the information we see, and thereby creating bubbles and all other kinds of complex social phenomena that we see today. In general, he's one of the most passionate and brilliant people I have the fortune of knowing. I hope to talk to him many more times on this podcast in the future. Quick mention of our sponsors, the Jordan Harbinger Show, Four Sigmatic Mushroom Coffee, Better Help Online Therapy, and ExpressVPN. So the choice is wisdom, caffeine, sanity, or privacy. Choose wisely, my friends, and if you wish, click the sponsor links below to get a discount to support this podcast. As a side note, let me say that much of the researchers in the machine learning and artificial intelligence communities do not spend much time thinking deeply about existential risks of AI. Because our current algorithms are seen as useful but dumb, it's difficult to imagine how they may become destructive to the fabric of human civilization in the foreseeable future. I understand this mindset, but it's very troublesome. To me, this is both a dangerous and uninspiring perspective, reminiscent of a lobster sitting in a pot of lukewarm water that a minute ago was cold. I feel a kinship with this lobster. I believe that already the algorithms that drive our interaction on social media have an intelligence and power that far outstripped the intelligence and power of any one human being. Now really is the time to think about this, to define the trajectory of the interplay of technology and human beings in our society. I think that the future of human civilization very well may be at stake over this very question of the role of artificial intelligence in our society. If you enjoy this thing, subscribe on YouTube, review it on Apple Podcasts, follow on Spotify, support on Patreon, or connect with me on Twitter at Lex Friedman. And now here's my conversation with Max Tagmark. So people might not know this, but you were actually episode number one of this podcast just a couple of years ago, and now we're back. And it so happens that a lot of exciting things happened in both physics and artificial intelligence, both fields that you're super passionate about. Can we try to catch up to some of the exciting things happening in artificial intelligence, especially in the context of the way it's cracking open the different problems of the sciences? Yeah, I'd love to, especially now as we start 2021 here, it's a really fun time to think about what were the biggest breakthroughs in AI, not the ones necessarily that media wrote about, but that really matter. And what does that mean for our ability to do better science? What does it mean for our ability to help people around the world? And what does it mean for new problems that they could cause if we're not smart enough to avoid them? So, what did we learn basically from this? Yes, absolutely. So one of the amazing things you're part of is the AI Institute for Artificial Intelligence and Fundamental Interactions. What's up with this institute? What are you working on? What are you thinking about? The idea is something I'm very on fire with, which is basically AI meets physics. And it's been almost five years now since I shifted my own MIT research from physics to machine learning. And in the beginning, I noticed a lot of my colleagues, even though they were polite about it, well, I kind of, what is Max doing? What is this weird stuff? He's lost his mind. But then gradually, I, together with some colleagues, were able to persuade more and more of the other professors in our physics department to get interested in this. And now we got this amazing NSF center, so 20 million bucks for the next five years, MIT and a bunch of neighboring universities here also. And I noticed now those colleagues who were looking at me funny have stopped asking what the point is of this, because it's becoming more clear. And I really believe that, of course, AI can help physics a lot to do better physics, but physics can also help AI a lot, both by building better hardware. My colleague, Marin Soljadzic, for example, is working on an optical chip for much faster machine learning where the computation is done not by moving electrons around but by moving photons around, dramatically less energy use, faster, better. We can also help AI a lot, I think, by having a different set of tools and a different, maybe more audacious attitude. AI has, to a significant extent, been an engineering discipline where you're just trying to make things that work and being more interested in maybe selling them than in figuring out exactly how they work and proving theorems about that they will always work. Contrast that with physics. When Elon Musk sends a rocket to the International Space Station, they didn't just train with machine learning, oh, let's fire it a little bit left, more to the left, a bit more to the right, or that also missed, let's try here. No, we figured out Newton's laws of gravitation and other things and got a really deep fundamental understanding. And that's what gives us such confidence in rockets. And my vision is that in the future, all machine learning systems that actually have impact on people's lives will be understood at a really, really deep level, right? So we trust them, not because some sales rep told us to, but because they've earned our trust. We can, and really safety-critical things even prove that they will always do what we expect them to do. That's very much the physics mindset. So it's interesting, if you look at big, breakthroughs that have happened in machine learning this year, from dancing robots, it's pretty fantastic. And not just because it's cool, but if you just think about not that many years ago, this YouTube video at this DARPA challenge where the MIT robot comes out of the car and face plants, how far we've come in just a few years. Similarly, Alpha Fold 2, crushing the protein folding problem. We can talk more about implications for medical research and stuff, but hey, that's huge progress. You can look at GPT-3 that can spout off English texts, which sometimes really, really blows you away. You can look at DeepMind's Mu-0, which doesn't just kick our butt and go, and chess, and shogi, but also in all these Atari games. And you don't even have to teach it the rules now. What all of those have in common is, besides being powerful, is we don't fully understand how they work. And that's fine if it's just some dancing robots, and the worst thing that can happen is they face plant. Or if they're playing Go, and the worst thing that can happen is that they make a bad move and lose the game. It's less fine if they're playing and it's less fine if that's what's controlling your self-driving car or your nuclear power plant. And we've seen already that even though Hollywood had all these movies where they try to make us worry about the wrong things, like machines turning evil, the actual bad things that have happened with automation have not been machines turning evil. They've been caused by over-trust in things we didn't understand as well as we thought we did. Even very simple automated systems like what Boeing put into the 737 MAX, killed a lot of people. Was it that that little simple system was evil? Of course not, but we didn't understand it as well as we should have. And we trusted without understanding. Exactly. That's the over-trust. We didn't even understand that we didn't understand. The humility is really at the core of being a scientist. I think step one, if you wanna be a scientist, is don't ever fool yourself into thinking you understand things when you actually don't. That's probably good advice for humans in general. I think humility in general can do us good, but in science, it's so spectacular. Why did we have the wrong theory of gravity ever from Aristotle onward until like Elio's time? Why would we believe something so dumb as that if I throw this water bottle, it's gonna go up with constant speed until it realizes that its natural motion is down? Changes its mind. Because people just kind of assumed Aristotle was right, he's an authority, we understand that. Why did we believe things like that the sun is going around the earth? Why did we believe that time flows at the same rate for everyone until Einstein? Same exact mistake over and over again. We just weren't humble enough to acknowledge that we actually didn't know for sure. We assumed we knew, so we didn't discover the truth because we assumed there was nothing there to be discovered. There was something to be discovered about the 737 max. And if you had been a bit more suspicious and tested it better, we would have found it. And it's the same thing with most harm that's been done by automation so far, I would say. So I don't know if you, did you hear of a company called Knight Capital? No. So good, that means you didn't invest in them earlier. They deployed this automated rating system. Yes. All nice and shiny. They didn't understand it as well as they thought. And it went about losing 10 million bucks per minute for 44 minutes straight. No. Until someone presumably was like, oh no, shut this up. Was it evil? No, it was again, misplaced trust, something they didn't fully understand. And there have been so many, even when people have been killed by robots, it's just quite rare still, but in factory accidents, it's in every single case been not malice, just that the robot didn't understand that a human is different from an auto part or whatever. And we, so this is where I think there's so much opportunity for a physics approach, where you just aim for a higher level of understanding. And if you look at the, all these systems that we talked about from reinforcement learning systems and dancing robots to all these neural networks, that power GPT-3 and go playing software. They're all basically black boxes, much like not so different from if you teach a human something, you have no idea how their brain works, right? Except the human brain at least has been error corrected during many, many centuries of evolution in a way that these, some of these systems have not, right? And my MIT research is entirely focused on demystifying this black box, intelligible intelligence is my slogan. That's a good line, intelligible intelligence. Yeah, that we shouldn't settle for something that seems intelligent, but it should be intelligible so that we actually trust it because we understand it, right? Like again, Elon trusts his rockets because he understands Newton's laws and thrust and everything works. And let me tell you what, can I tell you why I'm optimistic about this? Yes. I think there is, we've made a bit of a mistake where we, some people still think that somehow we're never gonna understand neural networks and we're just gonna have to learn to live with this. It's this very powerful black box. Basically for those who haven't spent time building their own, it's super simple what happens inside. You send in a long list of numbers and then you do a bunch of operations on them, multiply by matrices, et cetera, et cetera, and some other numbers come out, that's the output of it. And then there are a bunch of knobs you can tune and when you change them, it affects the computation, the input output relation. And then you just give the computer some definition of good and it keeps optimizing these knobs until it performs as good as possible. And often you go like, wow, that's really good. This robot can dance or this machine is beating me at chess now. And then the end you have something which even though you can look inside it, you have very little idea of how it works. You can print out tables of all the millions of parameters in there. Is it crystal clear now how it's working? And of course not, right? Many of my colleagues seem willing to settle for that. And I'm like, no, that's like the halfway point. Some have even gone as far as sort of guessing that the mist or the instruitability of this is where some of the power comes from and sort of some sort of mysticism. I think that's total nonsense. I think the real power of neural networks comes not from instruitability, but from differentiability. And what I mean by that is simply that the output depends, changes only smoothly if you tweak your knobs, and then you can use all these powerful methods we have for optimization in science. We can just tweak them a little bit and see did that get better or worse? That's the fundamental idea of machine learning, that the machine itself can keep optimizing until it gets better. Suppose you wrote this algorithm instead in Python or some other programming language, and then what the knobs did was they just changed random letters in your code. Now we're just epically fail. You change one thing and instead of saying print, it says Psynt, syntax error. You don't even know, was that for the better or for the worse? This to me is, this is what I believe is the fundamental power of neural networks. And just to clarify, the changing of the different letters in a program would not be a differentiable process. It would make it an invalid program typically, and then you wouldn't even know if you changed more letters if it would make it work again. So that's the magic of neural networks, the inscrutability. The differentiability, that every setting of the parameters is a program and you can tell is it better or worse. So you don't like the poetry of the mystery of neural networks as the source of its power? I generally like poetry, but- Not in this case. It's so misleading. And above all, it shortchanges us. It makes us underestimate the good things we can accomplish. So what we've been doing in my group is basically step one, train the mysterious neural network to do something well. And then step two, do some additional AI techniques to see if we can now transform this black box into something equally intelligent that you can actually understand. So for example, I'll give you one example, this AI Feynman project that we just published. So we took the 100 most famous or complicated equations from one of my favorite physics textbooks. In fact, the one that got me into physics in the first place, the Feynman lectures on physics. And so you have a formula, maybe it has what goes into the formula as six different variables, and then what comes out as one. So then you can make like a giant Excel spreadsheet with seven columns. You put in just random numbers for the six columns for those six input variables, and then you calculate with a formula, the seventh column, the output. So maybe it's like the force equals in the last column, some function of the other. And now the task is, okay, if I don't tell you what the formula was, can you figure that out from looking at my spreadsheet I gave you? Yes. This problem is called symbolic regression. If I tell you that the formula is what we call a linear formula. So it's just that the output is some sum of all the things, input, the times, some constants. That's the famous easy problem we can solve. We do it all the time in science and engineering. But the general one, if it's more complicated functions with logarithms or cosines or other math, it's a very, very hard one and probably impossible to do fast in general, just because the number of formulas with N symbols just grows exponentially. Just like the number of passwords you can make grow dramatically with length. So we had this idea that if you first have a neural network that can actually approximate the formula, you just trained it, even if you don't understand how it works, that can be a first step towards actually understanding how it works. So that's what we do first. And then we study that neural network now and put in all sorts of other data that wasn't in the original training data and use that to discover simplifying properties of the formula. And that lets us break it apart often into many simpler pieces and a kind of divide and conquer approach. So we were able to solve all of those 100 formulas, discover them automatically, plus a whole bunch of other ones. And it's actually kind of humbling to see that this code, which anyone who wants now who's listening to this can type pip install AI Feynman on their computer and run it. It can actually do what Johannes Kepler spent four years doing when he stared at Mars data until he's like, finally, Eureka, this is an ellipse. This will do it automatically for you in one hour. Or Max Planck, he was looking at how much radiation comes out at different wavelengths from a hot object and discovered the famous black body formula. This discovers it automatically. I'm actually excited about seeing if we can discover not just old formulas again, but new formulas that no one has seen before. I do like this process of using kind of a neural network to find some basic insights and then dissecting the neural network to then gain the final. So that's in that way you've forcing the explainability issue, really trying to analyze in your own network for the things it knows in order to come up with the final, beautiful, simple theory underlying the whole, the initial system that you were looking at. I love that. And the reason I'm so optimistic that it can be generalized so much more is because that's exactly what we do as human scientists. Think of Galileo whom we mentioned, right? I bet when he was a little kid, if his dad threw him an apple, he would catch it. Why? Because he had a neural network in his brain that he had trained to predict the parabolic orbit of apples that are thrown under gravity. If you throw a tennis ball to a dog, it also has this same ability of deep learning to figure out how the ball is gonna move and catch it. But Galileo went one step further when he got older. He went back and was like, wait a minute. I can write down a formula for this. Y equals X squared, a parabola. And he helped revolutionize physics as we know it, right? So there was a basic neural network in there from childhood that captured the experiences of observing different kinds of trajectories. And then he was able to go back in with another extra little neural network and analyze all those experiences and be like, wait a minute, there's a deeper rule here. Exactly, he was able to distill out in symbolic form what that complicated black box neural network was doing. Not only did the formula he got ultimately become more accurate, and similarly this is how Newton got Newton's laws, which is why Elon can send rockets to the space station now. So it's not only more accurate, but it's also simpler, much simpler, and it's so simple that we can actually describe it to our friends and each other, right? We've talked about it just in the context of physics now, but hey, isn't this what we're doing when we're talking to each other also? We go around with our neural networks, just like dogs and cats and chipmunks and blue jays, and we experience things in the world. But then we humans do this additional step on top of that where we then distill out certain high-level knowledge that we've extracted from this in a way that we can communicate it to each other in a symbolic form, in English in this case, right? So if we can do it, and we believe that we are information processing entities, then we should be able to make machine learning that does it also. Well, do you think the entire thing could be learning, because this dissection process for AI Feynman, the secondary stage feels like something like reasoning, and the initial step feels like more like the more basic kind of differentiable learning. Do you think the whole thing could be differentiable learning? Do you think the whole thing could be basically neural networks on top of each other? It's like turtles all the way down. Can it be neural networks all the way down? I mean, that's a really interesting question. We know that in your case, it is neural networks all the way down, because that's all you have in your skull as a bunch of neurons. Doing their thing, right? But if you ask the question more generally, what algorithms are being used in your brain? I think super interesting to compare. I think we've gotten a little bit backwards historically, because we humans first discovered good old fashioned AI, the logic based AI that we often call GoFi for good old fashioned AI. And then more recently, we did machine learning, because it required bigger computers. So we had to discover it later. So we think of machine learning with neural networks as the modern thing and the logic based AI as the old fashioned thing. But if you look at evolution on earth, right? It's actually been the other way around. I would say that, for example, an eagle has a better vision system that I have using, and dogs are just as good at casting tennis balls as I am. All this stuff which is done by training in neural network and not interpreting it in words, is something so many of our animal friends can do, at least as well as us, right? What is it that we humans can do that the chipmunks and the eagles cannot? It's more to do with this logic based stuff, right? Where we can extract out information in symbols, in language, and now even with equations, if you're a scientist, right? So basically what happened was, first we built these computers that could multiply numbers real fast and manipulate symbols, and we felt they were pretty dumb. And then we made neural networks that can see as well as a cat can and do a lot of this inscrutable black box neural networks. What we humans can do also is put the two together in a useful way. Yes, in our own brain. Yes, in our own brain. So if we ever wanna get artificial general intelligence that can do all jobs as well as humans can, right? Then that's what's gonna be required, to be able to combine the neural networks with symbolic, combine the old AI with the new AI in a good way. We do it in our brains, and there seems to be basically two strategies I see in industry now. One scares the heebie-jeebies out of me, and the other one I find much more encouraging. Okay, which one? Can we break them apart? Which of the two? The one that scares the heebie-jeebies out of me is this attitude that we're just gonna make ever bigger systems that we still don't understand until they can be as smart as humans. What could possibly go wrong? I think it's just such a reckless thing to do. And unfortunately, and if we actually succeed as a species to build artificial general intelligence then we still have no clue how it works. I think at least 50% chance we're gonna be extinct before too long. It's just gonna be an utter epic own goal. So it's that 44 minute losing money problem or the paperclip problem like where we don't understand how it works and it just in a matter of seconds runs away in some kind of direction that's going to be very problematic. Even long before you have to worry about the machines themselves somehow deciding to do things. And to us, we have to worry about people using machines. Machines that are short of AI, AGI and power to do bad things. I mean, just take a moment. And if anyone who's not worried particularly about advanced AI, just take 10 seconds and just think about your least favorite leader on the planet right now. Don't tell me who it is. I wanna keep this apolitical, but just see the face in front of you, that person for 10 seconds. Now imagine that that person has this incredibly powerful AI under their control and can use it to impose their will on the whole planet. How does that make you feel? Yeah. So can we break that apart just briefly for the 50% chance that we'll run to trouble with this approach. Do you see the bigger worry in that leader or humans using the system to do damage or are you more worried? And I think I'm in this camp, more worried about like accidental, unintentional destruction of everything. So like humans trying to do good and like in a way where everyone agrees it's kind of good, it's just that they're trying to do good without understanding. Cause I think every evil leader in history thought they're to some degree thought they're trying to do good. Oh yeah, I'm sure Hitler thought he was doing good. He was trying to do good too. I've been reading a lot about Stalin. I'm sure Stalin is from, he legitimately thought that communism was good for the world and that he was doing good. I think Mao Zedong thought what he was doing with a great leap forward was good too. Yeah. I'm actually concerned about both of those. Before I promised to answer this in detail, but before we do that, let me finish answering the first question. Cause I told you that there were two different groups we could get to artificial general intelligence and one scares the GBBs out of me, which is this one where we build something. We just say bigger neural networks, ever more hardware and it's just training the heck out of more data and poof, now it's very powerful. That I think is the most unsafe and reckless approach. The alternative to that is the intelligent, intelligible intelligence approach instead, where we say neural networks is just a tool to like for the first step to get the intuition, but then we're going to spend also serious resources sources on other AI techniques for demystifying this black box and figuring out what it's actually doing. So we can convert it into something that's equally intelligent, but that we actually understand what it's doing. Maybe we can even prove theorems about it. That this car here will never be hacked when it's driving because here's the proof. There is a whole science of this, but it doesn't work for neural networks that are big black boxes, but it works well and work to turn other kinds of codes. That approach I think is much more promising. That's exactly why I'm working on it, frankly, not just because I think it's cool for science, but because I think the more we understand these systems, the better the chances that we can make them do the things that are good for us that are actually intended, not unintended. So you think it's possible to prove things about something as complicated as a neural network? That's the hope? Well, ideally there's no reason there has to be a neural network in the end either. We discovered that Newton's laws of gravity with neural network in Newton's head, but that's not the way it's programmed into the navigation system of Elon Musk's rocket anymore. It's written in C++, or I don't know what language he uses exactly. And then there are software tools called symbolic verification. DARPA and the US military has done a lot of really great research on this because they really want to understand that when they build weapons systems, they don't just go fire at random or malfunction, right? And there's even an whole operating system called Cell 3 that's been developed by a DARPA grant where you can actually mathematically prove that this thing can never be hacked. Wow. One day I hope that will be something you can say about the OS that's running on our laptops too. As you know, we're not there, but I think we should be ambitious, frankly. And if we can use machine learning to help do the proofs and so on as well, right? Then it's much easier to verify that a proof is correct than to come up with a proof in the first place. That's really the core idea here. If someone comes on your podcast and says they proved the Riemann hypothesis or some sensational new theorem, it's much easier for someone else to take some smart math grad students and check, oh, there's an error here in equation five, or this really checks out than it was to discover the proof. Yeah, although some of those proofs are pretty complicated, but yes, it's still nevertheless much easier to verify the proof. I love the optimism. Even with the security of systems, there's a kind of cynicism that pervades people who think about this, which is like, well, it's hopeless. I mean, in the same sense, exactly like you're saying, when you don't know what works, oh, it's hopeless to understand what's happening. With security, people are just like, well, there's always going to be attack vectors and waste to attack the system. But you're right, we're just very new with these computational systems. We're even newer with these intelligent systems. And it's not out of the realm of possible, just like people that understand the movement of the stars and the planets and so on. It's entirely possible that within, hopefully soon, but it could be within 100 years, we start to have an obvious laws of gravity about intelligence and God forbid about consciousness too. That one is. Agreed, I think, of course, if you're selling computers that get hacked a lot, that's in your interest as a company that people think it's impossible to make it safe. So nobody's going to get the idea of suing you. But I want to really inject optimism here. It's absolutely possible to do much better than we're doing now. And your laptop does so much stuff. You don't need the music player to be super safe in your future self-driving car. If someone hacks it and starts playing music you don't like, the world won't end. But what you can do is you can break out and say the drive computer that controls your safety must be completely physically decoupled entirely from the entertainment system. And it must physically be such that it can't take on over the air updates while you're driving. And it can be, it can have, it's not that, it can have ultimately some operating system on it which is symbolically verified and proven that it's always going to do what it's supposed to do. We can basically have, and companies should take that attitude too. They should look at everything they do and say, what are the few systems in our company that threaten the whole life of the company if they get hacked, you know, and have the highest standards for them. And then they can save money by going for the El Cheapo poorly understood stuff for the rest, you know. This is very feasible, I think. And coming back to the bigger question that you worried about that there'll be unintentional failures, I think there are two quite separate risks here. We talked a lot about one of them, which is that the goals are noble of the human. The human says, I want this airplane to not crash because this is not Mohammed Atta now flying the airplane. And now there's this technical challenge of making sure that the autopilot is actually going to behave as the pilot wants. If we set that aside, there's also the separate question. How do you make sure that the goals of the pilot are actually aligned with the goals of the passenger? How do you make sure very much more broadly that if we can all agree as a species that we would like things to kind of go well for humanity as a whole, that the goals are aligned here, the alignment problem. And yeah, there's been a lot of progress in the sense that there's suddenly huge amounts of research going on on it about it. I'm very grateful to Elon Musk for giving us that money five years ago so we could launch the first research program on technical AI safety and alignment. There's a lot of stuff happening. But I think we need to do more than just make sure little machines do always what their owners do. That wouldn't have prevented September 11 if Mahmoud Abbas said, okay, autopilot, please fly into World Trade Center. And it's like, okay. That even happened in a different situation. There was this depressed pilot named Andreas Lubitz who told his German wings passenger jet to fly into the Alps. He just told the computer to change the altitude to a hundred meters or something like that. And you know what the computer said? Okay. And it had the frigging topographical map of the Alps in there, it had GPS, everything. No one had bothered teaching it even the basic kindergarten ethics of like, no, we never want airplanes to fly into mountains under any circumstances. And so we have to think beyond just the technical issues and think about how do we align in general incentives on this planet for the greater good? So starting with simple stuff like that, every airplane that has a computer in it should be taught whatever kindergarten ethics that's smart enough to understand. Like, no, don't fly into fixed objects if the pilot tells you to do so, then go on autopilot mode, send an email to the cops and land at the latest airport, nearest airport. Any car with a forward facing camera should just be programmed by the manufacturer so that it will never accelerate into a human ever. That would avoid things like the Nice attack and many horrible terrorist vehicle attacks where they deliberately did that, right? This was not some sort of thing, oh, you know, US and China, different views on, no, there was not a single car manufacturer in the world who wanted the cars to do this. They just hadn't thought to do the alignment. And if you look at more broadly problems that happen on this planet, the vast majority have to do with poor alignment. I mean, think about, let's go back really big because I know you're so good at that. Yeah, in the very, so long ago in evolution, we had these genes and they wanted to make copies of themselves, that's really all they cared about. So some genes said, hey, I'm gonna build a brain on this body I'm in so that I can get better at making copies of myself. And then they decided for their benefit to get copied more, align your brain's incentives with their incentives. So it didn't want you to starve to death. So it gave you an incentive to eat and it wanted you to make copies of the genes. So it gave you incentive to fall in love and do all sorts of naughty things to make copies of itself, right? So that was successful value alignment done on the genes. They created something more intelligent than themselves but they made sure to try to align the values. But then something went a little bit against the idea of what the genes wanted because a lot of humans discovered, hey, you know, yeah, we really like this business about sex that the genes have made us enjoy but we don't wanna have babies right now. So we're gonna hack the genes and use birth control. And I really feel like drinking a Coca-Cola right now but I don't wanna get a pot belly so I'm gonna drink Diet Coke. We have all these things we've figured out because we're smarter than the genes how we can actually subvert their intentions. So it's not surprising that we humans now when we're in the role of these genes creating other non-human entities with a lot of power have to face the same exact challenge. How do we make other powerful entities have incentives that are aligned with ours and that so they won't hack them? Corporations, for example, right? We humans decided to create corporations because it can benefit us greatly. Now all of a sudden there's a supermarket. I can go buy food there. I don't have to hunt. Awesome. And then to make sure that this corporation would do things that were good for us and not bad for us, we created institutions to keep them in check. Like if the local supermarket sells poisonous food, then the owners of a supermarket have to spend some years reflecting behind bars, right? So we created incentives to align them. But of course, just like we were able to see through this thing, birth control, if you're a powerful corporation, you also have an incentive to try to hack the institutions that are supposed to govern you because you ultimately as a corporation have an incentive to maximize your profits. It's like you have an incentive to maximize the enjoyment your brain has, not for your genes. So if they can figure out a way of bribing regulators, then they're gonna do that. In the US, we kind of caught onto that and made laws against corruption and bribery. Then in the late 1800s, Teddy Roosevelt realized that, no, we were still being kind of hacked because the Massachusetts railroad companies had like a bigger budget than the state of Massachusetts and they were doing a lot of very corrupt stuff. So he did the whole trust busting thing to try to align these other non-human entities, the companies again, more with the incentives of Americans as a whole. It's not surprising though that this is a battle you have to keep fighting. Now we have even larger companies than we ever had before. And of course, they're gonna try to again, subvert the institutions, not because, I think people make a mistake of getting all too black thinking about things in terms of good and evil, like arguing about whether corporations are good or evil or whether robots are good or evil. A robot isn't good or evil, it's tool. And you can use it for great things like robotic surgery or for bad things. And a corporation also is a tool of course. And if you have good incentives to the corporation, it'll do great things like start a hospital or a grocery store. If you have really bad incentives, then it's gonna start maybe marketing addictive drugs to people and you'll have an opioid epidemic, right? It's all about, I don't want, we should not make the mistake of getting into some sort of fairy tale, good evil thing about corporations or robots. We should focus on putting the right incentives in place. My optimistic vision is that if we can do that, then we can really get good things. We're not doing so great with that right now, either on AI, I think, or on other intelligent non-human entities like big companies, right? We just have a new secretary of defense who's gonna start up now in the Biden administration who is, was an active member of the board of Raytheon, for example. So, I have nothing against Raytheon. I'm not a pacifist, but there's an obvious conflict of interest if someone is in the job where they decide who's they're gonna contract with. And I think somehow we have, maybe we need another Teddy Roosevelt to come along again and say, hey, we want what's good for all Americans and we need to go do some serious realigning again of the incentives that we're giving to these big companies. And then we're gonna be better off. It seems that naturally with human beings, just like you beautifully described the history of this whole thing, it all started with the genes and they're probably pretty upset by all the unintended consequences that happened since. But it seems that it kind of works out. Like it's in this collective intelligence that emerges at the different levels, it seems to find sometimes last minute a way to realign the values or keep the values aligned. It's almost, it finds a way, like different leaders, different humans pop up all over the place that reset the system. Do you want, I mean, do you have an explanation why that is or is that just survivor bias? And also is that different, somehow fundamentally different than with AI systems where you're no longer dealing with something that was a direct, maybe companies are the same, a direct byproduct of the evolutionary process. I think there is one thing which has changed. That's why I'm not all optimistic. That's why I think there's about a 50% chance if we take the dumb route with artificial intelligence that we will, humanity will be extinct in this century. First, just the big picture, yeah, companies need to have right incentives, even governments, right? We used to have governments, usually there were just some king, who was the king because his dad was the king. And then there were some benefits of having this powerful kingdom or empire of any sort, because then it could prevent a lot of local squabbles. So at least everybody in that region would stop warring against each other and their incentives of different cities in the kingdom became more aligned, right? That was the whole selling point. Harari has a beautiful piece on how empires were collaboration enablers. And then we also, Harari says invented money for that reason so we could have better alignment and we could do trade even with people we didn't know. So this sort of stuff has been playing out since time immemorial, right? What's changed is that it happens on ever larger scales, right, the technology keeps getting better because science gets better. So now we can communicate over larger distances, transport things faster over larger distances. And so the entities get ever bigger, but our planet is not getting bigger anymore. So in the past, you could have one experiment that just totally screwed up like Easter Island, where they actually managed to have such poor alignment that when they went extinct, people there, there was no one else to come back and replace them, right? If Elon Musk doesn't get us to Mars and then we go extinct on a global scale and then we're not coming back, that's the fundamental difference. And that's a mistake we don't make for that reason. In the past, of course, history is full of fiascos, but it was never the whole planet. And then, okay, now there's this nice uninhabited land here some other people could move in and organize things better. This is different. The second thing which is also different is that technology gives us so much more empowerment, both to do good things and also to screw up. In the Stone Age, even if you had someone whose goals were really poorly aligned, maybe he was really pissed off because his Stone Age girlfriend dumped him and he just wanted to, if he wanted to kill as many people as he could, how many could he really take out with a rock and a stick before he was overpowered, right? Just handful, right? Now, with today's technology, if we have an accidental nuclear war between Russia and the US, which we almost have about a dozen times, and then we have a nuclear winter, it could take out 7 billion people or 6 billion people or we don't know. So the scale of the damage is bigger that we can do. And if, there's obviously no law of physics that says that technology will never get powerful enough that we could wipe out our species entirely, that would just be fantasy to think that science is somehow doomed to not get more powerful than that, right? And it's not at all unfeasible in our lifetime that someone could design a designer pandemic which spreads as easily as COVID but just basically kills everybody. We already had smallpox. It killed one third of everybody who got it. What do you think of the, here's an intuition, maybe it's completely naive and this optimistic intuition I have, which it seems, and maybe it's a biased experience that I have, but it seems like the most brilliant people I've met in my life all are really like fundamentally good human beings. And not like naive, good, like they really wanna do good for the world in a way that, well, maybe is aligned to my sense of what good means. And so I have a sense that the people that will be defining the very cutting edge of technology, there'll be much more of the ones that are doing good versus the ones that are doing evil. So the race, I'm optimistic on the us always like last minute coming up with a solution. So if there's an engineered pandemic that has the capability to destroy most of the human civilization, it feels like to me, either leading up to that before or as it's going on, there will be, we're able to rally the collective genius of the human species. I can tell by your smile that you're at least some percentage doubtful. But could that be a fundamental law of human nature? That evolution only creates, like karma is beneficial, good is beneficial and therefore we'll be all right. I hope you're right. I really would love it if you're right, if there's some sort of law of nature that says that we always get lucky in the last second because of karma. But I prefer not playing it so close and gambling on that. And I think, in fact, I think it can be dangerous to have too strong faith in that because it makes us complacent. Like if someone tells you, you never have to worry about your house burning down, then you're not gonna put in a smoke detector because why would you need to? Even if it's sometimes very simple precautions, we don't take them. If you're like, oh, the government is gonna take care of everything for us, I can always trust my politicians. We have to get our own responsibility. I think it's a healthier attitude to say, yeah, maybe things will work out, maybe I'm actually gonna have to myself step up and take responsibility and the stakes are so huge. I mean, if we do this right, we can develop all this ever more powerful technology and cure all diseases and create a future where humanity is healthy and wealthy for not just the next election cycle, but like billions of years throughout our universe. That's really worth working hard for and not just sitting and hoping for some sort of fairy tale karma. Well, I just mean, so you're absolutely right. From the perspective of the individual, like for me, the primary thing should be to take responsibility and to build the solutions that your skillset allows. Yeah, which is a lot. I think we underestimate often very much how much good we can do. If you or anyone listening to this is completely confident that our government would do a perfect job on handling any future crisis with engineered pandemics or future AI, I ask you to reflect a bit on what actually happened in 2020. Do you feel that the government by and large around the world has handled this flawlessly? That's a really sad and disappointing reality that hopefully is a wake up call for everybody, for the scientists, for the engineers, for the researchers in AI especially. It was disappointing to see how inefficient we were at collecting the right amount of data in a privacy preserving way and spreading that data and utilizing that data to make decisions, all that kind of stuff. Yeah, I think when something bad happens to me, I made myself a promise many years ago that I would not be a whiner. So when something bad happens to me, of course it's a process to disappointment, but then I try to focus on what did I learn from this that can make me a better person in the future. And there's usually something to be learned when I fail. And I think we should all ask ourselves, what can we learn from the pandemic about how we can do better in the future? And you mentioned there's a really good lesson. We were not as resilient as we thought we were and we were not as prepared maybe as we wish we were. You can even see very stark contrast around the planet. South Korea, they have over 50 million people. Do you know how many deaths they have from COVID last time I checked? No. About 500. Why is that? Well, the short answer is that they had prepared. They were incredibly quick, incredibly quick to get on it with very rapid testing and contact tracing and so on, which is why they never had more cases than they could contract trace effectively. They never even had to have the kind of big lockdowns we had in the West. But the deeper answer to it, it's not just the Koreans are just somehow better people. The reason I think they were better prepared was because they had already had a pretty bad hit from the SARS pandemic, which never became a pandemic, something like 17 years ago, I think. So it was kind of a fresh memory that, we need to be prepared for pandemics. So they were, right? And so maybe this is a lesson here for all of us to draw from COVID that rather than just wait for the next pandemic or the next problem with AI getting out of control or anything else, maybe we should just actually set aside a tiny fraction of our GDP to have people very systematically do some horizon scanning and say, okay, what are the things that could go wrong? And let's do get out and see which are the more likely ones and which are the ones that are actually actionable and then be prepared. So one of the observations as one little ant slash human that I am of disappointment is the political division over information that has been observed, that I observed this year, that it seemed the discussion was less about sort of, what happened and understanding what happened deeply and more about there's different truths out there. And it's like an argument, my truth is better than your truth. And it's like red versus blue or different, it was like this ridiculous discourse that doesn't seem to get at any kind of notion of the truth. It's not like some kind of scientific process, even science got politicized in ways that's very heartbreaking to me. You have an exciting project on the AI front of trying to rethink one of the, you mentioned corporations, there's one of the other collective intelligence systems that have emerged through all of this is social networks. And just the spread, the internet is the spread of information on the internet, our ability to share that information, there's all different kinds of news sources and so on. And so you said like, that's from first principles, let's rethink how we think about the news, how we think about information. Can you talk about this amazing effort that you're undertaking? Oh, I'd love to, this has been my big COVID project since nights and weekends on ever since the lockdown. To segue into this actually, let me come back to what you said earlier, that you had this hope that in your experience, people who you felt were very talented, often idealistic and wanted to do good. Frankly, I feel the same about all people by and large, there are always exceptions, but I think the vast majority of everybody, regardless of education and whatnot, really are fundamentally good, right? So how can it be that people still do so much nasty stuff, right? I think it has everything to do with the information that we're given. If you go into Sweden 500 years ago and you start telling all the farmers that those Danes in Denmark, they're so terrible people, and we have to invade them because they've done all these terrible things that you can't fact check yourself. A lot of people, Swedes did that, right? And we're seeing so much of this today in the world, both geopolitically, where we are told that China is bad and Russia is bad and Venezuela is bad, and people in those countries are often told that we are bad, and we also see it at a micro level, where people are told that, oh, those who voted for the other party are bad people. It's not just an intellectual disagreement, but they're bad people, and we're getting ever more divided. So how do you reconcile this with the fact that this intrinsic goodness in people? I think it's pretty obvious that it has again to do with the information that we're fed and given. We evolved to live in small groups where you might know 30 people in total. So you then had a system that was quite good for assessing who you could trust and who you could not. And if someone told you that Joe there is a jerk, but you had interacted with him yourself and seen him in action, and you would quickly realize maybe that that's actually not quite accurate, right? But now that we, the most people on the planet are people we've never met, it's very important that we have a way of trusting the information we're given. So, okay, so where does the news project come in? Well, throughout history, you can go read Machiavelli from the 1400s, and you'll see how already then they were busy manipulating people with propaganda and stuff. Propaganda is not new at all. And the incentives to manipulate people is just not new at all. What is it that's new? What's new is machine learning meets propaganda. That's what's new. That's why this has gotten so much worse. Some people like to blame certain individuals. Like in my liberal university bubble, many people blame Donald Trump and say it was his fault. I see it differently. I think Donald Trump just had this extreme skill at playing this game in the machine learning algorithm age. A game he couldn't have played 10 years ago. So what's changed? What's changed is, well, Facebook and Google and other companies, and I'm not badmouthing them. I have a lot of friends who work for these companies, good people, they deployed machine learning algorithms just to increase their profit a little bit, to just maximize the time people spent watching ads. And they had totally underestimated how effective they were gonna be. This was again, the black box, non-intelligible intelligence. They just noticed, oh, we're getting more ad revenue. Great, it took a long time until they even realized why and how and how damaging this was for society. Because of course, what the machine learning figured out was that the by far most effective way of gluing you to your little rectangle was to show you things that triggered strong emotions, anger, et cetera, resentment. And if it was true or not, didn't really matter. It was also easier to find stories that weren't true. If you weren't limited, that's just the limitation to show people. That's a very limiting fact. And before and long, we got these amazing filter bubbles on a scale we had never seen before. A couple of this to the fact that also the online news media were so effective that they killed a lot of print journalism. There's less than half as many journalists now in America, I believe, as there was a generation ago. You just couldn't compete with the online advertising. So all of a sudden, most people are not getting, even reading newspapers, they get their news from social media. And most people only get news in their little bubble. So along comes now some people like Donald Trump who figured out among the first successful politicians to figure out how to really play this new game and become very, very influential. But I think Donald Trump was a simple, he took advantage of it. He didn't create, the fundamental conditions were created by machine learning, sort of taking over the news media. So this is what motivated my little COVID project here. So, I said before, machine learning and tech in general, it's not evil, but it's also not good. It's just a tool that you can use for good things or bad things. And as it happens, machine learning and news is mainly used by the big players, big tech, to manipulate people into watch as many ads as possible, which had this unintended consequence of really screwing up our democracy and fragmenting it into filter bubbles. So I thought, well, machine learning algorithms were basically free. They can run on your smartphone for free also if someone gives them away to you, right? There's no reason why they only have to help the big guy to manipulate the little guy. They can just as well help the little guy to see through all the manipulation attempts from the big guy. So did this project, it's called, you can go to improvethenews.org. The first thing we've built is the little news aggregator. Looks a bit like Google News, except it has these sliders on it to help you break out of your filter bubble. So if you're reading, you can click, click, and go to your favorite topic. And then if you just slide the left, right slider all the way over to the left. There's two sliders, right? Yeah, there's the one, the most obvious one is the one that has left, right labeled on it. You go to left, you get one set of articles, you go to the right to see a very different truth appearing. Oh, that's literally left and right on the political spectrum. On the political spectrum. Yeah, so if you're reading about immigration, for example, it's very, very noticeable. And I think step one always, if you wanna not get manipulated, is just to be able to recognize the techniques people use. So it's very helpful to just see how they spin things on the two sides. I think many people are under the misconception that the main problem is fake news. It's not. I had an amazing team of MIT students where we did an academic project, use machine learning to detect the main kinds of bias over the summer. And yes, of course, sometimes there's fake news where someone just claims something that's false, right? Like, oh, Hillary Clinton just got divorced or something. But what we see much more of is actually just omissions. If you go to, there's some stories which just won't be mentioned by the left or the right because it doesn't suit their agenda. And then they'll instead mention other ones very, very, very much. So for example, we've had a number of stories about the Trump family's financial dealings. And then there's been a bunch of stories about the Biden family's, Hunter Biden's financial dealings. Surprise, surprise, they don't get equal coverage on the left and the right. One side loves to cover the Biden, Hunter Biden's stuff. And one side loves to cover the Trump. You can never guess which is which. But the great news is if you're a normal American citizen and you dislike corruption in all its forms, then slide, slide, you can just look at both sides and you'll see all the corrupt, those political corruption stories. It's really liberating to just take in the both sides, the spin on both sides. It somehow unlocks your mind to like think on your own, to realize that, I don't know, it's the same thing that was useful, right, in the Soviet Union times for when everybody was much more aware that they're surrounded by propaganda. It's so interesting what you're saying actually. So Noam Chomsky, used to be our MIT colleague, once said that propaganda is to democracy, what violence is to totalitarianism. And what he means by that is if you have a really totalitarian government, you don't need propaganda. People will do what you want them to do anyway, out of fear, right? But otherwise, you need propaganda. So I would say actually that the propaganda is much higher quality in democracies, much more believable. And it's really striking when I talk to colleagues, science colleagues like from Russia and China and so on, I noticed they are actually much more aware of the propaganda in their own media than many of my American colleagues are about the propaganda in Western media. That's brilliant, that means the propaganda in the Western media is just better. Yes, that's so brilliant. Even the propaganda. Oh man. But once you realize that, you realize there's also something very optimistic there that you can do about it, right? Because first of all, omissions, as long as there's no outright censorship, you can just look at both sides and pretty quickly piece together a much more accurate idea of what's actually going on, right? And develop a natural skepticism too. Yeah. Just an analytical scientific mind about the way you take in the information. Yeah, and I think, I have to say sometimes I feel that some of us in the academic bubble are too arrogant about this and somehow think, oh, it's just people who aren't as educated as us who are cool. When we are often just as gullible also, we read only our media and don't see through things. Anyone who looks at both sides like this in comparison will immediately start noticing the shenanigans being pulled. And I think what I tried to do with this app is that the big tech has to some extent try to blame the individual for being manipulated, much like big tobacco tried to blame the individuals entirely for smoking. And then later on, our government stepped up and said, actually, you can't just blame little kids for starting to smoke. We have to have more responsible advertising and this and that. I think it's a bit the same here. It's very convenient for a big tech to blame. So it's just people who are so dumb and get fooled. The blame usually comes in saying, oh, it's just human psychology. People just wanna hear what they already believe. But Professor David Rand at MIT actually partly debunked that with a really nice study showing that people tend to be interested in hearing things that go against what they believe if it's presented in a respectful way. Suppose, for example, that you have a company and you're just about to launch this project and you're convinced it's gonna work. And someone says, you know, Lex, I hate to tell you this, but this is gonna fail. And here's why. Would you be like, shut up, I don't wanna hear it. La la la la la la la la. Would you? You would be interested, right? And also if you're on an airplane, back in the pre-COVID times, you know, and the guy next to you is clearly from the opposite side of the political spectrum, but is very respectful and polite to you, wouldn't you be kind of interested to hear a bit about how he or she thinks about things? Of course. But it's not so easy to find out respectful disagreement now because like, for example, if you are a Democrat and you're like, I wanna see something on the other side. So you just go brightbart.com. And then after the first 10 seconds, you feel deeply insulted by something and they, it's not gonna work. Or if you take someone who votes Republican and they go to something on the left and they just get very offended very quickly by them having put a deliberately ugly picture of Donald Trump on the front page or something, it doesn't really work. So this news aggregator also has this nuanced slider which you can pull to the right and then to make it easier to get exposed to actually more sort of academic style or more respectful portrayals of different views. And finally, the one kind of bias I think people are mostly aware of is the left right, because it's so obvious, because both left and right are very powerful here. Both of them have well-funded TV stations and newspapers, and it's kind of hard to miss. But there's another one, the establishment slider, which is also really fun. I love to play with it. That's more about corruption. Because if you have a society where almost all the powerful entities want you to believe a certain thing, that's what you're gonna read in both the big media, mainstream media on the left and on the right, of course. And powerful companies can push back very hard, like tobacco companies pushed back very hard back in the day when some newspapers started writing articles about tobacco being dangerous. So it was hard to get a lot of coverage about it initially. And also if you look geopolitically, of course in any country when you read their media, you're mainly gonna be reading a lot of articles about how our country is the good guy and the other countries are the bad guys. So if you wanna have a really more nuanced understanding, like the Germans used to be told that, the British used to be told that the French were the bad guys and the French used to be told that the British were the bad guys. Now they visit each other's countries a lot and have a much more nuanced understanding. I don't think there's gonna be any more wars between France and Germany. But on the geopolitical scale, it's just as much as ever, big Cold War now, US, China, and so on. And if you wanna get a more nuanced understanding of what's happening geopolitically, then it's really fun to look at this establishment slider because it turns out there are tons of little newspapers, both on the left and on the right, who sometimes challenge establishment and say, maybe we shouldn't actually invade Iraq right now. Maybe this weapons of mass destruction thing is BS. If you look at the journalism research afterwards, you can actually see that, so clearly that both CNN and Fox were very pro, let's get rid of Saddam, there are weapons of mass destruction. Then there were a lot of smaller newspapers. They were like, wait a minute, this evidence seems a bit sketchy and maybe we, but of course they were so hard to find. Most people didn't even know they existed, right? Yet it would have been better for American national security if those voices had also come up. I think it harmed America's national security actually that we invaded Iraq. And arguably there's a lot more interest in that kind of thinking too from those small sources. So like the, when you say big, it's more about kind of the reach of the broadcast, but it's not big in terms of the interest. I think there's a lot of interest in that kind of anti-establishment or like skepticism towards, out of the box thinking. There's a lot of interest in that kind of thing. Do you see this news project or something like it being basically taken over the world as the main way we consume information? Like how do we get there? Like how do we, so okay, the idea is brilliant. It's a, you're calling it your little project in 2020, but how does that become the new way we consume information? I hope first of all, just to plant a little seed there because normally the big barrier of doing anything in media is you need a ton of money, but this costs no money at all. I've just been paying myself, pay a tiny amount of money each month to Amazon to run the thing in their cloud. We're not, there will never be any ads. The point is not to make any money off of it. And we just train machine learning algorithms to classify the articles and stuff. So it just kind of runs by itself. So if it actually gets good enough at some point that it starts catching on, it could scale. And if other people carbon copy it and make other versions that are better, that's more the merrier. I think there's a real opportunity for machine learning to empower the individual against the list of the powerful players. It's, as I said in the beginning here, it's been mostly the other way around so far that the big players have the AI and then they tell people this is the truth, this is how it is, but it can just as well go the other way around. Then when the internet was born actually, a lot of people had this hope that maybe this will be a great thing for democracy, make it easier to find out about things. And maybe machine learning and things like this can actually help again. And I have to say, I think it's more important than ever now because this is very linked also to the whole future of life, as we discussed earlier. We're getting this ever more powerful tech. It's frank, it's pretty clear if you look on the one or two generation, three generation time scale that there are only two ways this can end. Geopolitically. Either it ends great for all humanity or it ends terribly for all of us. There's really no in between. And we're so stuck in that, because technology knows no borders. And you can't have people fighting when the weapons just keep getting ever more powerful, indefinitely, eventually luck runs out. And right now we have, I love America, but the fact of the matter is what's good for America is not opposites in the long term to what's good for other countries. It would be if this was some sort of zero sum game, like it was thousands of years ago, when the only way one country could get more resources was to take land from other countries, because that was basically the resource. Look at the map of Europe, some countries kept getting bigger and smaller, endless wars. But then since 1945, there hasn't been any war in Western Europe, and they all got way richer because of tech. So the optimistic outcome is that the big winner in this century is gonna be America and China and Russia and everybody else, because technology just makes us all healthier and wealthier. And we just find some way of keeping the peace on this planet. But I think unfortunately, there are some pretty powerful forces right now that are pushing in exactly the opposite direction and trying to demonize other countries, which makes it more likely that this ever more powerful tech we're building is gonna use in disastrous ways. Yeah, for aggression versus cooperation, that kind of thing. Yeah, even look at just military AI now, right? 20, it was so awesome to see these dancing robots. I loved it, right? But one of the biggest growth areas in robotics now is of course, autonomous weapons. And 2020 was like the best marketing year ever for autonomous weapons, because in both Libya, civil war, and in Nagorno-Karabakh, they made the decisive difference, right? And everybody else is like watching this, oh yeah, we wanna build autonomous weapons too. In Libya, you had on one hand, our ally, the United Arab Emirates, that were flying their autonomous weapons that they bought from China, bombing Libyans. And on the other side, you had our other ally, Turkey, flying their drones. They had no skin in the game, any of these other countries. And of course, it was the Libyans who really got screwed. In Nagorno-Karabakh, you had actually, again, Turkey is sending drones built by this company that was actually founded by a guy who went to MIT AeroAstro, do you know that? Bakr-Ratiyar, yeah? So MIT has a direct responsibility for ultimately this. And a lot of civilians were killed there. And so because it was militarily so effective, now suddenly there's like a huge push. Oh yeah, yeah, let's go build ever more autonomy into these weapons and it's gonna be great. And I think actually people who are obsessed about some sort of future terminators scenario right now should start focusing on the fact that we have two much more urgent threats happening for machine learning. One of them is the whole destruction of democracy that we've talked about now, where our flow of information is being manipulated by machine learning. And the other one is that right now, this is the year when the big arms race in out of control arms race in lethal weapons is gonna start or it's gonna stop. So you have a sense that there is like 2020 was a instrumental catalyst for the race of for the autonomous weapons race. Yeah, because there was the first year when they proved decisive in the battlefield. And these ones are still not fully autonomous, mostly they're remote controlled, right? But we could very quickly make things about the size and cost of a smartphone, which you just put in the GPS coordinates or the face of the one you wanna kill is skin color or whatever, and it flies away and does it. And the real good reason why the US and all the other superpowers should put the kibosh on this is the same reason we decided to put the kibosh on bio weapons. So we gave the future of life award that we can talk more about later, Matthew Messelson from Harvard before for convincing Nixon to ban bio weapons. And I asked him, how did you do it? And he was like, well, I just said, look, we don't want there to be a $500 weapon of mass destruction that all our enemies can afford, even non-state actors. And Nixon was like, good point. It's in America's interest that the powerful weapons are all really expensive. So only we can afford them or maybe some more stable adversaries, right? Nuclear weapons are like that, but bio weapons were not like that. That's why we banned them. And that's why you never hear about them now. That's why we love biology. So you have a sense that it's possible for the big powerhouses in terms of the big names in terms of the big nations in the world to agree that autonomous weapons is not a race we wanna be on. That it doesn't end well. Yeah, because we know it's just gonna end in mass proliferation and every terrorist everywhere is gonna have these super cheap weapons that they will use against us. And our politicians have to constantly worry about being assassinated every time they go outdoors by some anonymous little mini drone. We don't want that. And even if the US and China and everyone else could just agree that you can only build these weapons if they cost at least 10 million bucks. That would be a huge win for the superpowers and frankly for everybody. And people often push back and say, well, it's so hard to prevent cheating. But hey, you can say the same about bio weapons. Take any of your MIT colleagues in biology. Of course they could build some nasty bio weapon if they'd really want to do. But first of all, they don't want to because they think it's disgusting because of the stigma. And second, even if there's some sort of nutcase and want to, it's very likely that some other grad students or someone would rat them out because everyone else thinks it's so disgusting. And in fact, we now know there was even a fair bit of cheating on the bio weapons ban, but no countries use them because it was so stigmatized that it just wasn't worth revealing that they had cheated. You talk about drones, but you kind of think that drones is the remote operation. Which they are mostly still. But you're not taking the next intellectual step of like, where does this go? You're kind of saying the problem with drones is that you're removing yourself from direct violence. Therefore you're not able to sort of maintain the common humanity required to make the proper decision strategically. But that's the criticism as opposed to like, if this is automated, and just exactly as you said, if you automate it and there's a race, then the technology can get better and better and better, which means getting cheaper and cheaper and cheaper. And unlike perhaps nuclear weapons, which is connected to resources in a way, like it's hard to get the, it's hard to engineer. Yeah, it feels like it's, you know, there's too much overlap between the tech industry and autonomous weapons, to where you could have smartphone type of cheapness. If you look at drones, you know, it's a, you know, for a thousand dollars, you can have an incredible system that's able to maintain flight autonomously for you and take pictures and stuff. You could see that going into the autonomous weapons space that's, but like, why is that not thought about or discussed enough in the public do you think? You see those dancing Boston Dynamics robots, and everybody has this kind of, like as if this is like a far future. They have this like fear, like, oh, this will be Terminator in like some, I don't know, unspecified 20, 30, 40 years. And they don't think about, well, this is like some much less dramatic version of that is actually happening now. It's not gonna have, it's not gonna be legged, it's not gonna be dancing, but it's already has the capability to use artificial intelligence to kill humans. Yeah, the Boston Dynamics legged robots, I think the reason we imagine them holding guns is just because you've all seen Arnold Schwarzenegger, right? That's our reference point. That's pretty useless. That's not gonna be the main military use of them. They might be useful in law enforcement in the future. And then there's a whole debate about do you want robots showing up at your house with guns, telling you to, who will be perfectly obedient to whatever dictator controls them? But let's leave that aside for a moment and look at what's actually relevant now. So there's a spectrum of things you can do with AI in the military. And again, to put my card on the table, I'm not the pacifist, I think we should have good defense. So for example, a Predator drone is basically a fancy little remote controlled airplane, right? There's a human piloting it and the decision ultimately about whether to kill somebody with it is made by a human still. And this is a line I think we should never cross. There's a current DOD policy. Again, you have to have a human in the loop. I think algorithms should never make life or death decisions. They should be left to humans. Now, why might we cross that line? Well, first of all, these are expensive, right? So for example, when Azerbaijan had all these drones and Armenia didn't have any, they start trying to jerry-rig little cheap things, fly around, but then of course the Armenians would jam them. The Azeris would jam them. And remote control things can be jammed. That makes them inferior. Also, there's a bit of a time delay between, if we're piloting something from far away, speed of light and the human has a reaction time as well, it would be nice to eliminate that jamming possibility in the time delay by having it fully autonomous. But now you might be crossing that exact line. You might program it to just, oh yeah, dear drone, go hover over this country for a while. And whenever you find someone who is a bad guy, kill them. Now the machine is making these sort of decisions. And some people who defend this still say, well, that's morally fine because we are the good guys and we will tell it the definition of bad guy that we think is moral. But now it would be very naive to think that if ISIS buys that same drone, that they're gonna use our definition of bad guy. Maybe for them, bad guy is someone wearing a US Army uniform. Or maybe there will be some weird ethnic group who decides that someone of another ethnic group, they're the bad guys. The thing is human soldiers, with all our faults, we still have some basic wiring in us. No, it's not okay to kill kids and civilians. And Thomas Weapon has none of that. It's just gonna do whatever is programmed. It's like the perfect Adolf Eichmann on steroids. Like they told him, Adolf Eichmann, we want you to do this and this and this to make the Holocaust more efficient. And he was like, yeah. And off he went and did it, right? Do we really wanna make machines that are like that, like completely amoral and will take the user's definition of who's the bad guy? And do we then wanna make them so cheap that all our adversaries can have them? Like what could possibly go wrong? That's I think the big argument for why we wanna, this year, really put the kibosh on this. And I think you can tell there's a lot of very active debate even going on within the US military and undoubtedly in other militaries around the world also about whether we should have some sort of international agreement to at least require that these weapons have to be above a certain size and cost so that things just don't totally spiral out of control. And finally, just for your question, but is it possible to stop it? Because some people tell me, oh, just give up. But again, so Matthew Messelson again from Harvard, right? Who the bioweapons hero, he had exactly this criticism also with bioweapons. People were like, how can you check for sure that the Russians aren't cheating? And he told me this, I think really ingenious insight. He said, you know, Max, some people think you have to have inspections and things and you have to make sure that people, you can catch the cheaters with a 100% chance. You don't need 100%, he said. 1% is usually enough because if it's another big state, like suppose China and the US have signed the treaty, drawing a certain line and saying, yeah, these kinds of drones are okay, but these fully autonomous ones are not. Now suppose you are China and you have cheated and secretly developed some clandestine little thing, or you're thinking about doing it. What's your calculation that you do? Well, you're like, okay, what's the probability that we're gonna get caught? If the probability is 100%, of course we're not gonna do it. But if the probability is 5% that we're gonna get caught, then it's gonna be like a huge embarrassment for us. And we still have our nuclear weapons anyway, so it doesn't really make an enormous difference in terms of deterring the US. And that feeds the stigma that you kind of establish, like this fabric, this universal stigma over China. Over the thing. Exactly, it's very reasonable for them to say, well, we probably get away with it, but if we don't, then the US will know we cheated. And then they're gonna go full tilt with their program and say, look, the Chinese are cheaters, and now we have all these weapons against us, and that's bad. So the stigma alone is very, very powerful. And again, look what happened with bioweapons, right? It's been 50 years now. When was the last time you read about a bioterrorism attack? The only deaths I really know about with bioweapons that have happened, when we Americans managed to kill some of our own with anthrax, the idiot who sent them to Tom Daschle and others in letters, right? And similarly in Sverdlovsk in the Soviet Union, they had some anthrax in some lab there, maybe they were cheating or who knows, and it leaked out and killed a bunch of Russians. I'd say that's a pretty good success, right? 50 years, just two own goals by the superpowers, and then nothing. And that's why whenever I ask anyone what they think about biology, they think it's great. They associate it with new cures, new diseases, maybe a good vaccine. This is how I want to think about AI in the future. I want, and I want others to think about AI too, as a source of all these great solutions to our problems, not as, oh, AI, oh yeah, that's the reason I feel scared going outside these days. Yeah, it's kind of brilliant that the bioweapons and nuclear weapons we've figured out, I mean, of course there's still a huge source of danger, but we figured out some way of creating rules and social stigma over these weapons that then creates a stability to our, whatever that game theoretic stability that, of course. Exactly, exactly. And we don't have that with AI, and you're kind of screaming from the top of the mountain about this, that we need to find that, because just like, you know, it's very possible, you know, with Future of Life, as you point out, Institute Awards pointed out that, you know, with nuclear weapons, we could have destroyed ourselves quite a few times, and it's, you know, it's a learning experience that doesn't, is very costly. We gave, you know, this Future of Life Award, we gave it the first time to this guy, Vasily Arkhipov, you know, he was on, most people haven't even heard of him. Yeah, can you say who he is? Vasily Arkhipov, he has, in my opinion, made the greatest positive contribution to humanity of any human in modern history. And maybe it sounds like hyperbole here, like I'm just over the top, but let me tell you the story, and I think maybe you'll agree. So during the Cuban Missile Crisis, we Americans first didn't know that the Russians had sent four submarines, but we caught two of them, and we didn't know that, so we dropped practice depth charges on the one that he was on, try to force it to the surface. But we didn't know that this nuclear submarine actually was a nuclear submarine with a nuclear torpedo. We also didn't know that they had authorization to launch it without clearance from Moscow, and we also didn't know that they were running out of electricity, their batteries were almost dead, they were running out of oxygen, sailors were fainting left and right, the temperature was about 110, 120 Fahrenheit on board, it was really hellish conditions, really just a kind of doomsday. And at that point, these giant explosions start happening from Americans dropping these. The captain thought World War III had begun. They decided that they were gonna launch the nuclear torpedo, and one of them shouted, we're all gonna die, but we're not gonna disgrace our Navy. We don't know what would have happened if there had been a giant mushroom cloud all of a sudden against Americans, but since everybody had their hands on the trigger, you don't have to be too creative to think that it could have led to an all-out nuclear war, in which case we wouldn't be having this conversation now. What actually took place was, they needed three people to approve this. The captain had said yes, there was the Communist Party political officer, he also said yes, let's do it. And the third man was this guy, Vasily Arkhipov, who said, and yet. For some reason, he was just more chill than the others, and he was the right man at the right time. I don't want us, as a species, rely on the right person being there at the right time. We tracked down his family living in relative poverty outside Moscow. When he flew his daughter, he had passed away, and flew them to London. They had never been to the West even. It was incredibly moving to get to honor them for this. The next year we gave a little bit of a surprise, we gave this Future Life Award to Stanislav Petrov. Have you heard of him? Yes. So he was in charge of the Soviet early warning station, which was built with Soviet technology, and honestly not that reliable. It said that there were five US missiles coming in. Again, if they had launched at that point, we probably wouldn't be having this conversation. He decided, based on mainly gut instinct, to just not escalate this. And I'm very glad he wasn't replaced by an AI that was just automatically following orders. And then we gave the third one to Matthew Messelson. Last year we gave this award to these guys who actually use technology for good, not avoiding something bad, but for something good. The guys who eliminated this disease, which is way worse than COVID, that had killed half a billion people in its final century, smallpox. So you mentioned it earlier. COVID, on average, kills less than 1% of people who get it, smallpox about 30%. And they just ultimately, Victor Zhdanov and Bill Fagy, most of my colleagues have never heard of either of them, one American, one Russian, they did this amazing effort. Not only was Zhdanov able to get the US and the Soviet Union to team up against smallpox during the Cold War, but Bill Fagy came up with this ingenious strategy for making it actually go all the way to defeat the disease without funding for vaccinating everyone. And as a result, we haven't had any, we went from 15 million deaths the year I was born in smallpox. So what do we have in COVID now? A little bit short of 2 million, right? Yes. To zero deaths, of course, this year and forever. There have been 200 million people, the estimate, who would have died since then by smallpox had it not been for this. So isn't science awesome when you use it for good? And the reason we wanna celebrate these sort of people is to remind them of this. Science is so awesome when you use it for good. And those awards, actually, the variety there paints a very interesting picture. So the first two are looking at, it's kind of exciting to think that these average humans in some sense, that there are products of billions of other humans that came before them, evolution. And some little, you said gut, but there's something in there that stopped the annihilation of the human race. And that's a magical thing, but that's like this deeply human thing. And then there's the other aspect where, that's also very human, which is to build solution to the existential crises that we're facing, to build it, to take the responsibility and to come up with different technologies and so on. And both of those are deeply human. The gut and the mind, whatever that is that creates. The best is when they work together. Arkhipov, I wish I could have met him, of course, but he had passed away. He was really a fantastic military officer combining all the best traits that we in America admire in our military, because first of all, he was very loyal, of course. He never even told anyone about this during his whole life, even though you think he had some bragging rights, right? But he just was like, this is just business, just doing my job. It only came out later after his death. And second, the reason he did the right thing was not because he was some sort of liberal or some sort of, not because he was just, oh, you know, peace and love. It was partly because he had been the captain on another submarine that had a nuclear reactor meltdown. And it was his heroism that helped contain this. That's why he died of cancer later also. But he'd seen many of his crew members die. And I think for him, that gave him this gut feeling that if there's a nuclear war between the US and the Soviet Union, the whole world is gonna go through what I saw, my own experience, gonna go through what I saw, my dear crew members, stuff like that. It wasn't just an abstract thing for him. I think it was real. And second though, not just the gut, the mind, right? He was for some reason, very level-headed personality and very smart guy, which is exactly what we want our best fighter pilots to be also, right? I never forget Neil Armstrong when he's landing on the moon and almost running out of gas. And he doesn't even change, when they say 30 seconds, he doesn't even change the tone of voice, just keeps going. Arkhipov, I think, was just like that. So when the explosions start going off and his captain is screaming, and we should nuke them and all, he's like, I don't think the Americans are trying to sink us. I think they're trying to send us a message. That's a pretty badass coolness. Because he said, if they wanted to sink us, and he said, listen, listen, it's alternating. One loud explosion on the left, one on the right, one on the left, one on the right. He was the only one who noticed this pattern. And he's like, I think this is, I'm trying to send us a signal that they wanted to surface, and they're not gonna sink us. And somehow, this is how he then managed to ultimately, with his combination of gut, and also just cool analytical thinking, was able to deescalate the whole thing. And yeah, so this is some of the best in humanity. I guess coming back to what we talked about earlier is the combination of the neural network, the instinctive, with, I'm tearing up here, I'm getting emotional. But he is one of my superheroes, having both the heart and the mind combined. And especially in that time, there's something about the, I mean, this is a very, in America, people are used to this kind of idea of being the individual, of like, on your own thinking. I think under, in the Soviet Union, under communism, it's actually much harder to do that. Oh yeah, he didn't even, he even got, he didn't get any accolades either when he came back for this, right? They just wanted to hush the whole thing up. Yeah, there's echoes of that with Chernobyl, there's all kinds of, that's one, that's a really hopeful thing that amidst big centralized powers, whether it's companies or states, there's still the power of the individual to think on their own, to act. But I think we need to think of people like this, not as a panacea we can always count on, but rather as a wake-up call. So because of them, because of Arkhipov, we are alive to learn from this lesson, to learn from the fact that we shouldn't keep playing Russian roulette and almost have a nuclear war by mistake now and then, because relying on luck is not a good long-term strategy. If you keep playing Russian roulette over and over again, the probability of surviving just drops exponentially with time. And if you have some probability of having an accidental nuclear war every year, the probability of not having one also drops exponentially. I think we can do better than that. So I think the message is very clear, once in a while shit happens, and there's a lot of very concrete things we can do to reduce the risk of things like that happening in the first place. On the AI front, if we could just link on that for a second. So you're friends with, you often talk with Elon Musk throughout history, you've did a lot of interesting things together. He has a set of fears about the future of artificial intelligence, AGI. Do you have a sense, we've already talked about the things we should be worried about with AI. Do you have a sense of the shape of his fears in particular about AI, which subset of what we've talked about, whether it's creating, it's that direction of creating sort of these giant computational systems that are not explainable, they're not intelligible intelligence, and then as a branch of that, is it the manipulation by big corporations of that or individual evil people to use that for destruction or the unintentional consequences? Do you have a sense of where his thinking is on this? From my many conversations with Elon, yeah, I certainly have a model of how he thinks. It's actually very much like the way I think also, I'll elaborate on it a bit. I just wanna push back on when you said evil people. I don't think it's a very helpful concept, evil people. Sometimes people do very, very bad things, but they usually do it because they think it's a good thing because somehow other people had told them that that was a good thing or given them incorrect information or whatever, right? I believe in the fundamental goodness of humanity that if we educate people well and they find out how things really are, people generally wanna do good and be good. Hence the value alignment. As opposed to it's about information, about knowledge, and then once we have that, we'll likely be able to do good in the way that's aligned with everybody else who thinks it's good. Yeah, and it's not just the individual people we have to align. So we don't just want people to be educated to know the way things actually are and to treat each other well, but we also need to align other non-human entities. We talked about corporations, there has to be institutions so that what they do is actually good for the country they're in and we should align, make sure that what countries do is actually good for the species as a whole, et cetera. Coming back to Elon, yeah, my understanding of how Elon sees this is really quite similar to my own, which is one of the reasons I like him so much and enjoy talking with him so much. I feel he's quite different from most people in that he thinks much more than most people about the really big picture, not just what's gonna happen in the next election cycle, but in millennia, millions and billions of years from now. And when you look in this more cosmic perspective, it's so obvious that we're gazing out into this universe that as far as we can tell, is mostly dead with life being almost imperceptibly tiny perturbation. And he sees this enormous opportunity for our universe to come alive, for us to become an interplanetary species. Mars is obviously just first stop on this cosmic journey. And precisely because he thinks more long-term, it's much more clear to him than to most people that what we do with this Russian roulette thing we keep playing with our nukes as a really poor strategy, really reckless strategy. And also that we're just building these ever more powerful AI systems that we don't understand, it's also a severely reckless strategy. I feel Elon is very much a humanist in the sense that he wants an awesome future for humanity. He wants it to be us that control the machines rather than the machines that control us. Yes. And why shouldn't we insist on that? We're building them after all, right? Why should we build things that just make us into some little cog in the machinery that has no further say in the matter, right? It's not my idea of an inspiring future either. Yeah, if you think on the cosmic scale in terms of both time and space, so much is put into perspective. Yeah. Whenever I have a bad day, that's what I think about it. Immediately makes me feel better. It makes me sad that for us individual humans, at least for now, the ride ends too quickly. We don't get to experience the cosmic scale. Yeah, I mean, I think of our universe sometimes as an organism that has only begun to wake up a tiny bit, just like when you're in the very first little glimmers of consciousness you have in the morning when you start coming around. Before the coffee. Before the coffee, even before you get out of bed, before you even open your eyes, start to wake up a little bit. There's something here. That's very much how I think of where we are. All those galaxies out there, I think they're really beautiful. But why are they beautiful? They're beautiful because conscious entities are actually observing them and experiencing them through our telescopes. I define consciousness as subjective experience, whether it be colors or emotions or sounds. So beauty is an experience. Meaning is an experience. Purpose is an experience. If there was no conscious experience observing these galaxies, they wouldn't be beautiful. If we do something dumb with advanced AI in the future here and Earth originating, life goes extinct. And that was it for this. If there is nothing else with telescopes in our universe, then it's kind of game over for beauty and meaning and purpose in our whole universe. And I think that would be just such an opportunity lost, frankly. And I think when Elon points this out, he gets very unfairly maligned in the media for all the dumb media bias reasons we talked about. They want to print precisely the things about Elon out of context that are really clickbaity. He has gotten so much flack for this summoning the demon statement. I happen to know exactly the context because I was in the front row when he gave that talk. It was at MIT, you'll be pleased to know. It was the AeroAstro anniversary. They had Buzz Aldrin there from the moon landing, whole house, a Kresge auditorium packed with MIT students. And he had this amazing Q&A, it might've gone for an hour. And they talked about rockets and Mars and everything. At the very end, this one student who has actually hit my class asked him, what about AI? Elon makes this one comment and they take this out of context, print it, goes viral. Or is it like with AI, we're summoning the demon, something like that. And try to cast him as some sort of doom and gloom dude. You know Elon, he's not the doom and gloom dude. He is such a positive visionary. And the whole reason he warns about this is because he realizes more than most what the opportunity cost is of screwing up, that there is so much awesomeness in the future that we can and our descendants can enjoy if we don't screw up, right? I get so pissed off when people try to cast him as some sort of technophobic Luddite. And at this point, it's kind of ludicrous when I hear people say that people who worry about artificial general intelligence are Luddites because of course, if you look more closely, you have some of the most outspoken people making warnings are people like Professor Stuart Russell from Berkeley, who's written the bestselling AI textbook. So claiming that he's a Luddite who doesn't understand AI is the joke is really on the people who said it. But I think more broadly, this message is really not sunk in at all. What it is that people worry about, they think that Elon and Stuart Russell and others are worried about the dancing robots picking up an AR-15 and going on a rampage, right? They think they're worried about robots turning evil. They're not, I'm not. The risk is not malice, it's competence. The risk is just that we build some systems that are incredibly competent, which means they're always gonna get their goals accomplished even if they clash with our goals. That's the risk. Why did we humans drive the West African black rhino extinct? Is it because we're malicious, evil, rhinoceros haters? No, it's just because our goals didn't align with the goals of those rhinos and tough luck for the rhinos. So the point is just we don't wanna put ourselves in the position of those rhinos creating something more powerful than us if we haven't first figured out how to align the goals. And I am optimistic, I think we could do it if we worked really hard on it because I spent a lot of time around intelligent entities that were more intelligent than me, my mom and my dad. And I was little and that was fine because their goals were actually aligned with mine quite well. But we've seen today many examples of where the goals of our powerful systems are not so aligned. So those click through optimization algorithms that are polarized social media, right? They were actually pretty poorly aligned with what was good for democracy, it turned out. And again, almost all problems we've had in the machine learning again came so far, not from malice, but from poor alignment. And that's exactly why we should be concerned about it in the future. Do you think it's possible that with systems like Neuralink and brain computer interfaces, again, thinking of the cosmic scale, Elon's talked about this, but others have as well throughout history of figuring out how the exact mechanism of how to achieve that kind of alignment. So one of them is having a symbiosis with AI, which is like coming up with clever ways where we're like stuck together in this weird relationship, whether it's biological or in some kind of other way. Do you think that's a possibility of having that kind of symbiosis? Or do we wanna instead kind of focus on this distinct entities of us humans talking to these intelligible, self-doubting AIs, maybe like Stuart Russell thinks about it. It's like we're self-doubting and full of uncertainty and then our AI systems are full of uncertainty. We communicate back and forth and in that way achieve symbiosis. I honestly don't know. I was thinking about it, I honestly don't know. I would say that because we don't know for sure which of any of our ideas will work, but we do know that if we don't, I'm pretty convinced that if we don't get any of these things to work and just barge ahead, then our species is probably gonna go extinct this century. I think it's- This century, you think we're facing this crisis is the 21st century crisis. This century will be remembered. On a hard drive. On a hard drive somewhere or maybe by future generations as like there'll be future of Life Institute awards for people that have done something about AI. It could also end even worse, where we're not superseded by leaving any AI behind either. We just totally wipe out, like on Easter Island. Our century is long. No, there are still 79 years left of it. Think about how far we've come just in the last 30 years. We can talk more about what might go wrong, but you asked me this really good question about what's the best strategy. Is it Neuralink or do it Russell's approach or whatever? I think when we did the Manhattan Project, we didn't know if any of our four ideas for enriching uranium and getting out the uranium-235 were gonna work. We felt this was really important to get it before Hitler did. So you know what we did? We tried all four of them. Here, I think it's analogous where there's the greatest threat that's ever faced our species. And of course, US national security by implication. We don't know, we don't have any method that's guaranteed to work, but we have a lot of ideas. So we should invest pretty heavily in pursuing all of them with an open mind and the hope that one of them at least works. These are, the good news is the century is long, and it might take decades until we have artificial general intelligence. So we have some time hopefully, but it takes a long time to solve these very, very difficult problems. It's gonna actually be the most difficult problem we were ever trying to solve as a species. So we have to start now. So we don't wanna have, rather than begin thinking about it the night before some people who've had too much red bull switching on and we have to, coming back to your question, we have to pursue all of these different avenues and see. If you're my investment advisor, and I was trying to invest in the future, how do you think the human species is most likely to destroy itself in the century? Yeah, so if the crises, many of the crises we're facing are really before us within the next 100 years, how do we make explicit, make known the unknowns and solve those problems to avoid the biggest, starting with the biggest existential crisis? So as your investment advisor, how are you planning to make money on us destroying ourselves, I have to ask? I don't know. It might be the Russian origins. Somehow it's involved. At the micro level of detailed strategies, of course, these are unsolved problems. For AI alignment, we can break it into three sub problems that are all unsolved. I think you want first to make machines understand our goals, then adopt our goals and then retain our goals. So to hit on all three real quickly. The problem when Andreas Lubitz told his autopilot to fly into the Alps was that the computer didn't even understand anything about his goals. It was too dumb. It could have understood actually, but you would have had to put some effort in as a system designer to don't fly into mountains. So that's the first challenge. How do you program into computers, human values, human goals? We could start, rather than saying, oh, it's so hard, we should start with the simple stuff, as I said, self-driving cars, airplanes, just put in all the goals that we all agree on already, and then have a habit of whenever machines get smarter, so they can understand one level higher goals, put them into. The second challenge is getting them to adopt the goals. It's easy for situations like that, where you just program it in, but when you have self-learning systems like children, any parent knows that there's a difference between getting our kids to understand what we want them to do and to actually adopt our goals. With humans, with children, fortunately, they go through this phase. First, they're too dumb to understand what our goals are, and then they have this period, some years, when they're both smart enough to understand them and malleable enough that we have a chance to raise them well, and then they become teenagers, and it's too late. But we have this window. With machines, the intelligence might grow so fast that that window is pretty short. So that's a research problem. The third one is how do you make sure they keep the goals if they keep learning more and getting smarter? Many sci-fi movies are about how you have something in which initially was a line, but then things kind of go off-heel. My kids were very, very excited about their Legos when they were little, now they're just gathering dust in the basement. If we create machines that are really on board with the goal of taking care of humanity, we don't want them to get as bored with us as my kids got with Legos. So this is another research challenge. How can you make some sort of recursively self-improving system retain certain basic goals? That said, a lot of adult people still play with Legos, so maybe we succeeded with the Legos. Maybe, I like your optimism. But above all- So not all AI systems have to maintain the goals, right? Some just some fraction. Yeah, so there's a lot of talented AI researchers now who have heard of this and wanna work on it. Not so much funding for it yet. Of the billions that go into building AI more powerful, it's only a minuscule fraction. So far going into the safety research. My attitude is generally we should not try to slow down the technology, but we should go into the safety research. We should not try to slow down the technology, but we should greatly accelerate the investment in this sort of safety research. And also make sure, it's been, this was very embarrassing last year, but the NSF decided to give out six of these big institutes. We got one of them for AI and science, you asked me about. Another one was supposed to be for AI safety research, and they gave it to people studying oceans and climate and stuff. So I'm all for studying oceans and climates, have some money that actually goes into AI safety research also and doesn't just get grabbed by whatever. That's a fantastic investment. And then at the higher level, you asked this question, okay, what can we do? What are the biggest risks? I think we cannot just consider this to be only a technical problem. Again, because if you solve only the technical problem, can I play with your robot? Yes, please. If we can get our machines to just blindly obey the orders we give them, so we can always trust that it will do what we want, that might be great for the owner of the robot, that might not be so great for the rest of humanity if that person is that least favorite world leader or whatever you imagine. So we have to also take a look at the apply alignment, not just to the machines, but to all the other powerful structures. That's why it's so important to strengthen our democracy again. As I said, to have institutions that make sure that the playing field is not rigged so that corporations are given the right incentives to do the things that both make profit and are good for people, to make sure that countries have incentives to do things that are both good for their people and don't screw up the rest of the world. And this is not just something for AI nerds to geek out on. This is an interesting challenge for political scientists, economists, and so many other thinkers. So one of the magical things that perhaps makes this earth quite unique is that it's home to conscious beings. So you mentioned consciousness. Perhaps as a small aside, because we didn't really get specific to how we might do the alignment. Like you said, it's just a really important research problem. But do you think engineering consciousness into AI systems is a possibility? It's something that we might one day do? Or is there something fundamental to consciousness that is... Is there something about consciousness that is fundamental to humans and humans only? I think it's possible. I think both consciousness and intelligence are information processing. Certain types of information processing. And that fundamentally, it doesn't matter whether the information is processed by carbon atoms in neurons and brains or by silicon atoms and so on in our technology. Some people disagree. This is what I think as a physicist. That consciousness is the same kind of... You said consciousness is information processing. So meaning, I think you had a quote of something like it's information knowing itself, that kind of thing. I think consciousness is the way information feels when it's being processed. When it's being prepared. We don't know exactly what those complex ways are. It's clear that most of the information processing in our brains does not create an experience. We're not even aware of it. For example, you're not aware of your heartbeat regulation right now, even though it's clearly being done by your body. It's just kind of doing its own thing. When you go jogging, there's a lot of complicated stuff about how you put your foot down. And we know it's hard. That's why robots used to fall over so much. But you're mostly unaware about it. Your brain, your CEO consciousness module just sends an email. Hey, I want to keep jogging along this path. The rest is on autopilot. So most of it is not conscious. But somehow there are some of the information processing which is, we don't know what exactly. I think this is a science problem that I hope one day have some equation for or something. So we can be able to build a consciousness detector and say, yeah, here there is some consciousness. Here there's not. Oh, don't boil that lobster because it's feeling pain. Or it's okay because it's not feeling pain. Right now we treat this as sort of just metaphysics. But it would be very useful in emergency rooms to know if a patient has locked in syndrome and is conscious or if they are actually just out. And in the future, if you build a very, very intelligent helper robot to take care of you, I think you'd like to know if you should feel guilty by shutting it down. Or if it's just like a zombie going through the motions like a fancy tape recorder. Right. And once we can make progress on the science of consciousness and figure out what is conscious and what isn't, then we, assuming we want to create positive experiences and not suffering, we'll probably choose to build some machines that are deliberately unconscious that do incredibly boring, repetitive jobs in an iron mine somewhere or whatever. And maybe we'll choose to create helper robots for the elderly that are conscious so that people don't just feel creeped out that the robot is just faking it when it acts like it's sad or happy. Like I said, elderly, I think everybody gets pretty deeply lonely in this world. Yeah. And so there's a place, I think for everybody to have a connection with conscious beings, whether they're human or otherwise. But I know for sure that I would, if I had a robot, if I was going to develop any kind of personal emotional connection with it, I would be very creeped out if I knew it in intellectual level, that the whole thing was just a fraud. Now today you can buy a little talking doll for a kid, which will say things. And the little child will often think that this is actually conscious and even real secrets to it that then go on the internet and with all sorts of creepy repercussions. I would not want to be just hacked and tricked like this. If I was going to be developing real emotional connections with a robot, I would want to know that this is actually real. It's acting conscious, acting happy because it actually feels it. And I think this is not sci-fi. I think it's possible to measure, to come up with tools. After we understand the science of consciousness, you're saying we'll be able to come up with tools that can measure consciousness and definitively say like this thing is experiencing the things it says it's experiencing. Kind of by definition, if it is a physical phenomena, information processing, and we know that some information processing is conscious and some isn't, well, then there is something there to be discovered with the methods of science. Giulio Tononi has stuck his neck out the farthest and written down some equations for a theory. Maybe that's right. Maybe it's wrong. We certainly don't know. But I applaud that kind of efforts to sort of take this, say this is not just something that philosophers can have beer and muse about, but something we can measure and study. And coming, being that back to us, I think what we would probably choose to do, as I said, is if we cannot figure this out, choose to make, to be quite mindful about what sort of consciousness, if any, we put in different machines that we have. We, and certainly we wouldn't want to make, we should not be making much machines that suffer without us even knowing it. Right. And if, at any point, someone decides to upload themselves, like Ray Kurzweil wants to do, I don't know if you've had him on your show. We agree, but then COVID happened, so we're waiting it out a little bit. You know, suppose he uploads himself into this Robo-Ray, and it talks like him and acts like him and laughs like him. And before he powers off his biological body, he would probably be pretty disturbed if he realized that there's no one home. This robot is not having any subjective experience. Right. If we, if humanity gets replaced by, by, by machine descendants, do all these cool things and build spaceships and, and go to intergalactic rock concerts. And it turned out, turns out that they are all unconscious, just going through the motions. Wouldn't that be like the ultimate robot zombie apocalypse? Right. Just a play for empty benches. Yeah. I have a sense that there's some kind of, once we understand consciousness better, we'll understand that there's some kind of continuum, and it would be a greater appreciation. And we'll probably understand, just like you said, it'd be unfortunate if it's a trick. We'll probably definitively understand that love is indeed a trick that we'll play on each other, uh, that we humans are, we convince ourselves we're conscious, but we're really, um, you know, awesome trees and dolphins are all the same kind of conscious. Can I try to cheer you up a little bit with a philosophical thought here about the love part? Yes, let's do it. You know, you might say, okay, yeah, love is just a collaboration enabler. And then you'll, and then maybe you can go and get depressed about that. But I think that would be the wrong conclusion, actually. You know, I know that the only reason I enjoy food is because my genes hacked me and they don't want me to starve to death, not because they care about me. It's consciously enjoying succulent to delights of pistachio ice cream, but they just, they just want me to make copies of them. The whole thing. So in a sense, the whole, the whole idea of love is just a scam. The whole enjoyment of food is also a scam like this. But does that mean I shouldn't take pleasure in this pistachio ice cream? I love pistachio ice cream. And I can tell you, I have, I have, I know this is an experimental fact. I enjoy pistachio ice cream every bit as much, even though I scientifically know exactly why, what kind of scam this was. Your genes really appreciate that you like the pistachio ice cream. Well, but I, my mind appreciates it too, you know, I have a conscious experience right now. Ultimately, all of my brain is also just something the genes built to copy themselves, but so what, you know, I'm grateful that, yeah, thanks genes for doing this, but you know, now it's my brain that's in charge here and I'm going to enjoy my conscious experience. Thank you very much. And not just the pistachio ice cream, but also the love I feel for my amazing wife and all the other delights of being conscious. I don't actually, Richard Feynman, I think said this so well. He is also the guy, you know, really got me into physics. Some art friend said that, oh, science kind of just is the party pooper. It's kind of ruins the fun, right? It went like, you have a beautiful flowers as the artist, and then the scientist is going to deconstruct that into just a blob of quirks and electrons. And, and Feynman pushed back on that in such a beautiful way, which I think also can be used to push back and make you not feel guilty about falling in love. So here's what Feynman basically said. He said to his friend, you know, yeah, I can also, as a scientist, see that this is a beautiful flower. Thank you very much. Maybe I'm, I can't draw as good a painting as you because I'm not as talented an artist, but yeah, I can really see the beauty in it. And it just, it also looks beautiful to me. But in addition to that, Feynman said as a scientist, I see even more beauty that the artist did not see. Right. Suppose this is the flower on a blossoming apple tree. You could say this tree has more beauty in it than just the colors and the fragrance. This tree is made of air. Feynman wrote. This is one of my favorite Feynman quotes ever. And it took the carbon out of the air and bound it in using the flaming heat of the sun, you know, to turn the air into tree. And when you burn logs in your fireplace, it's really beautiful to think that this is being reversed. Now the tree is going, the wood is going back into air and this flaming, beautiful dance of the fire that the artists can see is the flaming light of the sun that was bound into turn the air into tree. And then the ashes is the little residue that didn't come from the air that the tree sucked out of the ground. You know, Feynman said, these are beautiful things and science just adds. It doesn't subtract. And I feel exactly that way about love and about pistachio ice cream. Also, I can understand that it didn't even, there is even more nuance to the whole thing, right? At this very visceral level, you can fall in love just as much as someone who knows nothing about neuroscience, but you can also appreciate this even greater beauty in it. It's just like, isn't it remarkable that it came about from, from this completely lifeless universe, just a bunch of hot blob of plasma expanding. And then over the eons, you know, gradually first the strong nuclear force decided to combine quirks together into nuclei. And then the electric force bound in electrons and made atoms, and then they clustered from gravity and you've got planets and stars and this and that. And then natural selection came along and the genes had their little thing. And you started getting what went from seeming like a completely pointless universe that we're just trying to increase entropy and approach heat death into something that looked more goal oriented. Isn't that kind of beautiful? And then this goal orientedness revolution got ever more sophisticated where you got ever more. And then you started getting this thing, which is kind of like DeepMind's mu zero and steroids, self, the ultimate self play is not what DeepMind's AI does against itself to get better at go. It's what all these little cork blobs did against each other in the game of survival of the fittest. Now, when you had really dumb bacteria living in a simple environment, there wasn't much incentive to get an intelligent, but then the life made environment more complex and then there was more incentive to get even smarter. And that gave the other organisms more of incentive to also get smarter. And then here we are now just like, like mu zero learned to become world master at, at the go and chess from playing as itself by just playing against itself, all the quirks here on our planet and electrons have created giraffes and elephants and humans and pistachio ice cream. I just find that really beautiful. And I, to me, that just adds to the enjoyment of, of love. It doesn't subtract anything. Do you feel a little more? I feel way better. That was, that was incredible. So the, this self play of quirks taking back to the beginning of our conversation a little bit, you've, there's so many exciting possibilities about artificial intelligence, understanding the basic laws of physics. Do you think AI will help us unlock? There's been a, quite a bit of excitement throughout the history of physics of coming up with more and more general, simple laws that explain the nature of our reality. And then the ultimate of that would be a theory of everything that combines everything together. Do you think it's possible that, well, one, we humans, but perhaps AI systems will figure out a theory of physics that unifies all the laws of physics? Yeah, I think it's absolutely possible. I think it's very clear that we're going to see a great boost to science. We're already seeing a boost actually from, from machine learning, helping science, alpha fold was an example, you know, decades old protein folding problem. So gradually, yeah. Unless we go extinct by doing something dumb, like we discussed, I think it's very likely that our understanding of physics will become so good that our technology will no longer be limited by human intelligence, but instead be limited by the laws of physics. So our tech today is limited by what we've been able to invent, right? I think as AI progresses, it'll just be limited by the speed of light and other physical limits, which will mean it's going to be just dramatically beyond where we are now. Do you think it's a fundamentally mathematical pursuit of trying to understand like the laws of this, that govern our universe from a mathematical perspective? So almost like if it's AI, it's exploring the space of like theorems and those kinds of things, or is there some other, is there some other more computational ideas, more sort of empirical ideas? They're both. I would say it's really interesting to look out at the landscape of everything we call science today. So here you come now with this big new hammer, it says machine learning on it and ask, you know, where are there some nails that you can help with here that you can hammer? Ultimately, if machine learning gets the point that it can do everything better than us, it will be able to help across the whole space of science. But maybe we can anchor it by starting a little bit right now near term and see how we kind of move forward. So right now, first of all, you have a lot of big data science where, for example, with telescopes, we are able to collect way more data every hour than a grad student can just pour over like in the old times. Right. And machine learning is already being used very effectively, even at MIT, like to find planets around other stars, to detect exciting new signatures of new particle physics in the sky and to detect the ripples and the fabric of space time that we call gravitational waves caused by enormous black holes crashing into each other halfway across the observable universe. Machine learning is running and taking it right now, you know, doing all these things. And it's really helping all these experimental fields. There is a separate front of physics, computational physics, which is getting an enormous boost also. So we had to do all our computations by hand, right? People would have these giant books with tables of logarithms and, oh my God, it pains me to even think how long time it would have taken to do simple stuff. Then we started to get little calculators and computers that could do some basic math for us. Now what we're starting to see is kind of a shift from GoFi computational physics to neural network computational physics. What I mean by that is most computational physics would be done by humans programming in the intelligence of how to do the computation into the computer. Just as when Gary Kasparov got his posterior kicked by IBM's Deep Blue in chess, humans had programmed in exactly how to play chess. Intelligence came from the humans. It wasn't learned, right? Mu zero can be not only Kasparov in chess, but also Stockfish, which is the best sort of GoFi chess program by learning. We're seeing more of that now, that shift beginning to happen in physics. Let me give you an example. So Lattice QCD is an area of physics whose goal is basically to take the periodic table and just compute the whole thing from first principles. This is not the search for theory of everything. We already know the theory that's supposed to produce as output the periodic table, which atoms are stable, how heavy they are, all that good stuff. There's spectral lines. It's a theory, Lattice QCD. You can put it on your t-shirt. Our colleague Frank Wilczek got the Nobel Prize for working on it. But the math is just too hard for us to solve. We have not been able to start with these equations. And solve them to the extent that we can predict, oh yeah. And then there is carbon and this is what the spectrum of the carbon atom looks like. But awesome people are building these super computer simulations where you just put in these equations and you make a big cubic lattice of space, or actually it's a very small lattice because you're going to the subatomic scale, down to the subatomic scale, and you try to solve it, and you get a very, very big cube of space, and you're like, oh, I'm going to solve this. But it's just so computationally expensive that we still haven't been able to calculate things as accurately as we measure them in many cases. And now machine learning is really revolutionizing this. So my colleague, Fiyala Shanahan at MIT, for example, she's been using this really cool machine learning technique called normalizing flows, where she's realized she can actually speed up the calculation dramatically by having the AI in other area like this, where we suck up an enormous amount of super computer time to do physics as black hole collisions. So now that we've done the sexy stuff of detecting a bunch of this, LIGO and other experiments, we want to be able to know what we're seeing. And so it's a very simple conceptual problem. It's the two body problem. Newton solved it for classical gravity, hundreds of years ago. But the two body problem is still not fully solved. For black holes. Yes, and Einstein's gravity, because they won't just orbit each other forever anymore, two things. They give off gravitational waves and make sure they crash into each other. And the game, what you want to do is you want to figure out, okay, what kind of wave comes out as a function of the masses of the two black holes, as a function of how they're spinning relative to each other, et cetera. And that is so hard. It can take months of super computer time and massive numbers of cores to do it. Wouldn't it be great if you can use machine learning to greatly speed that up? Now you can use the expensive old go-fi calculation as the truth, and then see if machine learning can figure out a smarter, faster way of getting the right answer. Yet another area, like computational physics. These are probably the big three that suck up the most computer time. Lattice QCD, black hole collisions, and cosmological simulations, where you take not a subatomic thing and try to figure out the mass of the proton, but you take something that's enormous and try to look at how all the galaxies get formed in there. There again, there are a lot of very cool ideas right now about how you can use machine learning to do this sort of stuff better. The difference between this and the big data is you kind of make the data yourself. And then finally, we're looking over the physics landscape and saying, what can we hammer with machine learning? So we talked about experimental data, big data, discovering cool stuff that we humans then look more closely at. Then we talked about taking the expensive computations we're doing now and figuring out how to do them much faster and better with AI. And finally, let's go really theoretical. Things like discovering equations, having deep fundamental insights. This is something closest to what I've been doing in my group. We talked earlier about the whole AI Feynman project, where if you just have some data, how do you automatically discover equations that seem to describe this well, that you can then go back as a human and work with and test and explore? You asked a really good question also about if this is sort of a search problem in some sense. That's very deep, actually, what you said, because it is. Suppose I ask you to prove some mathematical theorem. What is a proof in math? It's just a long string of steps, logical steps that you can write out with symbols. And once you find it, it's very easy to write a program to check whether it's a valid proof or not. Why is it so hard to prove it then? Well, because there are ridiculously many possible candidate proofs you could write down. If the proof contains 10,000 symbols, even if there were only 10 options for what each symbol could be, that's 10 to the power of 1,000 possible proofs, which is way more than there are atoms in our universe. You could say it's trivial to prove these things. You just write a computer, generate all strings, and then check, is this a valid proof? No. Is this a valid proof? No. And then you just keep doing this forever. But it is fundamentally a search problem. You just want to search the space of all those all strings of symbols to find one that is the proof, right? And there's a whole area of machine learning called search. How do you search with some giant space to find the needle in the haystack? It's easier in cases where there's a clear measure of good, like you're not just right or wrong, but this is better and this is worse. You can maybe get some hints as to which direction to go in. That's why we talked about neural networks work so well. I mean, that's such a human thing of that moment of genius of figuring out the intuition of good, essentially. I mean, we thought that that was... Or is it? Maybe it's not right. We thought that about chess, right? Exactly. The ability to see like 10, 15, sometimes 20 steps ahead was not a calculation that humans were performing. It was some kind of weird intuition about different patterns, about board positions, about the relative positions, somehow stitching stuff together. And a lot of it is just like intuition. But then you have like Alpha, I guess Zero be the first one that did the self-play. It just came up with this. It was able to learn through self-play mechanism, this kind of intuition. But just like you said, it's so fascinating to think whether in the space of totally new ideas, can that be done in developing theorems? We know it can be done by neural networks because we did it with the neural networks in the craniums of the great mathematicians of humanity. And I'm so glad you brought up Alpha Zero because that's the counter example. It turned out we were flattering ourselves when we said intuition is something different. Only humans can do it. It's not information processing. It used to be that way. Again, it's really instructive, I think, to compare the chess computer Deep Blue that beat Kasparov with Alpha Zero that beat Lisa Doll at the go. Because for Deep Blue, there was no intuition. There was some pro, humans had programmed in some intuition. After humans had played a lot of games, they told the computer, you know, count the pawn as one point, the bishop is three points, the rook is five points, and so on. You add it all up and then you add some extra points for past pawns and subtract if the opponent has it and blah, blah, blah, blah. And then what Deep Blue did was just search. Just very brute force and tried many, many moves ahead and all these combinations and a pruned tree search. And it could think much faster than Kasparov and it won. Right. And that, I think, inflated our egos in a way it shouldn't have because people started to say, yeah, yeah, it's just brute force search, but has no intuition. Yeah. Alpha zero really popped our bubble there because what Alpha Zero does, yes, it does also do some of that tree search, but it also has this intuition module, which in geek speak is called a value function, where it just looks at the board and comes up with a number for how good is that position. The difference was no human told it how good the position is. It just learned it. And Mu Zero is the coolest or scariest of all, depending on your mood, because it, the same basic AI system will learn what the good board position is, regardless of whether it's chess or go or Shogi or Pac-Man or Lady Pac-Man or Breakout or Space Invaders or any number, bunch of other games. You don't tell anything and it gets this intuition after a while for what's good. So this is very hopeful for science, I think, because if it can get intuition for what's a good position there, maybe it can also get intuition for what are some good directions to go if you're trying to prove something. I often, one of the more most fun things in my science career is when I've been able to prove some theorem about something and it's very heavily intuition guided. Of course, I don't sit and try all random strings. I have a hunch that this reminds me a little bit of about this other proof I've seen for this thing. So maybe I first, what if I try this? Nah, that didn't work out. But this reminds me actually the way this failed reminds me of that. And so combining the intuition with all these brute force capabilities, I think it's going to be able to help physics too. Do you think there'll be a day when an AI system being the primary contributor, let's say 90% plus wins the Nobel Prize in physics? Obviously they'll give it to the humans because we humans don't like to give prizes to machines. It'll give it to the humans behind the system. You could argue that AI has already been involved in some Nobel prizes, probably maybe something with black holes and stuff like that. Yeah, we don't like giving prizes to other life forms. If someone wins a horse racing contest, they don't give the prize to the horse either. That's true. But do you think we might be able to see something like that in our lifetimes when AI... So the first system I would say that makes us think about a Nobel Prize seriously is like Alpha Fold is making us think about in medicine and physiology, a Nobel Prize, perhaps discoveries that are a direct result of something that's discovered by Alpha Fold. Do you think in physics we might be able to see that in our lifetimes? I think what's probably going to happen is more of a blurring of the distinctions. So today if somebody uses a computer to do a computation that gives them the Nobel Prize, nobody's going to dream of giving the prize to the computer. They're going to be like, that was just a tool. I think for these things also people are just going to for a long time view the computer as a tool. But what's going to change is the ubiquity of machine learning. I think at some point in my lifetime, finding a human physicist who knows nothing about machine learning is going to be almost as hard as it is today finding a human physicist who doesn't say, oh, I don't know anything about computers or I don't use math. It would just be a ridiculous concept. But the thing is, there is a magic moment though, like with Alpha Zero, when the system surprises us in a way where the best people in the world truly learn something from the system in a way where you feel like it's another entity. Like the way people, the way Magnus Carlsen, the way certain people are looking at the work of Alpha Zero, it truly is no longer a tool in the sense that it doesn't feel like a tool. It feels like some other entity. So there is a magic difference where you're like, if an AI system is able to come up with an insight that surprises everybody in some major way that's a phase shift in our understanding of some particular science or some particular aspect of physics, I feel like that is no longer a tool. And then you can start to say that it perhaps deserves the prize. So for sure, the more important, the more fundamental transformation of the 21st century science is exactly what you're saying, which is probably everybody will be doing machine learning. It's to some degree, like if you want to be successful at unlocking the mysteries of science, you should be doing machine learning. But it's just exciting to think about like, whether there'll be one that comes along that's super surprising and they'll make us question like who the real inventors are in this world. Yeah. Yeah, I think the question of isn't if it's going to happen, but when, but it's important also in my mind, the time when that happens is also more or less the same time when we get artificial general intelligence. And then we have a lot bigger things to worry about than whether you get the Nobel Prize or not. Right. Because when you have machines that can outperform our best scientists at science, they can probably outperform us at a lot of other stuff as well, which can at a minimum, you know, make them incredibly powerful agents in the world. And I think it's a mistake to think we only have to start worrying about loss of control when machines get to AGI across the board, where they can do everything, all our jobs. Long before that, they'll be hugely influential. We talked at length about how the hacking of our minds with algorithms trying to get us glued to our screens, right, has already had a big impact on society. That was an incredibly dumb algorithm in the grand scheme of things, right? The supervised machine learning, yet that had huge impact. So I just don't want us to be lulled into a false sense of security and think there won't be any societal impact until things reach human level because it's happening already. I was just thinking the other week, you know, when I see some scaremonger going, oh, the robots are coming. The implication is always that they're coming to kill us. And maybe you should have worried about that if you were in Nagorno-Karabakh during the recent war there. But more seriously, the robots are coming right now, but they're mainly not coming to kill us. They're coming to hack us. They're coming to hack our minds into buying things that maybe we didn't need, to vote for people who may not have our best interests in mind. And it's kind of humbling, I think, actually, as a human being to admit that it turns out that our minds are actually much more hackable than we thought. And the ultimate insult is that we are actually getting hacked by the machine learning algorithms that are in some objective sense much dumber than us. But maybe we shouldn't be so surprised because, you know, how do you feel about cute puppies? Love them. So, you know, you would probably argue that in some across the board measure, you're more intelligent than they are. But boy, are our cute puppies good at hacking us, right? Yeah. They move into our house, persuade us to feed them and do all these things. What do they ever do for us? Yeah. Other than being cute and making us feel good, right? So if puppies can hack us, maybe we shouldn't be so surprised if pretty dumb machine learning algorithms can hack us too. Not to speak of cats, which is another level. And I think we should, to counter your previous point about there, let us not think about evil creatures in this world. We can all agree that cats are as close to objective evil as we can get. But that's just me saying that. OK. Have you seen that? The cartoon, I think it's maybe the onion where this incredibly cute kitten and it just says underneath something that thinks about murder all day. Exactly. That's accurate. You've mentioned offline that there might be a link between post biological AGI and SETI. So last time we talked, you've talked about this intuition. We humans might be quite unique in our galactic neighborhood, perhaps our galaxy, perhaps the entirety of the observable universe, who might be the only intelligent civilization here, which is. And you argue pretty well for that thought. So I have a few little questions around this. One, the scientific question, in which way would you be if you were wrong in that intuition? In which way do you think you would be surprised? Like, why were you wrong? We find out that you ended up being wrong. In which dimension? So like, is it because we can't see them? Is it because the nature of their intelligence or the nature of their life is totally different than we can possibly imagine? Is it because the I mean, something about the great filters and surviving them? Or maybe because we're being protected from signals? So all those explanations for. For why we haven't heard a big, loud, like red light that says we're here. Yeah. So there are actually two separate things there that I could be wrong about. Two separate claims that I made, right? None of them. One, one of them is. I made the claim, I think. Most civilizations. When you're going from simple bacteria like things to space, space colonizing civilizations, they spend only a very, very tiny fraction of their other, of their life being where we are. That I could be wrong about. The other one I could be wrong about is that it's quite different states. It's a very, very different state. It's a very, very different state. The other one I could be wrong about is that it's quite different statements that I think that actually I'm guessing that we are the only civilization in our observable universe from which light has reached us so far that's actually gotten far enough to invent telescopes. So let's talk about maybe both of them in turn, because they really are different. The first one, if you look at the N equals one, the data point we have on this planet. Yeah. So we spent four and a half billion years futzing around on this planet with life. And most of it was pretty lame stuff from an intelligence perspective. Bacteria and then the dinosaurs spent, but things gradually accelerated. Then the dinosaurs spent over a hundred million years stomping around here without even inventing smartphones. And then very recently, we've only spent 400 years going from Newton to us, right? In terms of technology. And look what we've done even when I was a little kid, there was no internet even. So I think it's pretty likely in this case of this planet, we're either going to really get our act together and start spreading life into space the century. And doing all sorts of great things, or we're going to wipe out. It's a little hard. I could be wrong in the sense that maybe what happened on this earth is very atypical. And for some reason, what's more common on other planets is that they spend an enormously long time futzing around with the ham radio and things, but they just never really take it to the next level. For reasons I haven't understood, I'm humble and open to that. But I would bet at least 10 to 1, that our situation is more typical because the whole thing with Moore's law and accelerating technology, it's pretty obvious why it's happening. Everything that grows exponentially, we call it an explosion, whether it's a population explosion or a nuclear explosion, it's always caused by the same thing. It's that the next step triggers a step after that. Today's technology enables tomorrow's technology and that enables the next level. And because the technology is always better, of course, the steps can come faster and faster. On the other question that I might be wrong about, that's the much more controversial one, I think. But before we close out on this thing about the first one, if it's true that most civilizations spend only a very short amount of their total time in the stage, say, between inventing telescopes or mastering electricity and doing space travel. If that's actually generally true, then that should apply also elsewhere out there. So we should be very, very surprised if we find some random civilization and we happen to catch them exactly in that very, very short stage. It's much more likely that they're going to be able to do that. Or that we find some civilization that's already post-biological and has done some really cool galactic construction projects in their galaxy. Would we be able to recognize them, do you think? Is it possible that we just can't? I mean, this post-biological world, could it be just existing in some other dimension? It could be just all a virtual reality game, for them, or something, I don't know, that it changes completely where we won't be able to detect? We have to be, honestly, very humble about this. I think I said earlier the number one principle being scientists is you have to be humble and willing to acknowledge that everything we think just might be totally wrong. Of course, you could imagine some civilization where they all decide to become Buddhists and very inward-looking, and they're all going to move into their little virtual reality and not disturb the flora and fauna around them, and we might not notice them. But this is a numbers game, right? If you have millions of civilizations out there, or billions of them, all it takes is one with a more ambitious mentality that decides, hey, we are going to go out and settle a bunch of other things that are going to be in our own galaxy, and then it doesn't matter if they're a bunch of quiet Buddhists, we're still going to notice that expansionist one, right? And it seems like quite the stretch to assume that, you know, we know even in our own galaxy that there are probably a billion or more planets that are pretty Earth-like, and many of them were formed over a billion years before ours, so it had a big head start. So if you actually assume also that life happens kind of automatically on an Earth-like planet, I think it's quite the stretch to then go and say, okay, so we have billions of other civilizations out there that also have our level of tech, and they all decided to become Buddhists, and not a single one decided to go Hitler on the galaxy and say, we need to go on and colonize, or not a single one decided for more benevolent reasons to go out and get more resources. That seems like a bit of a stretch, frankly, and this leads into the second thing you challenged me that I might be wrong about, how rare or common is life? So Francis Drake, when he wrote down the Drake equation, multiplied together a huge number of factors, and then we know that life is a very, very, very, very, very, very, very, very, very large number of factors, and we don't know any of them, so we know even less about what you get when you multiply together the whole product. Since then, a lot of those factors have become much better known. One of his big uncertainties was how common is it that a solar system even has a planet? Well, now we know it's very common. Earth-like planets, we know we have better... There are a dime a dozen, there are many, many of them, even in our galaxy. At the same time, I'm a big supporter of the SETI project and its cousins, and I think we should keep doing this, and we've learned a lot. We've learned that so far, all we have is still unconvincing hints, nothing more. There are certainly many scenarios where it would be dead obvious if there were 100 million other human-like civilizations in our galaxy, it would not be that hard to notice some of them with today's technology, and we haven't. What we can say is, well, okay, we can rule out that there is a human-level civilization on the Moon, and in fact, in many nearby solar systems, where we cannot rule out, of course, that there is something like Earth sitting in a galaxy five billion light-years away. But we've ruled out a lot, and that's already kind of shocking given that there are all these planets there. So where are they? Where are they all? That's the classic Fermi paradox. So my argument, which might very be wrong, it's very simple, really, it just goes like this. Okay, we have no clue about this. It could be the probability of getting life on a random planet, it could be 10 to the minus one, a priori, or 10 to the minus 10, 10 to the minus 20, 10 to the minus 30, 10 to the minus 40. Basically, every order of magnitude is about equally likely. When you then do the math and ask how close is our nearest neighbor, it's again equally likely that it's 10 meters away, 10 to 20 meters away, 10 to the 30 meters away. We have some nerdy ways of talking about this with Bayesian statistics and a uniform log prior, but that's irrelevant. This is the simple basic argument. And now comes the data. So we can say, okay, how many were there? All these orders of magnitude, 10 to the 26 meters away, there's the edge of our observable universe. If it's farther than that, light hasn't even reached us yet. If it's less than 10 to the 16 meters away, well, it's within Earth's range. It's no farther away than the sun. We can definitely rule that out. So I think about it like this. A priori, before we looked with telescopes, it could be 10 to 10 meters, 10 to 20, 10 to 30, 10 to 40, 10 to 50, 10 to blah, blah, equally likely anywhere here. And now we've ruled out this chunk. Yeah. And most of it is outside. And here is the edge of our observable universe already. So I'm certainly not saying I don't think there's any life elsewhere in space. If space is infinite, then you're basically 100% guaranteed that there is. But the probability that there is life, that the nearest neighbor happens to be in this little region between where we would have seen it already and where we will never see it, there's actually significantly less than one, I think. And I think there's a moral lesson from this, which is really important, which is to be good stewards of this planet and this shot we've had. It can be very dangerous to say, oh, it's fine if we nuke our planet or ruin the climate or mess it up with unaligned AI, because I know there is this nice Star Trek fleet out there. They're going to swoop in and take over where we failed. Just like it wasn't the big deal that the Easter Island losers wiped themselves out. That's a dangerous way of lulling yourself into false sense of security. If it's actually the case that it might be up to us and only us, the whole future of intelligent life in our observable universe, then I think it really puts a lot of responsibility on our shoulders. It's inspiring. It's a little bit terrifying, but it's also inspiring. But it's empowering, I think, most of all, because the biggest problem today is, and I see this even when I teach, so many people feel that it doesn't matter what they do or what we do. We feel disempowered. Oh, it makes no difference. This is about as far from that as you can come. But we realize that what we do on our little spinning ball here in our lifetime could make the difference for the entire future of life in our universe. How empowering is that? Yes, survival of consciousness. A very similar kind of empowering aspect of the Drake equation is, say there is a huge number of intelligent civilizations that spring up everywhere, but because of the Drake equation, which is the lifetime of a civilization, maybe many of them hit a wall. And just like you said, it's clear that for us, the great filter, the one possible great filter seems to be coming in the next 100 years. So it's also empowering to say, okay, well, we have a chance to not. I mean, the way great filters work, it does just get most of them. Exactly. Nick Bostrom has articulated this really beautifully, too. Every time yet another search for life on Mars comes back negative or something, I'm like, yes, yes. Our odds for us surviving. You already made the argument in broad brush there, right? But just to unpack it. The point is, we already know there is a crap ton of planets out there that are Earth-like, and we also know that most of them do not seem to have anything like our kind of life on them. So what went wrong? There's clearly one step along the evolutionary, at least one, filter roadblock in going from no life to spacefaring life. And where is it? Is it in front of us or is it behind us? Right? If there's no filter behind us, and we keep finding all sorts of little mice on Mars or whatever, right? That's actually very depressing because that makes it much more likely that the filter is in front of us. And what actually is going on is the ultimate dark joke that whenever a civilization invents sufficiently powerful tech, it just sets your clock, and then after a little while it goes poof for one reason or another and wipes itself out. Wouldn't that be utterly depressing if we're actually doomed? Whereas if it turns out that there is a great filter early on, that for whatever reason seems to be really hard to get to the stage of sexually reproducing organisms or even the first ribosome or whatever, right? Or maybe you have lots of planets with dinosaurs and cows, but for some reason they tend to get stuck there and never invent smartphones. All of those are huge boosts for our own odds because been there, done that. You know, it doesn't matter how hard or unlikely it was that we got past that roadblock because we already did. And then that makes it likely that the future is in our own hands. We're not doomed. So that's why I think the fact that life is rare in the universe, it's not just something that there is some evidence for, but also something we should actually hope for. So that's the end, the mortality, the death of human civilization that we've been discussing in life, maybe prospering beyond any kind of great filter. Do you think about your own death? Does it make you sad that you may not witness some of the, you lead a research group on working some of the biggest questions in the universe actually, both on the physics and the AI side? Does it make you sad that you may not be able to see some of these exciting things come to fruition that we've been talking about? Of course. Of course it sucks the fact that I'm going to die. I remember once when I was much younger, my dad made this remark that life is fundamentally tragic. And I'm like, what are you talking about daddy? And then many years later, I felt, now I feel I totally understand what he means. You know, we grow up, we're little kids and everything is infinite and it's so complicated. Everything is infinite and it's so cool. And then suddenly we find out that actually, you know, you got to serve only this, this is the, you can get game over at some point. So of course it's, it's something that's sad. Are you afraid? No, not in the sense that I think anything terrible is going to happen after I die or anything like that. No, I think it's really going to be a game over, but it's more that it makes me very acutely aware of what a wonderful gift this is that it gets to be alive right now. And is a steady reminder to just live life to the fullest and really enjoy it because it is finite. You know, and I think actually, and we know we all get the regular reminders when someone near and dear to us dies that, that, you know, one day it's going to be our turn. It adds this kind of focus. I wonder what it would feel like actually to be an immortal being if they might even enjoy some of the wonderful things of life, live it less just because there isn't that. Finiteness. Do you think that could be a feature, not a bug, the fact that we, beings are finite. Maybe there's lessons for engineering and artificial intelligence systems as well that are conscious. Like, do you think it makes, is it possible that the reason the pistachio ice cream is delicious is the fact that you're going to die one day and you will not have all the pistachio ice cream that you could eat because of that fact. Well, let me say two things. First of all, it's actually quite profound what you're saying. I do think I appreciate the pistachio ice cream a lot more knowing that I will, there's only a finite number of times I get to enjoy that. And I can only remember finite number of times in the past. And moreover, my life is not so long that it just starts to feel like things are repeating themselves in general. It's so new and fresh. I also think though that death is a little bit overrated in the sense that it comes from a sort of outdated view of physics and what life actually is. Because if you ask, okay, what is it that's going to die exactly? What am I really? When I say it, I feel sad about the idea of myself dying. Am I really sad that this skin cell here is going to die? Of course not, because it's going to die next week anyway, and I'll grow a new one. And it's not any of my cells that I'm associating really with who I really am, nor is it any of my atoms or quarks or electrons. In fact, basically all of my atoms get replaced on a regular basis. So what is it that's really me from a more modern physics perspective? It's the information in processing Amy. That's my memories, that's my values, my dreams, my passion, my love. That's what's really fundamentally me. And frankly, not all of that will die when my body dies. Richard Feynman, for example, his body died of cancer, but many of his ideas that he felt made him very him actually live on. This is my own little personal tribute to Richard Feynman. I try to keep a little bit of him alive in myself. I've even quoted him today. Yeah, he almost came alive for a brief moment in this conversation. Yeah. And this honestly gives me some solace. When I work as a teacher, I feel if I can actually share a bit about myself, that my students feel worthy enough to copy and adopt as a part of things that they know or they believe or aspire to, now I live on also a little bit in them. Being a teacher is a little bit of what I... That's something also that contributes to making me a little teeny bit less mortal, because I'm not at least not all going to die all at once. And I find that a beautiful tribute to people we do not respect. If we can remember them and carry in us the things that we felt was the most awesome about them, then they live on. And I'm getting a bit emotional here, but it's a very beautiful idea you bring up there. I think we should stop this old fashioned materialism and just equate who we are with our quarks and electrons. There's no scientific basis for that really. And it's also very uninspiring. Now, if you look a little bit towards the future, one thing which really sucks about humans dying is that even though some of their teachings and memories and stories and ethics and so on will be copied by those around them, hopefully, a lot of it can't be copied and just dies with them, with their brain. And that really sucks. That's the fundamental reason why we find it so tragic when someone goes from having all this information there to the more just gone, ruined. With more post-biological intelligence, that's going to shift a lot. The only reason it's so hard to make a backup of your brain in its entirety is exactly because it wasn't built for that. If you have a future machine intelligence, there's no reason for why it has to die at all. If you want to copy it, whatever it is, into some other quark blob, you can copy not just some of it, but all of it. In that sense, you can get immortality because all the information can be copied out of any individual entity. It's not just mortality that will change if we get to more post-biological life. It's also, with that, very much the whole individualism we have now. The reason that we make such a big difference between me and you is exactly because we're a little bit limited in how much we can copy. I would just love to go like this and copy your Russian skills, Russian-speaking skills. Wouldn't it be awesome? But I can't. I have to actually work for years if I want to get better on it. But if we were robots, just copying and pasting freely, then that completely washes away the sense of what immortality is. And also individuality a little bit. Maybe we would feel much more collaborative with each other. You can give me your Russian and I'll give you whatever. Suddenly you can speak Swedish. Maybe that's a bad trade for you, but whatever else you want from my brain. There have been a lot of sci-fi stories about hive minds and so on where people, where experiences can be more broadly shared. And I think one, we don't, I don't pretend to know what it would feel like to be a super intelligent machine, but I'm quite confident that however it feels about mortality and individuality will be very, very different from how it is for us. Well, for us, mortality and finiteness seems to be pretty important at this particular moment. And so all good things must come to an end. Just like this conversation. I saw that coming. Sorry, this is the world's worst transition. I could talk to you forever. It's such a huge honor that you spend time with me. Honor is mine. Thank you so much for getting me essentially to start this podcast by doing the first conversation, making me realize falling in love with conversation in itself. And thank you so much for inspiring so many people in the world with your books, with your research, with your talking and with the other, like this ripple effect of friends, including Elon and everybody else that you inspire. So thank you so much for talking to me. Thank you. I feel so fortunate that you're doing this podcast and getting so many interesting voices out there into the ether and not just the five second sound bites, but so many of the interviews are what you do. You really let people go in into depth in a way which we sorely need in this day and age. And that I got to be number one, I feel super honored. Yeah, you started it. Thank you so much, Max. Thanks for listening to this conversation with Max Tegmark and thank you to our sponsors, The Jordan Harbinger Show, Four Sigmatic Mushroom Coffee, Better Help Online Therapy, and ExpressVPN. So the choice is wisdom, whether it's between sanity or privacy. Choose wisely, my friends. And if you wish, click the sponsor links below to get a discount and to support this podcast. And now let me leave you with some words from Max Tegmark. If consciousness is the way that information feels when it's processed in certain ways, then it must be substrate independent. It's only the structure of the matter. It's not the structure of the matter doing the information processing. Thank you for listening and hope to see you next time.
 The following is a conversation with Max Tegmark,
 his second time on the podcast.
 In fact, the previous conversation
 was episode number one of this very podcast.
 He is a physicist and artificial intelligence researcher
 at MIT, co-founder of the Future of Life Institute,
 and author of Life 3.0,
 Being Human in the Age of Artificial Intelligence.
 He's also the head of a bunch of other huge,
 fascinating projects,
 and has written a lot of different things
 that you should definitely check out.
 He has been one of the key humans
 who has been outspoken about long-term existential risks
 of AI and also its exciting possibilities
 and solutions to real-world problems,
 most recently at the intersection of AI and physics,
 and also in re-engineering the algorithms that divide us
 by controlling the information we see,
 and thereby creating bubbles
 and all other kinds of complex social phenomena
 that we see today.
 In general, he's one of the most passionate
 and brilliant people I have the fortune of knowing.
 I hope to talk to him many more times
 on this podcast in the future.
 Quick mention of our sponsors,
 the Jordan Harbinger Show,
 Four Sigmatic Mushroom Coffee,
 Better Help Online Therapy, and ExpressVPN.
 So the choice is wisdom, caffeine, sanity, or privacy.
 Choose wisely, my friends,
 and if you wish, click the sponsor links below
 to get a discount to support this podcast.
 As a side note, let me say that much of the researchers
 in the machine learning
 and artificial intelligence communities
 do not spend much time thinking deeply
 about existential risks of AI.
 Because our current algorithms are seen as useful but dumb,
 it's difficult to imagine how they may become destructive
 to the fabric of human civilization
 in the foreseeable future.
 I understand this mindset, but it's very troublesome.
 To me, this is both a dangerous and uninspiring perspective,
 reminiscent of a lobster sitting in a pot of lukewarm water
 that a minute ago was cold.
 I feel a kinship with this lobster.
 I believe that already the algorithms
 that drive our interaction on social media
 have an intelligence and power
 that far outstripped the intelligence and power
 of any one human being.
 Now really is the time to think about this,
 to define the trajectory of the interplay of technology
 and human beings in our society.
 I think that the future of human civilization
 very well may be at stake over this very question
 of the role of artificial intelligence in our society.
 If you enjoy this thing, subscribe on YouTube,
 review it on Apple Podcasts, follow on Spotify,
 support on Patreon, or connect with me
 on Twitter at Lex Friedman.
 And now here's my conversation with Max Tagmark.
 So people might not know this,
 but you were actually episode number one of this podcast
 just a couple of years ago, and now we're back.
 And it so happens that a lot of exciting things happened
 in both physics and artificial intelligence,
 both fields that you're super passionate about.
 Can we try to catch up to some of the exciting things
 happening in artificial intelligence,
 especially in the context of the way it's cracking open
 the different problems of the sciences?
 Yeah, I'd love to, especially now as we start 2021 here,
 it's a really fun time to think about
 what were the biggest breakthroughs in AI,
 not the ones necessarily that media wrote about,
 but that really matter.
 And what does that mean for our ability to do better science?
 What does it mean for our ability
 to help people around the world?
 And what does it mean for new problems
 that they could cause if we're not smart enough
 to avoid them?
 So, what did we learn basically from this?
 Yes, absolutely.
 So one of the amazing things you're part of
 is the AI Institute for Artificial Intelligence
 and Fundamental Interactions.
 What's up with this institute?
 What are you working on?
 What are you thinking about?
 The idea is something I'm very on fire with,
 which is basically AI meets physics.
 And it's been almost five years now
 since I shifted my own MIT research from physics
 to machine learning.
 And in the beginning, I noticed a lot of my colleagues,
 even though they were polite about it,
 well, I kind of, what is Max doing?
 What is this weird stuff?
 He's lost his mind.
 But then gradually, I, together with some colleagues,
 were able to persuade more and more of the other professors
 in our physics department to get interested in this.
 And now we got this amazing NSF center,
 so 20 million bucks for the next five years,
 MIT and a bunch of neighboring universities here also.
 And I noticed now those colleagues
 who were looking at me funny have stopped asking
 what the point is of this, because it's becoming more clear.
 And I really believe that, of course,
 AI can help physics a lot to do better physics,
 but physics can also help AI a lot,
 both by building better hardware.
 My colleague, Marin Soljadzic, for example,
 is working on an optical chip for much faster machine learning
 where the computation is done not by moving electrons around
 but by moving photons around,
 dramatically less energy use, faster, better.
 We can also help AI a lot, I think,
 by having a different set of tools
 and a different, maybe more audacious attitude.
 AI has, to a significant extent,
 been an engineering discipline
 where you're just trying to make things that work
 and being more interested in maybe selling them
 than in figuring out exactly how they work
 and proving theorems about that they will always work.
 Contrast that with physics.
 When Elon Musk sends a rocket to the International Space Station,
 they didn't just train with machine learning,
 oh, let's fire it a little bit left, more to the left,
 a bit more to the right, or that also missed, let's try here.
 No, we figured out Newton's laws of gravitation
 and other things
 and got a really deep fundamental understanding.
 And that's what gives us such confidence in rockets.
 And my vision is that in the future,
 all machine learning systems
 that actually have impact on people's lives
 will be understood at a really, really deep level, right?
 So we trust them, not because some sales rep told us to,
 but because they've earned our trust.
 We can, and really safety-critical things even prove
 that they will always do what we expect them to do.
 That's very much the physics mindset.
 So it's interesting, if you look at big,
 breakthroughs that have happened
 in machine learning this year,
 from dancing robots, it's pretty fantastic.
 And not just because it's cool,
 but if you just think about not that many years ago,
 this YouTube video at this DARPA challenge
 where the MIT robot comes out of the car and face plants,
 how far we've come in just a few years.
 Similarly, Alpha Fold 2,
 crushing the protein folding problem.
 We can talk more about implications
 for medical research and stuff,
 but hey, that's huge progress.
 You can look at GPT-3 that can spout off English texts,
 which sometimes really, really blows you away.
 You can look at DeepMind's Mu-0,
 which doesn't just kick our butt and go,
 and chess, and shogi, but also in all these Atari games.
 And you don't even have to teach it the rules now.
 What all of those have in common is,
 besides being powerful,
 is we don't fully understand how they work.
 And that's fine if it's just some dancing robots,
 and the worst thing that can happen is they face plant.
 Or if they're playing Go,
 and the worst thing that can happen is
 that they make a bad move and lose the game.
 It's less fine if they're playing
 and it's less fine if that's what's controlling
 your self-driving car or your nuclear power plant.
 And we've seen already that even though Hollywood
 had all these movies where they try to make us worry
 about the wrong things, like machines turning evil,
 the actual bad things that have happened with automation
 have not been machines turning evil.
 They've been caused by over-trust
 in things we didn't understand
 as well as we thought we did.
 Even very simple automated systems
 like what Boeing put into the 737 MAX,
 killed a lot of people.
 Was it that that little simple system was evil?
 Of course not, but we didn't understand it
 as well as we should have.
 And we trusted without understanding.
 Exactly.
 That's the over-trust.
 We didn't even understand that we didn't understand.
 The humility is really at the core of being a scientist.
 I think step one, if you wanna be a scientist,
 is don't ever fool yourself into thinking
 you understand things when you actually don't.
 That's probably good advice for humans in general.
 I think humility in general can do us good,
 but in science, it's so spectacular.
 Why did we have the wrong theory of gravity
 ever from Aristotle onward until like Elio's time?
 Why would we believe something so dumb
 as that if I throw this water bottle,
 it's gonna go up with constant speed
 until it realizes that its natural motion is down?
 Changes its mind.
 Because people just kind of assumed
 Aristotle was right, he's an authority, we understand that.
 Why did we believe things like
 that the sun is going around the earth?
 Why did we believe that time flows at the same rate
 for everyone until Einstein?
 Same exact mistake over and over again.
 We just weren't humble enough to acknowledge
 that we actually didn't know for sure.
 We assumed we knew, so we didn't discover the truth
 because we assumed there was nothing there to be discovered.
 There was something to be discovered about the 737 max.
 And if you had been a bit more suspicious
 and tested it better, we would have found it.
 And it's the same thing with most harm
 that's been done by automation so far, I would say.
 So I don't know if you,
 did you hear of a company called Knight Capital?
 No.
 So good, that means you didn't invest in them earlier.
 They deployed this automated rating system.
 Yes.
 All nice and shiny.
 They didn't understand it as well as they thought.
 And it went about losing 10 million bucks per minute
 for 44 minutes straight.
 No.
 Until someone presumably was like,
 oh no, shut this up.
 Was it evil?
 No, it was again, misplaced trust,
 something they didn't fully understand.
 And there have been so many,
 even when people have been killed by robots,
 it's just quite rare still, but in factory accidents,
 it's in every single case been not malice,
 just that the robot didn't understand
 that a human is different from an auto part or whatever.
 And we, so this is where I think there's so much opportunity
 for a physics approach,
 where you just aim for a higher level of understanding.
 And if you look at the, all these systems
 that we talked about from reinforcement learning systems
 and dancing robots to all these neural networks,
 that power GPT-3 and go playing software.
 They're all basically black boxes,
 much like not so different from if you teach a human
 something, you have no idea how their brain works, right?
 Except the human brain at least has been error corrected
 during many, many centuries of evolution
 in a way that these, some of these systems have not, right?
 And my MIT research is entirely focused
 on demystifying this black box,
 intelligible intelligence is my slogan.
 That's a good line, intelligible intelligence.
 Yeah, that we shouldn't settle
 for something that seems intelligent,
 but it should be intelligible
 so that we actually trust it because we understand it, right?
 Like again, Elon trusts his rockets
 because he understands Newton's laws
 and thrust and everything works.
 And let me tell you what,
 can I tell you why I'm optimistic about this?
 Yes.
 I think there is, we've made a bit of a mistake
 where we, some people still think
 that somehow we're never gonna understand neural networks
 and we're just gonna have to learn to live with this.
 It's this very powerful black box.
 Basically for those who haven't spent time
 building their own, it's super simple what happens inside.
 You send in a long list of numbers
 and then you do a bunch of operations on them,
 multiply by matrices, et cetera, et cetera,
 and some other numbers come out, that's the output of it.
 And then there are a bunch of knobs you can tune
 and when you change them, it affects the computation,
 the input output relation.
 And then you just give the computer some definition of good
 and it keeps optimizing these knobs
 until it performs as good as possible.
 And often you go like, wow, that's really good.
 This robot can dance
 or this machine is beating me at chess now.
 And then the end you have something
 which even though you can look inside it,
 you have very little idea of how it works.
 You can print out tables
 of all the millions of parameters in there.
 Is it crystal clear now how it's working?
 And of course not, right?
 Many of my colleagues seem willing to settle for that.
 And I'm like, no, that's like the halfway point.
 Some have even gone as far as sort of guessing
 that the mist or the instruitability of this
 is where some of the power comes from
 and sort of some sort of mysticism.
 I think that's total nonsense.
 I think the real power of neural networks
 comes not from instruitability, but from differentiability.
 And what I mean by that is simply that
 the output depends, changes only smoothly
 if you tweak your knobs,
 and then you can use all these powerful methods we have
 for optimization in science.
 We can just tweak them a little bit
 and see did that get better or worse?
 That's the fundamental idea of machine learning,
 that the machine itself can keep optimizing
 until it gets better.
 Suppose you wrote this algorithm instead in Python
 or some other programming language,
 and then what the knobs did was they just changed
 random letters in your code.
 Now we're just epically fail.
 You change one thing and instead of saying print,
 it says Psynt, syntax error.
 You don't even know,
 was that for the better or for the worse?
 This to me is, this is what I believe
 is the fundamental power of neural networks.
 And just to clarify,
 the changing of the different letters in a program
 would not be a differentiable process.
 It would make it an invalid program typically,
 and then you wouldn't even know if you changed more letters
 if it would make it work again.
 So that's the magic of neural networks, the inscrutability.
 The differentiability,
 that every setting of the parameters is a program
 and you can tell is it better or worse.
 So you don't like the poetry of the mystery
 of neural networks as the source of its power?
 I generally like poetry, but-
 Not in this case.
 It's so misleading.
 And above all, it shortchanges us.
 It makes us underestimate
 the good things we can accomplish.
 So what we've been doing in my group is basically step one,
 train the mysterious neural network to do something well.
 And then step two, do some additional AI techniques
 to see if we can now transform this black box
 into something equally intelligent
 that you can actually understand.
 So for example, I'll give you one example,
 this AI Feynman project that we just published.
 So we took the 100 most famous or complicated equations
 from one of my favorite physics textbooks.
 In fact, the one that got me into physics
 in the first place, the Feynman lectures on physics.
 And so you have a formula,
 maybe it has what goes into the formula
 as six different variables,
 and then what comes out as one.
 So then you can make like a giant Excel spreadsheet
 with seven columns.
 You put in just random numbers for the six columns
 for those six input variables,
 and then you calculate with a formula,
 the seventh column, the output.
 So maybe it's like the force equals in the last column,
 some function of the other.
 And now the task is, okay,
 if I don't tell you what the formula was,
 can you figure that out
 from looking at my spreadsheet I gave you?
 Yes.
 This problem is called symbolic regression.
 If I tell you that the formula
 is what we call a linear formula.
 So it's just that the output is
 some sum of all the things, input,
 the times, some constants.
 That's the famous easy problem we can solve.
 We do it all the time in science and engineering.
 But the general one, if it's more complicated functions
 with logarithms or cosines or other math,
 it's a very, very hard one
 and probably impossible to do fast in general,
 just because the number of formulas with N symbols
 just grows exponentially.
 Just like the number of passwords you can make
 grow dramatically with length.
 So we had this idea that if you first have a neural network
 that can actually approximate the formula,
 you just trained it,
 even if you don't understand how it works,
 that can be a first step
 towards actually understanding how it works.
 So that's what we do first.
 And then we study that neural network now
 and put in all sorts of other data
 that wasn't in the original training data
 and use that to discover simplifying properties
 of the formula.
 And that lets us break it apart often
 into many simpler pieces
 and a kind of divide and conquer approach.
 So we were able to solve all of those 100 formulas,
 discover them automatically,
 plus a whole bunch of other ones.
 And it's actually kind of humbling to see
 that this code, which anyone who wants now
 who's listening to this can type pip install AI Feynman
 on their computer and run it.
 It can actually do what Johannes Kepler spent four years doing
 when he stared at Mars data
 until he's like, finally, Eureka, this is an ellipse.
 This will do it automatically for you in one hour.
 Or Max Planck, he was looking at how much radiation
 comes out at different wavelengths from a hot object
 and discovered the famous black body formula.
 This discovers it automatically.
 I'm actually excited about seeing if we can discover
 not just old formulas again,
 but new formulas that no one has seen before.
 I do like this process of using kind of a neural network
 to find some basic insights
 and then dissecting the neural network
 to then gain the final.
 So that's in that way you've forcing
 the explainability issue,
 really trying to analyze in your own network
 for the things it knows in order to come up
 with the final, beautiful, simple theory
 underlying the whole, the initial system
 that you were looking at.
 I love that.
 And the reason I'm so optimistic
 that it can be generalized so much more
 is because that's exactly what we do as human scientists.
 Think of Galileo whom we mentioned, right?
 I bet when he was a little kid,
 if his dad threw him an apple, he would catch it.
 Why?
 Because he had a neural network in his brain
 that he had trained to predict the parabolic orbit
 of apples that are thrown under gravity.
 If you throw a tennis ball to a dog,
 it also has this same ability of deep learning
 to figure out how the ball is gonna move and catch it.
 But Galileo went one step further when he got older.
 He went back and was like, wait a minute.
 I can write down a formula for this.
 Y equals X squared, a parabola.
 And he helped revolutionize physics as we know it, right?
 So there was a basic neural network in there
 from childhood that captured the experiences
 of observing different kinds of trajectories.
 And then he was able to go back in
 with another extra little neural network
 and analyze all those experiences
 and be like, wait a minute, there's a deeper rule here.
 Exactly, he was able to distill out in symbolic form
 what that complicated black box neural network was doing.
 Not only did the formula he got
 ultimately become more accurate,
 and similarly this is how Newton got Newton's laws,
 which is why Elon can send rockets to the space station now.
 So it's not only more accurate, but it's also simpler,
 much simpler, and it's so simple
 that we can actually describe it to our friends
 and each other, right?
 We've talked about it just in the context of physics now,
 but hey, isn't this what we're doing
 when we're talking to each other also?
 We go around with our neural networks,
 just like dogs and cats and chipmunks and blue jays,
 and we experience things in the world.
 But then we humans do this additional step on top of that
 where we then distill out certain high-level knowledge
 that we've extracted from this
 in a way that we can communicate it to each other
 in a symbolic form, in English in this case, right?
 So if we can do it, and we believe
 that we are information processing entities,
 then we should be able to make machine learning
 that does it also.
 Well, do you think the entire thing could be learning,
 because this dissection process for AI Feynman,
 the secondary stage feels like something like reasoning,
 and the initial step feels like more like
 the more basic kind of differentiable learning.
 Do you think the whole thing
 could be differentiable learning?
 Do you think the whole thing could be
 basically neural networks on top of each other?
 It's like turtles all the way down.
 Can it be neural networks all the way down?
 I mean, that's a really interesting question.
 We know that in your case,
 it is neural networks all the way down,
 because that's all you have in your skull
 as a bunch of neurons.
 Doing their thing, right?
 But if you ask the question more generally,
 what algorithms are being used in your brain?
 I think super interesting to compare.
 I think we've gotten a little bit backwards historically,
 because we humans first discovered good old fashioned AI,
 the logic based AI that we often call GoFi
 for good old fashioned AI.
 And then more recently, we did machine learning,
 because it required bigger computers.
 So we had to discover it later.
 So we think of machine learning with neural networks
 as the modern thing and the logic based AI
 as the old fashioned thing.
 But if you look at evolution on earth, right?
 It's actually been the other way around.
 I would say that, for example,
 an eagle has a better vision system that I have using,
 and dogs are just as good at casting tennis balls
 as I am.
 All this stuff which is done by training in neural network
 and not interpreting it in words,
 is something so many of our animal friends can do,
 at least as well as us, right?
 What is it that we humans can do
 that the chipmunks and the eagles cannot?
 It's more to do with this logic based stuff, right?
 Where we can extract out information in symbols,
 in language, and now even with equations,
 if you're a scientist, right?
 So basically what happened was,
 first we built these computers
 that could multiply numbers real fast
 and manipulate symbols,
 and we felt they were pretty dumb.
 And then we made neural networks
 that can see as well as a cat can
 and do a lot of this inscrutable black box neural networks.
 What we humans can do also
 is put the two together in a useful way.
 Yes, in our own brain.
 Yes, in our own brain.
 So if we ever wanna get artificial general intelligence
 that can do all jobs as well as humans can, right?
 Then that's what's gonna be required,
 to be able to combine the neural networks with symbolic,
 combine the old AI with the new AI in a good way.
 We do it in our brains,
 and there seems to be basically two strategies
 I see in industry now.
 One scares the heebie-jeebies out of me,
 and the other one I find much more encouraging.
 Okay, which one?
 Can we break them apart?
 Which of the two?
 The one that scares the heebie-jeebies out of me
 is this attitude that we're just gonna make
 ever bigger systems that we still don't understand
 until they can be as smart as humans.
 What could possibly go wrong?
 I think it's just such a reckless thing to do.
 And unfortunately, and if we actually succeed as a species
 to build artificial general intelligence
 then we still have no clue how it works.
 I think at least 50% chance
 we're gonna be extinct before too long.
 It's just gonna be an utter epic own goal.
 So it's that 44 minute losing money problem
 or the paperclip problem
 like where we don't understand how it works
 and it just in a matter of seconds runs away
 in some kind of direction
 that's going to be very problematic.
 Even long before you have to worry
 about the machines themselves
 somehow deciding to do things.
 And to us, we have to worry about people using machines.
 Machines that are short of AI, AGI and power to do bad things.
 I mean, just take a moment.
 And if anyone who's not worried particularly
 about advanced AI, just take 10 seconds
 and just think about your least favorite leader
 on the planet right now.
 Don't tell me who it is.
 I wanna keep this apolitical,
 but just see the face in front of you,
 that person for 10 seconds.
 Now imagine that that person has
 this incredibly powerful AI under their control
 and can use it to impose their will on the whole planet.
 How does that make you feel?
 Yeah.
 So can we break that apart just briefly
 for the 50% chance that we'll run to trouble
 with this approach.
 Do you see the bigger worry in that leader
 or humans using the system to do damage
 or are you more worried?
 And I think I'm in this camp,
 more worried about like accidental,
 unintentional destruction of everything.
 So like humans trying to do good
 and like in a way where everyone agrees it's kind of good,
 it's just that they're trying to do good
 without understanding.
 Cause I think every evil leader in history
 thought they're to some degree
 thought they're trying to do good.
 Oh yeah, I'm sure Hitler thought he was doing good.
 He was trying to do good too.
 I've been reading a lot about Stalin.
 I'm sure Stalin is from,
 he legitimately thought that communism was good
 for the world and that he was doing good.
 I think Mao Zedong thought what he was doing
 with a great leap forward was good too.
 Yeah.
 I'm actually concerned about both of those.
 Before I promised to answer this in detail,
 but before we do that,
 let me finish answering the first question.
 Cause I told you that there were two different groups
 we could get to artificial general intelligence
 and one scares the GBBs out of me,
 which is this one where we build something.
 We just say bigger neural networks, ever more hardware
 and it's just training the heck out of more data
 and poof, now it's very powerful.
 That I think is the most unsafe and reckless approach.
 The alternative to that is the intelligent,
 intelligible intelligence approach instead,
 where we say neural networks is just a tool
 to like for the first step to get the intuition,
 but then we're going to spend also serious resources
 sources on other AI techniques for demystifying
 this black box and figuring out what it's actually doing.
 So we can convert it into something
 that's equally intelligent,
 but that we actually understand what it's doing.
 Maybe we can even prove theorems about it.
 That this car here will never be hacked when it's driving
 because here's the proof.
 There is a whole science of this,
 but it doesn't work for neural networks
 that are big black boxes, but it works well
 and work to turn other kinds of codes.
 That approach I think is much more promising.
 That's exactly why I'm working on it, frankly,
 not just because I think it's cool for science,
 but because I think the more we understand these systems,
 the better the chances that we can make them do the things
 that are good for us that are actually intended,
 not unintended.
 So you think it's possible to prove things about something
 as complicated as a neural network?
 That's the hope?
 Well, ideally there's no reason there has to be
 a neural network in the end either.
 We discovered that Newton's laws of gravity
 with neural network in Newton's head,
 but that's not the way it's programmed
 into the navigation system of Elon Musk's rocket anymore.
 It's written in C++,
 or I don't know what language he uses exactly.
 And then there are software tools
 called symbolic verification.
 DARPA and the US military has done a lot
 of really great research on this
 because they really want to understand
 that when they build weapons systems,
 they don't just go fire at random or malfunction, right?
 And there's even an whole operating system called Cell 3
 that's been developed by a DARPA grant
 where you can actually mathematically prove
 that this thing can never be hacked.
 Wow.
 One day I hope that will be something you can say
 about the OS that's running on our laptops too.
 As you know, we're not there,
 but I think we should be ambitious, frankly.
 And if we can use machine learning
 to help do the proofs and so on as well, right?
 Then it's much easier to verify that a proof is correct
 than to come up with a proof in the first place.
 That's really the core idea here.
 If someone comes on your podcast
 and says they proved the Riemann hypothesis
 or some sensational new theorem,
 it's much easier for someone else
 to take some smart math grad students and check,
 oh, there's an error here in equation five,
 or this really checks out than it was to discover the proof.
 Yeah, although some of those proofs are pretty complicated,
 but yes, it's still nevertheless much easier
 to verify the proof.
 I love the optimism.
 Even with the security of systems,
 there's a kind of cynicism that pervades
 people who think about this, which is like,
 well, it's hopeless.
 I mean, in the same sense, exactly like you're saying,
 when you don't know what works,
 oh, it's hopeless to understand what's happening.
 With security, people are just like,
 well, there's always going to be attack vectors
 and waste to attack the system.
 But you're right, we're just very new
 with these computational systems.
 We're even newer with these intelligent systems.
 And it's not out of the realm of possible,
 just like people that understand the movement
 of the stars and the planets and so on.
 It's entirely possible that within, hopefully soon,
 but it could be within 100 years,
 we start to have an obvious laws of gravity
 about intelligence and God forbid about consciousness too.
 That one is.
 Agreed, I think, of course, if you're selling computers
 that get hacked a lot, that's in your interest
 as a company that people think it's impossible
 to make it safe.
 So nobody's going to get the idea of suing you.
 But I want to really inject optimism here.
 It's absolutely possible to do much better
 than we're doing now.
 And your laptop does so much stuff.
 You don't need the music player to be super safe
 in your future self-driving car.
 If someone hacks it and starts playing music
 you don't like, the world won't end.
 But what you can do is you can break out
 and say the drive computer that controls your safety
 must be completely physically decoupled entirely
 from the entertainment system.
 And it must physically be such that it can't take on
 over the air updates while you're driving.
 And it can be, it can have, it's not that,
 it can have ultimately some operating system on it
 which is symbolically verified and proven
 that it's always going to do what it's supposed to do.
 We can basically have, and companies should take
 that attitude too.
 They should look at everything they do and say,
 what are the few systems in our company
 that threaten the whole life of the company
 if they get hacked, you know,
 and have the highest standards for them.
 And then they can save money by going for the El Cheapo
 poorly understood stuff for the rest, you know.
 This is very feasible, I think.
 And coming back to the bigger question
 that you worried about that there'll be unintentional
 failures, I think there are two quite separate risks here.
 We talked a lot about one of them,
 which is that the goals are noble of the human.
 The human says, I want this airplane to not crash
 because this is not Mohammed Atta now flying the airplane.
 And now there's this technical challenge of making sure
 that the autopilot is actually going to behave
 as the pilot wants.
 If we set that aside, there's also the separate question.
 How do you make sure that the goals of the pilot
 are actually aligned with the goals of the passenger?
 How do you make sure very much more broadly
 that if we can all agree as a species
 that we would like things to kind of go well
 for humanity as a whole, that the goals are aligned here,
 the alignment problem.
 And yeah, there's been a lot of progress
 in the sense that there's suddenly huge amounts
 of research going on on it about it.
 I'm very grateful to Elon Musk for giving us that money
 five years ago so we could launch the first research program
 on technical AI safety and alignment.
 There's a lot of stuff happening.
 But I think we need to do more than just make sure
 little machines do always what their owners do.
 That wouldn't have prevented September 11
 if Mahmoud Abbas said, okay, autopilot,
 please fly into World Trade Center.
 And it's like, okay.
 That even happened in a different situation.
 There was this depressed pilot named Andreas Lubitz
 who told his German wings passenger jet
 to fly into the Alps.
 He just told the computer to change the altitude
 to a hundred meters or something like that.
 And you know what the computer said?
 Okay.
 And it had the frigging topographical map of the Alps
 in there, it had GPS, everything.
 No one had bothered teaching it even the basic
 kindergarten ethics of like, no, we never want airplanes
 to fly into mountains under any circumstances.
 And so we have to think beyond just the technical issues
 and think about how do we align in general incentives
 on this planet for the greater good?
 So starting with simple stuff like that,
 every airplane that has a computer in it
 should be taught whatever kindergarten ethics
 that's smart enough to understand.
 Like, no, don't fly into fixed objects
 if the pilot tells you to do so,
 then go on autopilot mode, send an email to the cops
 and land at the latest airport, nearest airport.
 Any car with a forward facing camera should just be programmed
 by the manufacturer so that it will never accelerate
 into a human ever.
 That would avoid things like the Nice attack
 and many horrible terrorist vehicle attacks
 where they deliberately did that, right?
 This was not some sort of thing, oh, you know,
 US and China, different views on, no,
 there was not a single car manufacturer in the world
 who wanted the cars to do this.
 They just hadn't thought to do the alignment.
 And if you look at more broadly problems that happen
 on this planet, the vast majority have to do
 with poor alignment.
 I mean, think about, let's go back really big
 because I know you're so good at that.
 Yeah, in the very, so long ago in evolution,
 we had these genes and they wanted to make copies
 of themselves, that's really all they cared about.
 So some genes said, hey, I'm gonna build a brain
 on this body I'm in so that I can get better
 at making copies of myself.
 And then they decided for their benefit to get copied more,
 align your brain's incentives with their incentives.
 So it didn't want you to starve to death.
 So it gave you an incentive to eat
 and it wanted you to make copies of the genes.
 So it gave you incentive to fall in love
 and do all sorts of naughty things
 to make copies of itself, right?
 So that was successful value alignment done on the genes.
 They created something more intelligent than themselves
 but they made sure to try to align the values.
 But then something went a little bit against the idea
 of what the genes wanted because a lot of humans discovered,
 hey, you know, yeah, we really like this business
 about sex that the genes have made us enjoy
 but we don't wanna have babies right now.
 So we're gonna hack the genes and use birth control.
 And I really feel like drinking a Coca-Cola right now
 but I don't wanna get a pot belly
 so I'm gonna drink Diet Coke.
 We have all these things we've figured out
 because we're smarter than the genes
 how we can actually subvert their intentions.
 So it's not surprising that we humans now
 when we're in the role of these genes
 creating other non-human entities with a lot of power
 have to face the same exact challenge.
 How do we make other powerful entities
 have incentives that are aligned with ours
 and that so they won't hack them?
 Corporations, for example, right?
 We humans decided to create corporations
 because it can benefit us greatly.
 Now all of a sudden there's a supermarket.
 I can go buy food there.
 I don't have to hunt.
 Awesome.
 And then to make sure that this corporation
 would do things that were good for us and not bad for us,
 we created institutions to keep them in check.
 Like if the local supermarket sells poisonous food,
 then the owners of a supermarket
 have to spend some years reflecting behind bars, right?
 So we created incentives to align them.
 But of course, just like we were able to see
 through this thing, birth control,
 if you're a powerful corporation,
 you also have an incentive to try to hack the institutions
 that are supposed to govern you
 because you ultimately as a corporation
 have an incentive to maximize your profits.
 It's like you have an incentive
 to maximize the enjoyment your brain has,
 not for your genes.
 So if they can figure out a way of bribing regulators,
 then they're gonna do that.
 In the US, we kind of caught onto that
 and made laws against corruption and bribery.
 Then in the late 1800s, Teddy Roosevelt realized that,
 no, we were still being kind of hacked
 because the Massachusetts railroad companies
 had like a bigger budget than the state of Massachusetts
 and they were doing a lot of very corrupt stuff.
 So he did the whole trust busting thing
 to try to align these other non-human entities,
 the companies again, more with the incentives
 of Americans as a whole.
 It's not surprising though that this is a battle
 you have to keep fighting.
 Now we have even larger companies than we ever had before.
 And of course, they're gonna try to again,
 subvert the institutions, not because,
 I think people make a mistake of getting all too
 black thinking about things in terms of good and evil,
 like arguing about whether corporations are good or evil
 or whether robots are good or evil.
 A robot isn't good or evil, it's tool.
 And you can use it for great things like robotic surgery
 or for bad things.
 And a corporation also is a tool of course.
 And if you have good incentives to the corporation,
 it'll do great things like start a hospital
 or a grocery store.
 If you have really bad incentives, then it's gonna start
 maybe marketing addictive drugs to people
 and you'll have an opioid epidemic, right?
 It's all about, I don't want, we should not make the mistake
 of getting into some sort of fairy tale, good evil thing
 about corporations or robots.
 We should focus on putting the right incentives in place.
 My optimistic vision is that if we can do that,
 then we can really get good things.
 We're not doing so great with that right now,
 either on AI, I think, or on other intelligent
 non-human entities like big companies, right?
 We just have a new secretary of defense
 who's gonna start up now in the Biden administration
 who is, was an active member of the board of Raytheon,
 for example.
 So, I have nothing against Raytheon.
 I'm not a pacifist, but there's an obvious conflict
 of interest if someone is in the job where they decide
 who's they're gonna contract with.
 And I think somehow we have, maybe we need another
 Teddy Roosevelt to come along again and say,
 hey, we want what's good for all Americans
 and we need to go do some serious realigning again
 of the incentives that we're giving to these big companies.
 And then we're gonna be better off.
 It seems that naturally with human beings,
 just like you beautifully described the history
 of this whole thing, it all started with the genes
 and they're probably pretty upset
 by all the unintended consequences that happened since.
 But it seems that it kind of works out.
 Like it's in this collective intelligence
 that emerges at the different levels,
 it seems to find sometimes last minute
 a way to realign the values or keep the values aligned.
 It's almost, it finds a way, like different leaders,
 different humans pop up all over the place
 that reset the system.
 Do you want, I mean, do you have an explanation
 why that is or is that just survivor bias?
 And also is that different, somehow fundamentally different
 than with AI systems where you're no longer dealing
 with something that was a direct,
 maybe companies are the same, a direct byproduct
 of the evolutionary process.
 I think there is one thing which has changed.
 That's why I'm not all optimistic.
 That's why I think there's about a 50% chance
 if we take the dumb route with artificial intelligence
 that we will, humanity will be extinct in this century.
 First, just the big picture, yeah, companies need
 to have right incentives, even governments, right?
 We used to have governments,
 usually there were just some king,
 who was the king because his dad was the king.
 And then there were some benefits
 of having this powerful kingdom or empire of any sort,
 because then it could prevent a lot of local squabbles.
 So at least everybody in that region
 would stop warring against each other
 and their incentives of different cities
 in the kingdom became more aligned, right?
 That was the whole selling point.
 Harari has a beautiful piece on how empires
 were collaboration enablers.
 And then we also, Harari says invented money for that reason
 so we could have better alignment
 and we could do trade even with people we didn't know.
 So this sort of stuff has been playing out
 since time immemorial, right?
 What's changed is that it happens on ever larger scales,
 right, the technology keeps getting better
 because science gets better.
 So now we can communicate over larger distances,
 transport things faster over larger distances.
 And so the entities get ever bigger,
 but our planet is not getting bigger anymore.
 So in the past, you could have one experiment
 that just totally screwed up like Easter Island,
 where they actually managed to have such poor alignment
 that when they went extinct, people there,
 there was no one else to come back and replace them, right?
 If Elon Musk doesn't get us to Mars
 and then we go extinct on a global scale
 and then we're not coming back,
 that's the fundamental difference.
 And that's a mistake we don't make for that reason.
 In the past, of course, history is full of fiascos,
 but it was never the whole planet.
 And then, okay, now there's this nice uninhabited land here
 some other people could move in and organize things better.
 This is different.
 The second thing which is also different
 is that technology gives us so much more empowerment,
 both to do good things and also to screw up.
 In the Stone Age, even if you had someone whose goals
 were really poorly aligned,
 maybe he was really pissed off
 because his Stone Age girlfriend dumped him
 and he just wanted to,
 if he wanted to kill as many people as he could,
 how many could he really take out with a rock and a stick
 before he was overpowered, right?
 Just handful, right?
 Now, with today's technology,
 if we have an accidental nuclear war
 between Russia and the US,
 which we almost have about a dozen times,
 and then we have a nuclear winter,
 it could take out 7 billion people
 or 6 billion people or we don't know.
 So the scale of the damage is bigger that we can do.
 And if, there's obviously no law of physics that says
 that technology will never get powerful enough
 that we could wipe out our species entirely,
 that would just be fantasy to think that science
 is somehow doomed to not get more powerful than that, right?
 And it's not at all unfeasible in our lifetime
 that someone could design a designer pandemic
 which spreads as easily as COVID
 but just basically kills everybody.
 We already had smallpox.
 It killed one third of everybody who got it.
 What do you think of the, here's an intuition,
 maybe it's completely naive and this optimistic intuition
 I have, which it seems,
 and maybe it's a biased experience that I have,
 but it seems like the most brilliant people
 I've met in my life all are really like
 fundamentally good human beings.
 And not like naive, good,
 like they really wanna do good for the world
 in a way that, well, maybe is aligned to my sense
 of what good means.
 And so I have a sense that the people
 that will be defining the very cutting edge of technology,
 there'll be much more of the ones that are doing good
 versus the ones that are doing evil.
 So the race, I'm optimistic on the us always
 like last minute coming up with a solution.
 So if there's an engineered pandemic
 that has the capability to destroy
 most of the human civilization,
 it feels like to me, either leading up to that before
 or as it's going on, there will be,
 we're able to rally the collective genius
 of the human species.
 I can tell by your smile that you're
 at least some percentage doubtful.
 But could that be a fundamental law of human nature?
 That evolution only creates,
 like karma is beneficial, good is beneficial
 and therefore we'll be all right.
 I hope you're right.
 I really would love it if you're right,
 if there's some sort of law of nature that says
 that we always get lucky in the last second
 because of karma.
 But I prefer not playing it so close and gambling on that.
 And I think, in fact, I think it can be dangerous
 to have too strong faith in that
 because it makes us complacent.
 Like if someone tells you,
 you never have to worry about your house burning down,
 then you're not gonna put in a smoke detector
 because why would you need to?
 Even if it's sometimes very simple precautions,
 we don't take them.
 If you're like, oh, the government is gonna take care
 of everything for us, I can always trust my politicians.
 We have to get our own responsibility.
 I think it's a healthier attitude to say,
 yeah, maybe things will work out,
 maybe I'm actually gonna have to myself step up
 and take responsibility and the stakes are so huge.
 I mean, if we do this right,
 we can develop all this ever more powerful technology
 and cure all diseases and create a future
 where humanity is healthy and wealthy
 for not just the next election cycle,
 but like billions of years throughout our universe.
 That's really worth working hard for
 and not just sitting and hoping
 for some sort of fairy tale karma.
 Well, I just mean, so you're absolutely right.
 From the perspective of the individual,
 like for me, the primary thing should be
 to take responsibility and to build the solutions
 that your skillset allows.
 Yeah, which is a lot.
 I think we underestimate often very much
 how much good we can do.
 If you or anyone listening to this
 is completely confident that our government
 would do a perfect job on handling any future crisis
 with engineered pandemics or future AI,
 I ask you to reflect a bit on
 what actually happened in 2020.
 Do you feel that the government by and large
 around the world has handled this flawlessly?
 That's a really sad and disappointing reality
 that hopefully is a wake up call for everybody,
 for the scientists, for the engineers,
 for the researchers in AI especially.
 It was disappointing to see how inefficient we were
 at collecting the right amount of data
 in a privacy preserving way and spreading that data
 and utilizing that data to make decisions,
 all that kind of stuff.
 Yeah, I think when something bad happens to me,
 I made myself a promise many years ago
 that I would not be a whiner.
 So when something bad happens to me,
 of course it's a process to disappointment,
 but then I try to focus on what did I learn from this
 that can make me a better person in the future.
 And there's usually something to be learned when I fail.
 And I think we should all ask ourselves,
 what can we learn from the pandemic
 about how we can do better in the future?
 And you mentioned there's a really good lesson.
 We were not as resilient as we thought we were
 and we were not as prepared maybe as we wish we were.
 You can even see very stark contrast around the planet.
 South Korea, they have over 50 million people.
 Do you know how many deaths they have from COVID
 last time I checked?
 No.
 About 500.
 Why is that?
 Well, the short answer is that they had prepared.
 They were incredibly quick, incredibly quick to get on it
 with very rapid testing and contact tracing and so on,
 which is why they never had more cases
 than they could contract trace effectively.
 They never even had to have the kind of big lockdowns
 we had in the West.
 But the deeper answer to it,
 it's not just the Koreans are just somehow better people.
 The reason I think they were better prepared
 was because they had already had a pretty bad hit
 from the SARS pandemic, which never became a pandemic,
 something like 17 years ago, I think.
 So it was kind of a fresh memory that,
 we need to be prepared for pandemics.
 So they were, right?
 And so maybe this is a lesson here
 for all of us to draw from COVID
 that rather than just wait for the next pandemic
 or the next problem with AI getting out of control
 or anything else, maybe we should just actually set aside
 a tiny fraction of our GDP to have people very systematically
 do some horizon scanning and say,
 okay, what are the things that could go wrong?
 And let's do get out and see which are the more likely ones
 and which are the ones that are actually actionable
 and then be prepared.
 So one of the observations as one little ant slash human
 that I am of disappointment is the political division
 over information that has been observed,
 that I observed this year,
 that it seemed the discussion was less about sort of,
 what happened and understanding what happened deeply
 and more about there's different truths out there.
 And it's like an argument, my truth is better
 than your truth.
 And it's like red versus blue or different,
 it was like this ridiculous discourse
 that doesn't seem to get at any kind of notion of the truth.
 It's not like some kind of scientific process,
 even science got politicized in ways
 that's very heartbreaking to me.
 You have an exciting project on the AI front
 of trying to rethink one of the, you mentioned corporations,
 there's one of the other collective intelligence systems
 that have emerged through all of this is social networks.
 And just the spread, the internet is the spread
 of information on the internet, our ability
 to share that information, there's all different kinds
 of news sources and so on.
 And so you said like, that's from first principles,
 let's rethink how we think about the news,
 how we think about information.
 Can you talk about this amazing effort
 that you're undertaking?
 Oh, I'd love to, this has been my big COVID project
 since nights and weekends on ever since the lockdown.
 To segue into this actually, let me come back
 to what you said earlier, that you had this hope
 that in your experience, people who you felt
 were very talented, often idealistic and wanted to do good.
 Frankly, I feel the same about all people by and large,
 there are always exceptions, but I think the vast majority
 of everybody, regardless of education and whatnot,
 really are fundamentally good, right?
 So how can it be that people still do
 so much nasty stuff, right?
 I think it has everything to do with the information
 that we're given.
 If you go into Sweden 500 years ago
 and you start telling all the farmers that those Danes
 in Denmark, they're so terrible people,
 and we have to invade them because they've done
 all these terrible things that you can't fact check yourself.
 A lot of people, Swedes did that, right?
 And we're seeing so much of this today in the world,
 both geopolitically, where we are told that China is bad
 and Russia is bad and Venezuela is bad,
 and people in those countries are often told
 that we are bad, and we also see it at a micro level,
 where people are told that, oh, those who voted
 for the other party are bad people.
 It's not just an intellectual disagreement,
 but they're bad people, and we're getting ever more divided.
 So how do you reconcile this with the fact
 that this intrinsic goodness in people?
 I think it's pretty obvious that it has again to do
 with the information that we're fed and given.
 We evolved to live in small groups
 where you might know 30 people in total.
 So you then had a system that was quite good
 for assessing who you could trust and who you could not.
 And if someone told you that Joe there is a jerk,
 but you had interacted with him yourself
 and seen him in action, and you would quickly realize maybe
 that that's actually not quite accurate, right?
 But now that we, the most people on the planet
 are people we've never met, it's very important
 that we have a way of trusting the information we're given.
 So, okay, so where does the news project come in?
 Well, throughout history, you can go read Machiavelli
 from the 1400s, and you'll see how already then
 they were busy manipulating people with propaganda and stuff.
 Propaganda is not new at all.
 And the incentives to manipulate people is just not new at all.
 What is it that's new?
 What's new is machine learning meets propaganda.
 That's what's new.
 That's why this has gotten so much worse.
 Some people like to blame certain individuals.
 Like in my liberal university bubble,
 many people blame Donald Trump and say it was his fault.
 I see it differently.
 I think Donald Trump just had this extreme skill
 at playing this game in the machine learning algorithm age.
 A game he couldn't have played 10 years ago.
 So what's changed?
 What's changed is, well, Facebook and Google
 and other companies, and I'm not badmouthing them.
 I have a lot of friends who work for these companies,
 good people, they deployed machine learning algorithms
 just to increase their profit a little bit,
 to just maximize the time people spent watching ads.
 And they had totally underestimated
 how effective they were gonna be.
 This was again, the black box, non-intelligible intelligence.
 They just noticed, oh, we're getting more ad revenue.
 Great, it took a long time until they even realized why
 and how and how damaging this was for society.
 Because of course, what the machine learning figured out
 was that the by far most effective way of gluing you
 to your little rectangle was to show you things
 that triggered strong emotions, anger, et cetera, resentment.
 And if it was true or not, didn't really matter.
 It was also easier to find stories that weren't true.
 If you weren't limited,
 that's just the limitation to show people.
 That's a very limiting fact.
 And before and long, we got these amazing filter bubbles
 on a scale we had never seen before.
 A couple of this to the fact that also
 the online news media were so effective
 that they killed a lot of print journalism.
 There's less than half as many journalists now in America,
 I believe, as there was a generation ago.
 You just couldn't compete with the online advertising.
 So all of a sudden, most people are not getting,
 even reading newspapers,
 they get their news from social media.
 And most people only get news in their little bubble.
 So along comes now some people like Donald Trump
 who figured out among the first successful politicians
 to figure out how to really play this new game
 and become very, very influential.
 But I think Donald Trump was a simple,
 he took advantage of it.
 He didn't create, the fundamental conditions were created
 by machine learning, sort of taking over the news media.
 So this is what motivated my little COVID project here.
 So, I said before, machine learning and tech in general,
 it's not evil, but it's also not good.
 It's just a tool that you can use
 for good things or bad things.
 And as it happens, machine learning and news
 is mainly used by the big players, big tech,
 to manipulate people into watch as many ads as possible,
 which had this unintended consequence of really screwing up
 our democracy and fragmenting it into filter bubbles.
 So I thought, well,
 machine learning algorithms were basically free.
 They can run on your smartphone for free also
 if someone gives them away to you, right?
 There's no reason why they only have to help the big guy
 to manipulate the little guy.
 They can just as well help the little guy
 to see through all the manipulation attempts
 from the big guy.
 So did this project, it's called,
 you can go to improvethenews.org.
 The first thing we've built is the little news aggregator.
 Looks a bit like Google News,
 except it has these sliders on it
 to help you break out of your filter bubble.
 So if you're reading, you can click, click,
 and go to your favorite topic.
 And then if you just slide the left, right slider
 all the way over to the left.
 There's two sliders, right?
 Yeah, there's the one, the most obvious one
 is the one that has left, right labeled on it.
 You go to left, you get one set of articles,
 you go to the right to see a very different truth appearing.
 Oh, that's literally left and right on the political spectrum.
 On the political spectrum.
 Yeah, so if you're reading about immigration, for example,
 it's very, very noticeable.
 And I think step one always,
 if you wanna not get manipulated,
 is just to be able to recognize the techniques people use.
 So it's very helpful to just see how they spin things
 on the two sides.
 I think many people are under the misconception
 that the main problem is fake news.
 It's not.
 I had an amazing team of MIT students
 where we did an academic project,
 use machine learning to detect the main kinds of bias
 over the summer.
 And yes, of course, sometimes there's fake news
 where someone just claims something that's false, right?
 Like, oh, Hillary Clinton just got divorced or something.
 But what we see much more of is actually just omissions.
 If you go to, there's some stories
 which just won't be mentioned by the left or the right
 because it doesn't suit their agenda.
 And then they'll instead mention other ones
 very, very, very much.
 So for example, we've had a number of stories
 about the Trump family's financial dealings.
 And then there's been a bunch of stories
 about the Biden family's,
 Hunter Biden's financial dealings.
 Surprise, surprise, they don't get equal coverage
 on the left and the right.
 One side loves to cover the Biden, Hunter Biden's stuff.
 And one side loves to cover the Trump.
 You can never guess which is which.
 But the great news is if you're a normal American citizen
 and you dislike corruption in all its forms,
 then slide, slide, you can just look at both sides
 and you'll see all the corrupt,
 those political corruption stories.
 It's really liberating to just take in the both sides,
 the spin on both sides.
 It somehow unlocks your mind to like think on your own,
 to realize that, I don't know,
 it's the same thing that was useful, right,
 in the Soviet Union times for when everybody
 was much more aware that they're surrounded by propaganda.
 It's so interesting what you're saying actually.
 So Noam Chomsky, used to be our MIT colleague,
 once said that propaganda is to democracy,
 what violence is to totalitarianism.
 And what he means by that is if you have
 a really totalitarian government,
 you don't need propaganda.
 People will do what you want them to do anyway,
 out of fear, right?
 But otherwise, you need propaganda.
 So I would say actually that the propaganda
 is much higher quality in democracies,
 much more believable.
 And it's really striking when I talk to colleagues,
 science colleagues like from Russia and China and so on,
 I noticed they are actually much more aware
 of the propaganda in their own media
 than many of my American colleagues are about
 the propaganda in Western media.
 That's brilliant, that means the propaganda
 in the Western media is just better.
 Yes, that's so brilliant.
 Even the propaganda.
 Oh man.
 But once you realize that,
 you realize there's also something very optimistic there
 that you can do about it, right?
 Because first of all, omissions,
 as long as there's no outright censorship,
 you can just look at both sides and pretty quickly
 piece together a much more accurate idea
 of what's actually going on, right?
 And develop a natural skepticism too.
 Yeah.
 Just an analytical scientific mind
 about the way you take in the information.
 Yeah, and I think, I have to say sometimes I feel
 that some of us in the academic bubble
 are too arrogant about this and somehow think,
 oh, it's just people who aren't as educated
 as us who are cool.
 When we are often just as gullible also,
 we read only our media and don't see through things.
 Anyone who looks at both sides like this in comparison
 will immediately start noticing
 the shenanigans being pulled.
 And I think what I tried to do with this app
 is that the big tech has to some extent
 try to blame the individual for being manipulated,
 much like big tobacco tried to blame the individuals
 entirely for smoking.
 And then later on, our government stepped up and said,
 actually, you can't just blame little kids
 for starting to smoke.
 We have to have more responsible advertising
 and this and that.
 I think it's a bit the same here.
 It's very convenient for a big tech to blame.
 So it's just people who are so dumb and get fooled.
 The blame usually comes in saying,
 oh, it's just human psychology.
 People just wanna hear what they already believe.
 But Professor David Rand at MIT actually partly debunked that
 with a really nice study showing that people
 tend to be interested in hearing things
 that go against what they believe
 if it's presented in a respectful way.
 Suppose, for example, that you have a company
 and you're just about to launch this project
 and you're convinced it's gonna work.
 And someone says, you know, Lex,
 I hate to tell you this, but this is gonna fail.
 And here's why.
 Would you be like, shut up, I don't wanna hear it.
 La la la la la la la la.
 Would you?
 You would be interested, right?
 And also if you're on an airplane,
 back in the pre-COVID times, you know,
 and the guy next to you is clearly from the opposite side
 of the political spectrum,
 but is very respectful and polite to you,
 wouldn't you be kind of interested to hear a bit
 about how he or she thinks about things?
 Of course.
 But it's not so easy to find out respectful disagreement now
 because like, for example, if you are a Democrat
 and you're like, I wanna see something on the other side.
 So you just go brightbart.com.
 And then after the first 10 seconds,
 you feel deeply insulted by something and they,
 it's not gonna work.
 Or if you take someone who votes Republican
 and they go to something on the left
 and they just get very offended very quickly
 by them having put a deliberately ugly picture
 of Donald Trump on the front page or something,
 it doesn't really work.
 So this news aggregator also has this nuanced slider
 which you can pull to the right
 and then to make it easier to get exposed
 to actually more sort of academic style
 or more respectful portrayals of different views.
 And finally, the one kind of bias
 I think people are mostly aware of is the left right,
 because it's so obvious,
 because both left and right are very powerful here.
 Both of them have well-funded TV stations and newspapers,
 and it's kind of hard to miss.
 But there's another one, the establishment slider,
 which is also really fun.
 I love to play with it.
 That's more about corruption.
 Because if you have a society
 where almost all the powerful entities
 want you to believe a certain thing,
 that's what you're gonna read in both the big media,
 mainstream media on the left and on the right, of course.
 And powerful companies can push back very hard,
 like tobacco companies pushed back very hard back in the day
 when some newspapers started writing articles
 about tobacco being dangerous.
 So it was hard to get a lot of coverage about it initially.
 And also if you look geopolitically,
 of course in any country when you read their media,
 you're mainly gonna be reading a lot of articles
 about how our country is the good guy
 and the other countries are the bad guys.
 So if you wanna have a really more nuanced understanding,
 like the Germans used to be told that,
 the British used to be told that the French were the bad guys
 and the French used to be told
 that the British were the bad guys.
 Now they visit each other's countries a lot
 and have a much more nuanced understanding.
 I don't think there's gonna be any more wars
 between France and Germany.
 But on the geopolitical scale,
 it's just as much as ever,
 big Cold War now, US, China, and so on.
 And if you wanna get a more nuanced understanding
 of what's happening geopolitically,
 then it's really fun to look at this establishment slider
 because it turns out there are tons of little newspapers,
 both on the left and on the right,
 who sometimes challenge establishment and say,
 maybe we shouldn't actually invade Iraq right now.
 Maybe this weapons of mass destruction thing is BS.
 If you look at the journalism research afterwards,
 you can actually see that,
 so clearly that both CNN and Fox were very pro,
 let's get rid of Saddam,
 there are weapons of mass destruction.
 Then there were a lot of smaller newspapers.
 They were like, wait a minute,
 this evidence seems a bit sketchy and maybe we,
 but of course they were so hard to find.
 Most people didn't even know they existed, right?
 Yet it would have been better for American national security
 if those voices had also come up.
 I think it harmed America's national security
 actually that we invaded Iraq.
 And arguably there's a lot more interest
 in that kind of thinking too from those small sources.
 So like the, when you say big,
 it's more about kind of the reach of the broadcast,
 but it's not big in terms of the interest.
 I think there's a lot of interest
 in that kind of anti-establishment
 or like skepticism towards, out of the box thinking.
 There's a lot of interest in that kind of thing.
 Do you see this news project or something like it
 being basically taken over the world
 as the main way we consume information?
 Like how do we get there?
 Like how do we, so okay, the idea is brilliant.
 It's a, you're calling it your little project in 2020,
 but how does that become the new way we consume information?
 I hope first of all, just to plant a little seed there
 because normally the big barrier of doing anything in media
 is you need a ton of money, but this costs no money at all.
 I've just been paying myself,
 pay a tiny amount of money each month to Amazon
 to run the thing in their cloud.
 We're not, there will never be any ads.
 The point is not to make any money off of it.
 And we just train machine learning algorithms
 to classify the articles and stuff.
 So it just kind of runs by itself.
 So if it actually gets good enough at some point
 that it starts catching on, it could scale.
 And if other people carbon copy it
 and make other versions that are better,
 that's more the merrier.
 I think there's a real opportunity for machine learning
 to empower the individual against the list
 of the powerful players.
 It's, as I said in the beginning here,
 it's been mostly the other way around so far
 that the big players have the AI
 and then they tell people this is the truth,
 this is how it is, but it can just as well
 go the other way around.
 Then when the internet was born actually,
 a lot of people had this hope that maybe this will be
 a great thing for democracy,
 make it easier to find out about things.
 And maybe machine learning and things like this
 can actually help again.
 And I have to say, I think it's more important than ever now
 because this is very linked also to the whole future of life,
 as we discussed earlier.
 We're getting this ever more powerful tech.
 It's frank, it's pretty clear if you look on the one
 or two generation, three generation time scale
 that there are only two ways this can end.
 Geopolitically.
 Either it ends great for all humanity
 or it ends terribly for all of us.
 There's really no in between.
 And we're so stuck in that,
 because technology knows no borders.
 And you can't have people fighting
 when the weapons just keep getting ever more powerful,
 indefinitely, eventually luck runs out.
 And right now we have, I love America,
 but the fact of the matter is what's good for America
 is not opposites in the long term
 to what's good for other countries.
 It would be if this was some sort of zero sum game,
 like it was thousands of years ago,
 when the only way one country could get more resources
 was to take land from other countries,
 because that was basically the resource.
 Look at the map of Europe,
 some countries kept getting bigger and smaller,
 endless wars.
 But then since 1945,
 there hasn't been any war in Western Europe,
 and they all got way richer because of tech.
 So the optimistic outcome is that the big winner
 in this century is gonna be America and China and Russia
 and everybody else,
 because technology just makes us all healthier and wealthier.
 And we just find some way of keeping the peace
 on this planet.
 But I think unfortunately,
 there are some pretty powerful forces right now
 that are pushing in exactly the opposite direction
 and trying to demonize other countries,
 which makes it more likely
 that this ever more powerful tech we're building
 is gonna use in disastrous ways.
 Yeah, for aggression versus cooperation,
 that kind of thing.
 Yeah, even look at just military AI now, right?
 20, it was so awesome to see these dancing robots.
 I loved it, right?
 But one of the biggest growth areas in robotics now
 is of course, autonomous weapons.
 And 2020 was like the best marketing year ever
 for autonomous weapons,
 because in both Libya, civil war,
 and in Nagorno-Karabakh,
 they made the decisive difference, right?
 And everybody else is like watching this,
 oh yeah, we wanna build autonomous weapons too.
 In Libya, you had on one hand,
 our ally, the United Arab Emirates,
 that were flying their autonomous weapons
 that they bought from China, bombing Libyans.
 And on the other side, you had our other ally, Turkey,
 flying their drones.
 They had no skin in the game, any of these other countries.
 And of course, it was the Libyans who really got screwed.
 In Nagorno-Karabakh, you had actually,
 again, Turkey is sending drones built by this company
 that was actually founded by a guy
 who went to MIT AeroAstro, do you know that?
 Bakr-Ratiyar, yeah?
 So MIT has a direct responsibility for ultimately this.
 And a lot of civilians were killed there.
 And so because it was militarily so effective,
 now suddenly there's like a huge push.
 Oh yeah, yeah, let's go build ever more autonomy
 into these weapons and it's gonna be great.
 And I think actually people who are obsessed
 about some sort of future terminators scenario right now
 should start focusing on the fact that we have two
 much more urgent threats happening for machine learning.
 One of them is the whole destruction of democracy
 that we've talked about now,
 where our flow of information is being manipulated
 by machine learning.
 And the other one is that right now,
 this is the year when the big arms race
 in out of control arms race in lethal weapons
 is gonna start or it's gonna stop.
 So you have a sense that there is like 2020
 was a instrumental catalyst for the race
 of for the autonomous weapons race.
 Yeah, because there was the first year
 when they proved decisive in the battlefield.
 And these ones are still not fully autonomous,
 mostly they're remote controlled, right?
 But we could very quickly make things
 about the size and cost of a smartphone,
 which you just put in the GPS coordinates
 or the face of the one you wanna kill
 is skin color or whatever, and it flies away and does it.
 And the real good reason why the US
 and all the other superpowers should put the kibosh on this
 is the same reason we decided
 to put the kibosh on bio weapons.
 So we gave the future of life award
 that we can talk more about later,
 Matthew Messelson from Harvard before
 for convincing Nixon to ban bio weapons.
 And I asked him, how did you do it?
 And he was like, well, I just said,
 look, we don't want there to be a $500 weapon
 of mass destruction that all our enemies can afford,
 even non-state actors.
 And Nixon was like,
 good point.
 It's in America's interest that the powerful weapons
 are all really expensive.
 So only we can afford them
 or maybe some more stable adversaries, right?
 Nuclear weapons are like that,
 but bio weapons were not like that.
 That's why we banned them.
 And that's why you never hear about them now.
 That's why we love biology.
 So you have a sense that it's possible
 for the big powerhouses in terms of the big names
 in terms of the big nations in the world
 to agree that autonomous weapons
 is not a race we wanna be on.
 That it doesn't end well.
 Yeah, because we know it's just gonna end
 in mass proliferation and every terrorist everywhere
 is gonna have these super cheap weapons
 that they will use against us.
 And our politicians have to constantly worry
 about being assassinated every time they go outdoors
 by some anonymous little mini drone.
 We don't want that.
 And even if the US and China
 and everyone else could just agree
 that you can only build these weapons
 if they cost at least 10 million bucks.
 That would be a huge win for the superpowers
 and frankly for everybody.
 And people often push back and say,
 well, it's so hard to prevent cheating.
 But hey, you can say the same about bio weapons.
 Take any of your MIT colleagues in biology.
 Of course they could build some nasty bio weapon
 if they'd really want to do.
 But first of all, they don't want to
 because they think it's disgusting because of the stigma.
 And second, even if there's some sort of nutcase
 and want to, it's very likely that some other grad students
 or someone would rat them out
 because everyone else thinks it's so disgusting.
 And in fact, we now know there was even a fair bit
 of cheating on the bio weapons ban,
 but no countries use them because it was so stigmatized
 that it just wasn't worth revealing that they had cheated.
 You talk about drones, but you kind of think
 that drones is the remote operation.
 Which they are mostly still.
 But you're not taking the next intellectual step
 of like, where does this go?
 You're kind of saying the problem with drones
 is that you're removing yourself from direct violence.
 Therefore you're not able to sort of maintain
 the common humanity required
 to make the proper decision strategically.
 But that's the criticism as opposed to like,
 if this is automated, and just exactly as you said,
 if you automate it and there's a race,
 then the technology can get better and better and better,
 which means getting cheaper and cheaper and cheaper.
 And unlike perhaps nuclear weapons,
 which is connected to resources in a way,
 like it's hard to get the, it's hard to engineer.
 Yeah, it feels like it's, you know,
 there's too much overlap between the tech industry
 and autonomous weapons,
 to where you could have smartphone type of cheapness.
 If you look at drones, you know, it's a, you know,
 for a thousand dollars, you can have an incredible system
 that's able to maintain flight autonomously for you
 and take pictures and stuff.
 You could see that going into the autonomous weapons space
 that's, but like, why is that not thought about
 or discussed enough in the public do you think?
 You see those dancing Boston Dynamics robots,
 and everybody has this kind of,
 like as if this is like a far future.
 They have this like fear, like, oh, this will be Terminator
 in like some, I don't know, unspecified 20, 30, 40 years.
 And they don't think about, well, this is like
 some much less dramatic version of that
 is actually happening now.
 It's not gonna have, it's not gonna be legged,
 it's not gonna be dancing,
 but it's already has the capability
 to use artificial intelligence to kill humans.
 Yeah, the Boston Dynamics legged robots,
 I think the reason we imagine them holding guns
 is just because you've all seen Arnold Schwarzenegger, right?
 That's our reference point.
 That's pretty useless.
 That's not gonna be the main military use of them.
 They might be useful in law enforcement in the future.
 And then there's a whole debate about
 do you want robots showing up at your house with guns,
 telling you to, who will be perfectly obedient
 to whatever dictator controls them?
 But let's leave that aside for a moment
 and look at what's actually relevant now.
 So there's a spectrum of things you can do
 with AI in the military.
 And again, to put my card on the table,
 I'm not the pacifist, I think we should have good defense.
 So for example, a Predator drone is basically
 a fancy little remote controlled airplane, right?
 There's a human piloting it
 and the decision ultimately about whether
 to kill somebody with it is made by a human still.
 And this is a line I think we should never cross.
 There's a current DOD policy.
 Again, you have to have a human in the loop.
 I think algorithms should never make life or death decisions.
 They should be left to humans.
 Now, why might we cross that line?
 Well, first of all, these are expensive, right?
 So for example, when Azerbaijan had all these drones
 and Armenia didn't have any,
 they start trying to jerry-rig little cheap things,
 fly around, but then of course the Armenians would jam them.
 The Azeris would jam them.
 And remote control things can be jammed.
 That makes them inferior.
 Also, there's a bit of a time delay between,
 if we're piloting something from far away, speed of light
 and the human has a reaction time as well,
 it would be nice to eliminate that jamming possibility
 in the time delay by having it fully autonomous.
 But now you might be crossing that exact line.
 You might program it to just, oh yeah, dear drone,
 go hover over this country for a while.
 And whenever you find someone who is a bad guy,
 kill them.
 Now the machine is making these sort of decisions.
 And some people who defend this still say,
 well, that's morally fine because we are the good guys
 and we will tell it the definition of bad guy
 that we think is moral.
 But now it would be very naive to think
 that if ISIS buys that same drone,
 that they're gonna use our definition of bad guy.
 Maybe for them, bad guy is someone
 wearing a US Army uniform.
 Or maybe there will be some weird ethnic group
 who decides that someone of another ethnic group,
 they're the bad guys.
 The thing is human soldiers, with all our faults,
 we still have some basic wiring in us.
 No, it's not okay to kill kids and civilians.
 And Thomas Weapon has none of that.
 It's just gonna do whatever is programmed.
 It's like the perfect Adolf Eichmann on steroids.
 Like they told him, Adolf Eichmann,
 we want you to do this and this and this
 to make the Holocaust more efficient.
 And he was like, yeah.
 And off he went and did it, right?
 Do we really wanna make machines that are like that,
 like completely amoral and will take the user's definition
 of who's the bad guy?
 And do we then wanna make them so cheap
 that all our adversaries can have them?
 Like what could possibly go wrong?
 That's I think the big argument for why we wanna,
 this year, really put the kibosh on this.
 And I think you can tell there's a lot of very active debate
 even going on within the US military
 and undoubtedly in other militaries around the world also
 about whether we should have
 some sort of international agreement
 to at least require that these weapons
 have to be above a certain size and cost
 so that things just don't totally spiral out of control.
 And finally, just for your question,
 but is it possible to stop it?
 Because some people tell me, oh, just give up.
 But again, so Matthew Messelson again from Harvard, right?
 Who the bioweapons hero,
 he had exactly this criticism also with bioweapons.
 People were like, how can you check for sure
 that the Russians aren't cheating?
 And he told me this, I think really ingenious insight.
 He said, you know, Max, some people think
 you have to have inspections and things
 and you have to make sure that people,
 you can catch the cheaters with a 100% chance.
 You don't need 100%, he said.
 1% is usually enough because if it's another big state,
 like suppose China and the US have signed the treaty,
 drawing a certain line and saying,
 yeah, these kinds of drones are okay,
 but these fully autonomous ones are not.
 Now suppose you are China and you have cheated
 and secretly developed some clandestine little thing,
 or you're thinking about doing it.
 What's your calculation that you do?
 Well, you're like, okay, what's the probability
 that we're gonna get caught?
 If the probability is 100%,
 of course we're not gonna do it.
 But if the probability is 5% that we're gonna get caught,
 then it's gonna be like a huge embarrassment for us.
 And we still have our nuclear weapons anyway,
 so it doesn't really make an enormous difference
 in terms of deterring the US.
 And that feeds the stigma that you kind of establish,
 like this fabric, this universal stigma over China.
 Over the thing.
 Exactly, it's very reasonable for them to say,
 well, we probably get away with it,
 but if we don't, then the US will know we cheated.
 And then they're gonna go full tilt with their program
 and say, look, the Chinese are cheaters,
 and now we have all these weapons against us,
 and that's bad.
 So the stigma alone is very, very powerful.
 And again, look what happened with bioweapons, right?
 It's been 50 years now.
 When was the last time you read about a bioterrorism attack?
 The only deaths I really know about with bioweapons
 that have happened, when we Americans managed to kill
 some of our own with anthrax,
 the idiot who sent them to Tom Daschle
 and others in letters, right?
 And similarly in Sverdlovsk in the Soviet Union,
 they had some anthrax in some lab there,
 maybe they were cheating or who knows,
 and it leaked out and killed a bunch of Russians.
 I'd say that's a pretty good success, right?
 50 years, just two own goals by the superpowers,
 and then nothing.
 And that's why whenever I ask anyone
 what they think about biology, they think it's great.
 They associate it with new cures, new diseases,
 maybe a good vaccine.
 This is how I want to think about AI in the future.
 I want, and I want others to think about AI too,
 as a source of all these great solutions to our problems,
 not as, oh, AI, oh yeah, that's the reason I feel scared
 going outside these days.
 Yeah, it's kind of brilliant that the bioweapons
 and nuclear weapons we've figured out, I mean,
 of course there's still a huge source of danger,
 but we figured out some way of creating rules
 and social stigma over these weapons
 that then creates a stability to our,
 whatever that game theoretic stability that, of course.
 Exactly, exactly.
 And we don't have that with AI,
 and you're kind of screaming from the top of the mountain
 about this, that we need to find that,
 because just like, you know, it's very possible,
 you know, with Future of Life, as you point out,
 Institute Awards pointed out that, you know,
 with nuclear weapons, we could have destroyed ourselves
 quite a few times, and it's, you know,
 it's a learning experience that doesn't, is very costly.
 We gave, you know, this Future of Life Award,
 we gave it the first time to this guy, Vasily Arkhipov,
 you know, he was on, most people haven't even heard of him.
 Yeah, can you say who he is?
 Vasily Arkhipov, he has, in my opinion,
 made the greatest positive contribution to humanity
 of any human in modern history.
 And maybe it sounds like hyperbole here,
 like I'm just over the top, but let me tell you the story,
 and I think maybe you'll agree.
 So during the Cuban Missile Crisis,
 we Americans first didn't know
 that the Russians had sent four submarines,
 but we caught two of them,
 and we didn't know that,
 so we dropped practice depth charges
 on the one that he was on,
 try to force it to the surface.
 But we didn't know that this nuclear submarine
 actually was a nuclear submarine with a nuclear torpedo.
 We also didn't know that they had authorization to launch it
 without clearance from Moscow,
 and we also didn't know
 that they were running out of electricity,
 their batteries were almost dead,
 they were running out of oxygen,
 sailors were fainting left and right,
 the temperature was about 110, 120 Fahrenheit on board,
 it was really hellish conditions,
 really just a kind of doomsday.
 And at that point, these giant explosions start happening
 from Americans dropping these.
 The captain thought World War III had begun.
 They decided that they were gonna launch the nuclear torpedo,
 and one of them shouted,
 we're all gonna die,
 but we're not gonna disgrace our Navy.
 We don't know what would have happened
 if there had been a giant mushroom cloud all of a sudden
 against Americans,
 but since everybody had their hands on the trigger,
 you don't have to be too creative to think
 that it could have led to an all-out nuclear war,
 in which case we wouldn't be having this conversation now.
 What actually took place was,
 they needed three people to approve this.
 The captain had said yes,
 there was the Communist Party political officer,
 he also said yes, let's do it.
 And the third man was this guy, Vasily Arkhipov,
 who said, and yet.
 For some reason, he was just more chill than the others,
 and he was the right man at the right time.
 I don't want us, as a species,
 rely on the right person being there at the right time.
 We tracked down his family living in relative poverty
 outside Moscow.
 When he flew his daughter, he had passed away,
 and flew them to London.
 They had never been to the West even.
 It was incredibly moving to get to honor them for this.
 The next year we gave a little bit of a surprise,
 we gave this Future Life Award to Stanislav Petrov.
 Have you heard of him?
 Yes.
 So he was in charge of the Soviet early warning station,
 which was built with Soviet technology,
 and honestly not that reliable.
 It said that there were five US missiles coming in.
 Again, if they had launched at that point,
 we probably wouldn't be having this conversation.
 He decided, based on mainly gut instinct,
 to just not escalate this.
 And I'm very glad he wasn't replaced by an AI
 that was just automatically following orders.
 And then we gave the third one to Matthew Messelson.
 Last year we gave this award to these guys
 who actually use technology for good,
 not avoiding something bad, but for something good.
 The guys who eliminated this disease,
 which is way worse than COVID,
 that had killed half a billion people
 in its final century, smallpox.
 So you mentioned it earlier.
 COVID, on average, kills less than 1% of people
 who get it, smallpox about 30%.
 And they just ultimately, Victor Zhdanov and Bill Fagy,
 most of my colleagues have never heard of either of them,
 one American, one Russian, they did this amazing effort.
 Not only was Zhdanov able to get the US and the Soviet Union
 to team up against smallpox during the Cold War,
 but Bill Fagy came up with this ingenious strategy
 for making it actually go all the way
 to defeat the disease without funding
 for vaccinating everyone.
 And as a result, we haven't had any,
 we went from 15 million deaths
 the year I was born in smallpox.
 So what do we have in COVID now?
 A little bit short of 2 million, right?
 Yes.
 To zero deaths, of course, this year and forever.
 There have been 200 million people,
 the estimate, who would have died since then by smallpox
 had it not been for this.
 So isn't science awesome when you use it for good?
 And the reason we wanna celebrate these sort of people
 is to remind them of this.
 Science is so awesome when you use it for good.
 And those awards, actually, the variety there
 paints a very interesting picture.
 So the first two are looking at,
 it's kind of exciting to think that these average humans
 in some sense, that there are products of billions
 of other humans that came before them, evolution.
 And some little, you said gut,
 but there's something in there
 that stopped the annihilation of the human race.
 And that's a magical thing,
 but that's like this deeply human thing.
 And then there's the other aspect where,
 that's also very human,
 which is to build solution to the existential crises
 that we're facing, to build it, to take the responsibility
 and to come up with different technologies and so on.
 And both of those are deeply human.
 The gut and the mind, whatever that is that creates.
 The best is when they work together.
 Arkhipov, I wish I could have met him, of course,
 but he had passed away.
 He was really a fantastic military officer
 combining all the best traits that we in America admire
 in our military, because first of all,
 he was very loyal, of course.
 He never even told anyone about this during his whole life,
 even though you think he had some bragging rights, right?
 But he just was like, this is just business,
 just doing my job.
 It only came out later after his death.
 And second, the reason he did the right thing
 was not because he was some sort of liberal
 or some sort of, not because he was just, oh, you know,
 peace and love.
 It was partly because he had been the captain
 on another submarine that had a nuclear reactor meltdown.
 And it was his heroism that helped contain this.
 That's why he died of cancer later also.
 But he'd seen many of his crew members die.
 And I think for him, that gave him this gut feeling
 that if there's a nuclear war between the US
 and the Soviet Union, the whole world is gonna go through
 what I saw, my own experience,
 gonna go through what I saw, my dear crew members,
 stuff like that.
 It wasn't just an abstract thing for him.
 I think it was real.
 And second though, not just the gut, the mind, right?
 He was for some reason, very level-headed personality
 and very smart guy, which is exactly what we want
 our best fighter pilots to be also, right?
 I never forget Neil Armstrong when he's landing on the moon
 and almost running out of gas.
 And he doesn't even change, when they say 30 seconds,
 he doesn't even change the tone of voice,
 just keeps going.
 Arkhipov, I think, was just like that.
 So when the explosions start going off
 and his captain is screaming,
 and we should nuke them and all, he's like,
 I don't think the Americans are trying to sink us.
 I think they're trying to send us a message.
 That's a pretty badass coolness.
 Because he said, if they wanted to sink us,
 and he said, listen, listen, it's alternating.
 One loud explosion on the left,
 one on the right, one on the left, one on the right.
 He was the only one who noticed this pattern.
 And he's like, I think this is,
 I'm trying to send us a signal
 that they wanted to surface,
 and they're not gonna sink us.
 And somehow, this is how he then managed to ultimately,
 with his combination of gut,
 and also just cool analytical thinking,
 was able to deescalate the whole thing.
 And yeah, so this is some of the best in humanity.
 I guess coming back to what we talked about earlier
 is the combination of the neural network,
 the instinctive, with, I'm tearing up here,
 I'm getting emotional.
 But he is one of my superheroes,
 having both the heart and the mind combined.
 And especially in that time, there's something about the,
 I mean, this is a very, in America,
 people are used to this kind of idea
 of being the individual, of like, on your own thinking.
 I think under, in the Soviet Union, under communism,
 it's actually much harder to do that.
 Oh yeah, he didn't even, he even got,
 he didn't get any accolades either
 when he came back for this, right?
 They just wanted to hush the whole thing up.
 Yeah, there's echoes of that with Chernobyl,
 there's all kinds of, that's one,
 that's a really hopeful thing
 that amidst big centralized powers,
 whether it's companies or states,
 there's still the power of the individual
 to think on their own, to act.
 But I think we need to think of people like this,
 not as a panacea we can always count on,
 but rather as a wake-up call.
 So because of them, because of Arkhipov,
 we are alive to learn from this lesson,
 to learn from the fact that we shouldn't keep playing
 Russian roulette and almost have a nuclear war
 by mistake now and then,
 because relying on luck is not a good long-term strategy.
 If you keep playing Russian roulette over and over again,
 the probability of surviving just drops exponentially
 with time.
 And if you have some probability
 of having an accidental nuclear war every year,
 the probability of not having one also drops exponentially.
 I think we can do better than that.
 So I think the message is very clear,
 once in a while shit happens,
 and there's a lot of very concrete things we can do
 to reduce the risk of things like that happening
 in the first place.
 On the AI front, if we could just link on that for a second.
 So you're friends with, you often talk with Elon Musk
 throughout history,
 you've did a lot of interesting things together.
 He has a set of fears about the future
 of artificial intelligence, AGI.
 Do you have a sense, we've already talked about
 the things we should be worried about with AI.
 Do you have a sense of the shape of his fears in particular
 about AI, which subset of what we've talked about,
 whether it's creating,
 it's that direction of creating
 sort of these giant computational systems
 that are not explainable,
 they're not intelligible intelligence,
 and then as a branch of that,
 is it the manipulation by big corporations of that
 or individual evil people to use that for destruction
 or the unintentional consequences?
 Do you have a sense of where his thinking is on this?
 From my many conversations with Elon,
 yeah, I certainly have a model of how he thinks.
 It's actually very much like the way I think also,
 I'll elaborate on it a bit.
 I just wanna push back on when you said evil people.
 I don't think it's a very helpful concept,
 evil people.
 Sometimes people do very, very bad things,
 but they usually do it because they think it's a good thing
 because somehow other people had told them
 that that was a good thing
 or given them incorrect information or whatever, right?
 I believe in the fundamental goodness of humanity
 that if we educate people well
 and they find out how things really are,
 people generally wanna do good and be good.
 Hence the value alignment.
 As opposed to it's about information, about knowledge,
 and then once we have that,
 we'll likely be able to do good in the way
 that's aligned with everybody else who thinks it's good.
 Yeah, and it's not just the individual people
 we have to align.
 So we don't just want people to be educated
 to know the way things actually are
 and to treat each other well,
 but we also need to align other non-human entities.
 We talked about corporations, there has to be institutions
 so that what they do is actually good
 for the country they're in
 and we should align, make sure that what countries do
 is actually good for the species as a whole, et cetera.
 Coming back to Elon,
 yeah, my understanding of how Elon sees this
 is really quite similar to my own,
 which is one of the reasons I like him so much
 and enjoy talking with him so much.
 I feel he's quite different from most people
 in that he thinks much more than most people
 about the really big picture,
 not just what's gonna happen in the next election cycle,
 but in millennia, millions and billions of years from now.
 And when you look in this more cosmic perspective,
 it's so obvious that we're gazing out into this universe
 that as far as we can tell, is mostly dead
 with life being almost imperceptibly tiny perturbation.
 And he sees this enormous opportunity
 for our universe to come alive,
 for us to become an interplanetary species.
 Mars is obviously just first stop on this cosmic journey.
 And precisely because he thinks more long-term,
 it's much more clear to him than to most people
 that what we do with this Russian roulette thing
 we keep playing with our nukes
 as a really poor strategy, really reckless strategy.
 And also that we're just building
 these ever more powerful AI systems
 that we don't understand,
 it's also a severely reckless strategy.
 I feel Elon is very much a humanist
 in the sense that he wants an awesome future for humanity.
 He wants it to be us that control the machines
 rather than the machines that control us.
 Yes.
 And why shouldn't we insist on that?
 We're building them after all, right?
 Why should we build things that just make us
 into some little cog in the machinery
 that has no further say in the matter, right?
 It's not my idea of an inspiring future either.
 Yeah, if you think on the cosmic scale
 in terms of both time and space,
 so much is put into perspective.
 Yeah.
 Whenever I have a bad day, that's what I think about it.
 Immediately makes me feel better.
 It makes me sad that for us individual humans,
 at least for now, the ride ends too quickly.
 We don't get to experience the cosmic scale.
 Yeah, I mean, I think of our universe sometimes
 as an organism that has only begun to wake up a tiny bit,
 just like when you're in the very first little glimmers
 of consciousness you have in the morning
 when you start coming around.
 Before the coffee.
 Before the coffee, even before you get out of bed,
 before you even open your eyes,
 start to wake up a little bit.
 There's something here.
 That's very much how I think of where we are.
 All those galaxies out there,
 I think they're really beautiful.
 But why are they beautiful?
 They're beautiful because conscious entities
 are actually observing them
 and experiencing them through our telescopes.
 I define consciousness as subjective experience,
 whether it be colors or emotions or sounds.
 So beauty is an experience.
 Meaning is an experience.
 Purpose is an experience.
 If there was no conscious experience observing these galaxies,
 they wouldn't be beautiful.
 If we do something dumb with advanced AI in the future here
 and Earth originating, life goes extinct.
 And that was it for this.
 If there is nothing else with telescopes in our universe,
 then it's kind of game over for beauty and meaning
 and purpose in our whole universe.
 And I think that would be just such an opportunity lost,
 frankly.
 And I think when Elon points this out,
 he gets very unfairly maligned in the media
 for all the dumb media bias reasons we talked about.
 They want to print precisely the things about Elon
 out of context that are really clickbaity.
 He has gotten so much flack
 for this summoning the demon statement.
 I happen to know exactly the context
 because I was in the front row when he gave that talk.
 It was at MIT, you'll be pleased to know.
 It was the AeroAstro anniversary.
 They had Buzz Aldrin there from the moon landing,
 whole house, a Kresge auditorium packed with MIT students.
 And he had this amazing Q&A,
 it might've gone for an hour.
 And they talked about rockets and Mars and everything.
 At the very end, this one student
 who has actually hit my class asked him, what about AI?
 Elon makes this one comment
 and they take this out of context, print it, goes viral.
 Or is it like with AI,
 we're summoning the demon, something like that.
 And try to cast him as some sort of doom and gloom dude.
 You know Elon, he's not the doom and gloom dude.
 He is such a positive visionary.
 And the whole reason he warns about this
 is because he realizes more than most
 what the opportunity cost is of screwing up,
 that there is so much awesomeness in the future
 that we can and our descendants can enjoy
 if we don't screw up, right?
 I get so pissed off when people try to cast him
 as some sort of technophobic Luddite.
 And at this point, it's kind of ludicrous
 when I hear people say that people who worry
 about artificial general intelligence are Luddites
 because of course, if you look more closely,
 you have some of the most outspoken people
 making warnings are people like
 Professor Stuart Russell from Berkeley,
 who's written the bestselling AI textbook.
 So claiming that he's a Luddite who doesn't understand AI
 is the joke is really on the people who said it.
 But I think more broadly,
 this message is really not sunk in at all.
 What it is that people worry about,
 they think that Elon and Stuart Russell and others
 are worried about the dancing robots
 picking up an AR-15 and going on a rampage, right?
 They think they're worried about robots turning evil.
 They're not, I'm not.
 The risk is not malice, it's competence.
 The risk is just that we build some systems
 that are incredibly competent,
 which means they're always gonna get their goals accomplished
 even if they clash with our goals.
 That's the risk.
 Why did we humans drive the West African black rhino extinct?
 Is it because we're malicious, evil, rhinoceros haters?
 No, it's just because our goals didn't align
 with the goals of those rhinos
 and tough luck for the rhinos.
 So the point is just we don't wanna put ourselves
 in the position of those rhinos
 creating something more powerful than us
 if we haven't first figured out how to align the goals.
 And I am optimistic, I think we could do it
 if we worked really hard on it
 because I spent a lot of time around intelligent entities
 that were more intelligent than me, my mom and my dad.
 And I was little and that was fine
 because their goals were actually aligned
 with mine quite well.
 But we've seen today many examples
 of where the goals of our powerful systems
 are not so aligned.
 So those click through optimization algorithms
 that are polarized social media, right?
 They were actually pretty poorly aligned
 with what was good for democracy, it turned out.
 And again, almost all problems we've had
 in the machine learning again came so far, not from malice,
 but from poor alignment.
 And that's exactly why we should be concerned
 about it in the future.
 Do you think it's possible that with systems
 like Neuralink and brain computer interfaces,
 again, thinking of the cosmic scale,
 Elon's talked about this,
 but others have as well throughout history
 of figuring out how the exact mechanism
 of how to achieve that kind of alignment.
 So one of them is having a symbiosis with AI,
 which is like coming up with clever ways
 where we're like stuck together in this weird relationship,
 whether it's biological or in some kind of other way.
 Do you think that's a possibility
 of having that kind of symbiosis?
 Or do we wanna instead kind of focus on this
 distinct entities of us humans
 talking to these intelligible, self-doubting AIs,
 maybe like Stuart Russell thinks about it.
 It's like we're self-doubting and full of uncertainty
 and then our AI systems are full of uncertainty.
 We communicate back and forth
 and in that way achieve symbiosis.
 I honestly don't know.
 I was thinking about it,
 I honestly don't know.
 I would say that because we don't know for sure
 which of any of our ideas will work,
 but we do know that if we don't,
 I'm pretty convinced that if we don't get
 any of these things to work and just barge ahead,
 then our species is probably gonna go extinct this century.
 I think it's-
 This century, you think we're facing this crisis
 is the 21st century crisis.
 This century will be remembered.
 On a hard drive.
 On a hard drive somewhere or maybe by future generations
 as like there'll be future of Life Institute awards
 for people that have done something about AI.
 It could also end even worse,
 where we're not superseded by leaving any AI behind either.
 We just totally wipe out, like on Easter Island.
 Our century is long.
 No, there are still 79 years left of it.
 Think about how far we've come just in the last 30 years.
 We can talk more about what might go wrong,
 but you asked me this really good question
 about what's the best strategy.
 Is it Neuralink or do it Russell's approach or whatever?
 I think when we did the Manhattan Project,
 we didn't know if any of our four ideas
 for enriching uranium and getting out the uranium-235
 were gonna work.
 We felt this was really important
 to get it before Hitler did.
 So you know what we did?
 We tried all four of them.
 Here, I think it's analogous where there's the greatest
 threat that's ever faced our species.
 And of course, US national security by implication.
 We don't know, we don't have any method
 that's guaranteed to work, but we have a lot of ideas.
 So we should invest pretty heavily in pursuing all of them
 with an open mind and the hope that one of them
 at least works.
 These are, the good news is the century is long,
 and it might take decades until we have artificial
 general intelligence.
 So we have some time hopefully,
 but it takes a long time to solve these very,
 very difficult problems.
 It's gonna actually be the most difficult problem
 we were ever trying to solve as a species.
 So we have to start now.
 So we don't wanna have, rather than begin thinking about it
 the night before some people who've had too much red bull
 switching on and we have to, coming back to your question,
 we have to pursue all of these different avenues and see.
 If you're my investment advisor,
 and I was trying to invest in the future,
 how do you think the human species is most likely
 to destroy itself in the century?
 Yeah, so if the crises, many of the crises we're facing
 are really before us within the next 100 years,
 how do we make explicit,
 make known the unknowns and solve those problems
 to avoid the biggest,
 starting with the biggest existential crisis?
 So as your investment advisor,
 how are you planning to make money on us
 destroying ourselves, I have to ask?
 I don't know.
 It might be the Russian origins.
 Somehow it's involved.
 At the micro level of detailed strategies,
 of course, these are unsolved problems.
 For AI alignment, we can break it into three sub problems
 that are all unsolved.
 I think you want first to make machines understand our goals,
 then adopt our goals and then retain our goals.
 So to hit on all three real quickly.
 The problem when Andreas Lubitz told his autopilot
 to fly into the Alps was that the computer
 didn't even understand anything about his goals.
 It was too dumb.
 It could have understood actually,
 but you would have had to put some effort in
 as a system designer to don't fly into mountains.
 So that's the first challenge.
 How do you program into computers,
 human values, human goals?
 We could start, rather than saying, oh, it's so hard,
 we should start with the simple stuff, as I said,
 self-driving cars, airplanes, just put in all the goals
 that we all agree on already, and then have a habit
 of whenever machines get smarter,
 so they can understand one level higher goals,
 put them into.
 The second challenge is getting them to adopt the goals.
 It's easy for situations like that,
 where you just program it in,
 but when you have self-learning systems like children,
 any parent knows that
 there's a difference between getting our kids
 to understand what we want them to do
 and to actually adopt our goals.
 With humans, with children, fortunately,
 they go through this phase.
 First, they're too dumb to understand
 what our goals are, and then they have this period,
 some years, when they're both smart enough
 to understand them and malleable enough
 that we have a chance to raise them well,
 and then they become teenagers, and it's too late.
 But we have this window.
 With machines, the intelligence might grow so fast
 that that window is pretty short.
 So that's a research problem.
 The third one is how do you make sure they keep the goals
 if they keep learning more and getting smarter?
 Many sci-fi movies are about how you have something
 in which initially was a line,
 but then things kind of go off-heel.
 My kids were very, very excited about their Legos
 when they were little,
 now they're just gathering dust in the basement.
 If we create machines that are really on board
 with the goal of taking care of humanity,
 we don't want them to get as bored with us
 as my kids got with Legos.
 So this is another research challenge.
 How can you make some sort of recursively
 self-improving system retain certain basic goals?
 That said, a lot of adult people still play with Legos,
 so maybe we succeeded with the Legos.
 Maybe, I like your optimism.
 But above all- So not all AI systems
 have to maintain the goals, right?
 Some just some fraction.
 Yeah, so there's a lot of talented AI researchers now
 who have heard of this and wanna work on it.
 Not so much funding for it yet.
 Of the billions that go into building AI more powerful,
 it's only a minuscule fraction.
 So far going into the safety research.
 My attitude is generally we should not try to slow down
 the technology, but we should go into the safety research.
 We should not try to slow down the technology,
 but we should greatly accelerate the investment
 in this sort of safety research.
 And also make sure, it's been,
 this was very embarrassing last year,
 but the NSF decided to give out
 six of these big institutes.
 We got one of them for AI and science,
 you asked me about.
 Another one was supposed to be for AI safety research,
 and they gave it to people studying oceans
 and climate and stuff.
 So I'm all for studying oceans and climates,
 have some money that actually goes into AI safety research
 also and doesn't just get grabbed by whatever.
 That's a fantastic investment.
 And then at the higher level, you asked this question,
 okay, what can we do?
 What are the biggest risks?
 I think we cannot just consider this
 to be only a technical problem.
 Again, because if you solve only the technical problem,
 can I play with your robot?
 Yes, please.
 If we can get our machines to just blindly obey
 the orders we give them,
 so we can always trust that it will do what we want,
 that might be great for the owner of the robot,
 that might not be so great for the rest of humanity
 if that person is that least favorite world leader
 or whatever you imagine.
 So we have to also take a look at the apply alignment,
 not just to the machines,
 but to all the other powerful structures.
 That's why it's so important to strengthen our democracy again.
 As I said, to have institutions that make sure
 that the playing field is not rigged
 so that corporations are given the right incentives
 to do the things that both make profit and are good for people,
 to make sure that countries have incentives
 to do things that are both good for their people
 and don't screw up the rest of the world.
 And this is not just something for AI nerds to geek out on.
 This is an interesting challenge for political scientists,
 economists, and so many other thinkers.
 So one of the magical things that perhaps makes this earth quite unique
 is that it's home to conscious beings.
 So you mentioned consciousness.
 Perhaps as a small aside,
 because we didn't really get specific to how we might do the alignment.
 Like you said, it's just a really important research problem.
 But do you think engineering consciousness into AI systems is a possibility?
 It's something that we might one day do?
 Or is there something fundamental to consciousness that is...
 Is there something about consciousness
 that is fundamental to humans and humans only?
 I think it's possible.
 I think both consciousness and intelligence are information processing.
 Certain types of information processing.
 And that fundamentally, it doesn't matter whether the information is processed by
 carbon atoms in neurons and brains or by silicon atoms and so on in our technology.
 Some people disagree.
 This is what I think as a physicist.
 That consciousness is the same kind of...
 You said consciousness is information processing.
 So meaning, I think you had a quote of something like
 it's information knowing itself, that kind of thing.
 I think consciousness is the way information feels when it's being processed.
 When it's being prepared.
 We don't know exactly what those complex ways are.
 It's clear that most of the information processing in our brains
 does not create an experience.
 We're not even aware of it.
 For example, you're not aware of your heartbeat regulation right now,
 even though it's clearly being done by your body.
 It's just kind of doing its own thing.
 When you go jogging, there's a lot of complicated stuff about how you put your foot down.
 And we know it's hard.
 That's why robots used to fall over so much.
 But you're mostly unaware about it.
 Your brain, your CEO consciousness module just sends an email.
 Hey, I want to keep jogging along this path.
 The rest is on autopilot.
 So most of it is not conscious.
 But somehow there are some of the information processing which is,
 we don't know what exactly.
 I think this is a science problem that I hope one day
 have some equation for or something.
 So we can be able to build a consciousness detector and say,
 yeah, here there is some consciousness.
 Here there's not.
 Oh, don't boil that lobster because it's feeling pain.
 Or it's okay because it's not feeling pain.
 Right now we treat this as sort of just metaphysics.
 But it would be very useful in emergency rooms to know if a patient
 has locked in syndrome and is conscious or if they are actually just out.
 And in the future, if you build a very, very intelligent helper robot to take care of you,
 I think you'd like to know if you should feel guilty by shutting it down.
 Or if it's just like a zombie going through the motions like a fancy tape recorder.
 Right.
 And once we can make progress on the science of consciousness and figure out
 what is conscious and what isn't, then we, assuming we want to create positive experiences
 and not suffering, we'll probably choose to build some machines that are deliberately unconscious
 that do incredibly boring, repetitive jobs in an iron mine somewhere or whatever.
 And maybe we'll choose to create helper robots for the elderly that are conscious
 so that people don't just feel creeped out that the robot is just faking it
 when it acts like it's sad or happy.
 Like I said, elderly, I think everybody gets pretty deeply lonely in this world.
 Yeah.
 And so there's a place, I think for everybody to have a connection with conscious beings,
 whether they're human or otherwise.
 But I know for sure that I would, if I had a robot,
 if I was going to develop any kind of personal emotional connection with it,
 I would be very creeped out if I knew it in intellectual level,
 that the whole thing was just a fraud.
 Now today you can buy a little talking doll for a kid, which will say things.
 And the little child will often think that this is actually conscious
 and even real secrets to it that then go on the internet
 and with all sorts of creepy repercussions.
 I would not want to be just hacked and tricked like this.
 If I was going to be developing real emotional connections with a robot,
 I would want to know that this is actually real.
 It's acting conscious, acting happy because it actually feels it.
 And I think this is not sci-fi.
 I think it's possible to measure, to come up with tools.
 After we understand the science of consciousness,
 you're saying we'll be able to come up with tools that can measure consciousness
 and definitively say like this thing is experiencing the things it says it's experiencing.
 Kind of by definition, if it is a physical phenomena, information processing,
 and we know that some information processing is conscious and some isn't,
 well, then there is something there to be discovered with the methods of science.
 Giulio Tononi has stuck his neck out the farthest and written down some equations
 for a theory.
 Maybe that's right.
 Maybe it's wrong.
 We certainly don't know.
 But I applaud that kind of efforts to sort of take this,
 say this is not just something that philosophers can have beer and muse about,
 but something we can measure and study.
 And coming, being that back to us, I think what we would probably choose to do,
 as I said, is if we cannot figure this out,
 choose to make, to be quite mindful about what sort of consciousness, if any,
 we put in different machines that we have.
 We, and certainly we wouldn't want to make,
 we should not be making much machines that suffer without us even knowing it.
 Right.
 And if, at any point, someone decides to upload themselves,
 like Ray Kurzweil wants to do, I don't know if you've had him on your show.
 We agree, but then COVID happened, so we're waiting it out a little bit.
 You know, suppose he uploads himself into this Robo-Ray,
 and it talks like him and acts like him and laughs like him.
 And before he powers off his biological body,
 he would probably be pretty disturbed if he realized that there's no one home.
 This robot is not having any subjective experience.
 Right.
 If we, if humanity gets replaced by, by, by machine descendants,
 do all these cool things and build spaceships and,
 and go to intergalactic rock concerts.
 And it turned out, turns out that they are all unconscious,
 just going through the motions.
 Wouldn't that be like the ultimate robot zombie apocalypse?
 Right.
 Just a play for empty benches.
 Yeah.
 I have a sense that there's some kind of, once we understand consciousness better,
 we'll understand that there's some kind of continuum,
 and it would be a greater appreciation.
 And we'll probably understand, just like you said,
 it'd be unfortunate if it's a trick.
 We'll probably definitively understand that love is indeed a trick
 that we'll play on each other, uh, that we humans are,
 we convince ourselves we're conscious, but we're really, um, you know,
 awesome trees and dolphins are all the same kind of conscious.
 Can I try to cheer you up a little bit with a philosophical thought here about the love part?
 Yes, let's do it.
 You know, you might say, okay, yeah, love is just a collaboration enabler.
 And then you'll, and then maybe you can go and get depressed about that.
 But I think that would be the wrong conclusion, actually.
 You know, I know that the only reason I enjoy food is because my genes hacked me
 and they don't want me to starve to death, not because they care about me.
 It's consciously enjoying succulent to delights of pistachio ice cream,
 but they just, they just want me to make copies of them.
 The whole thing.
 So in a sense, the whole, the whole idea of love is just a
 scam. The whole enjoyment of food is also a scam like this.
 But does that mean I shouldn't take pleasure in this pistachio ice cream?
 I love pistachio ice cream.
 And I can tell you, I have, I have, I know this is an experimental fact.
 I enjoy pistachio ice cream every bit as much, even though I scientifically know
 exactly why, what kind of scam this was.
 Your genes really appreciate that you like the pistachio ice cream.
 Well, but I, my mind appreciates it too, you know,
 I have a conscious experience right now.
 Ultimately, all of my brain is also just something the genes built to copy
 themselves, but so what, you know, I'm grateful that, yeah, thanks genes for
 doing this, but you know, now it's my brain that's in charge here and I'm
 going to enjoy my conscious experience.
 Thank you very much.
 And not just the pistachio ice cream, but also the love I feel for my amazing
 wife and all the other delights of being conscious.
 I don't actually, Richard Feynman, I think said this so well.
 He is also the guy, you know, really got me into physics.
 Some art friend said that, oh, science kind of just is the party pooper.
 It's kind of ruins the fun, right?
 It went like, you have a beautiful flowers as the artist, and then the
 scientist is going to deconstruct that into just a blob of quirks and electrons.
 And, and Feynman pushed back on that in such a beautiful way, which I think also
 can be used to push back and make you not feel guilty about falling in love.
 So here's what Feynman basically said.
 He said to his friend, you know, yeah, I can also, as a scientist, see that this
 is a beautiful flower.
 Thank you very much.
 Maybe I'm, I can't draw as good a painting as you because I'm not as
 talented an artist, but yeah, I can really see the beauty in it.
 And it just, it also looks beautiful to me.
 But in addition to that, Feynman said as a scientist, I see even more beauty
 that the artist did not see.
 Right.
 Suppose this is the flower on a blossoming apple tree.
 You could say this tree has more beauty in it than just the colors and the
 fragrance.
 This tree is made of air.
 Feynman wrote.
 This is one of my favorite Feynman quotes ever.
 And it took the carbon out of the air and bound it in using the flaming heat of
 the sun, you know, to turn the air into tree.
 And when you burn logs in your fireplace, it's really beautiful to think that this
 is being reversed.
 Now the tree is going, the wood is going back into air and this flaming,
 beautiful dance of the fire that the artists can see is the flaming light of
 the sun that was bound into turn the air into tree.
 And then the ashes is the little residue that didn't come from the air that the
 tree sucked out of the ground.
 You know, Feynman said, these are beautiful things and science just adds.
 It doesn't subtract.
 And I feel exactly that way about love and about pistachio ice cream.
 Also, I can understand that it didn't even, there is even more nuance to the
 whole thing, right?
 At this very visceral level, you can fall in love just as much as someone who
 knows nothing about neuroscience, but you can also appreciate this even greater
 beauty in it.
 It's just like, isn't it remarkable that it came about from, from this completely
 lifeless universe, just a bunch of hot blob of plasma expanding.
 And then over the eons, you know, gradually first the strong nuclear force
 decided to combine quirks together into nuclei.
 And then the electric force bound in electrons and made atoms, and then they
 clustered from gravity and you've got planets and stars and this and that.
 And then natural selection came along and the genes had their little thing.
 And you started getting what went from seeming like a completely pointless
 universe that we're just trying to increase entropy and approach heat death
 into something that looked more goal oriented.
 Isn't that kind of beautiful?
 And then this goal orientedness revolution got ever more sophisticated where
 you got ever more.
 And then you started getting this thing, which is kind of like DeepMind's
 mu zero and steroids, self, the ultimate self play is not what DeepMind's AI does
 against itself to get better at go.
 It's what all these little cork blobs did against each other in the game of
 survival of the fittest.
 Now, when you had really dumb bacteria living in a simple environment, there
 wasn't much incentive to get an intelligent, but then the life made environment
 more complex and then there was more incentive to get even smarter.
 And that gave the other organisms more of incentive to also get smarter.
 And then here we are now just like, like mu zero learned to become world
 master at, at the go and chess from playing as itself by just playing
 against itself, all the quirks here on our planet and electrons have created
 giraffes and elephants and humans and pistachio ice cream.
 I just find that really beautiful.
 And I, to me, that just adds to the enjoyment of, of love.
 It doesn't subtract anything.
 Do you feel a little more?
 I feel way better.
 That was, that was incredible.
 So the, this self play of quirks taking back to the beginning of our
 conversation a little bit, you've, there's so many exciting possibilities
 about artificial intelligence, understanding the basic laws of physics.
 Do you think AI will help us unlock?
 There's been a, quite a bit of excitement throughout the history of physics of
 coming up with more and more general, simple laws that explain the nature of
 our reality.
 And then the ultimate of that would be a theory of everything that combines
 everything together.
 Do you think it's possible that, well, one, we humans, but perhaps AI systems
 will figure out a theory of physics that unifies all the laws of physics?
 Yeah, I think it's absolutely possible.
 I think it's very clear that we're going to see a great boost to science.
 We're already seeing a boost actually from, from machine learning, helping
 science, alpha fold was an example, you know, decades old protein folding
 problem.
 So gradually, yeah.
 Unless we go extinct by doing something dumb, like we discussed, I think it's
 very likely that our understanding of physics will become so good that our
 technology will no longer be limited by human intelligence, but instead be
 limited by the laws of physics.
 So our tech today is limited by what we've been able to invent, right?
 I think as AI progresses, it'll just be limited by the speed of light and
 other physical limits, which will mean it's going to be just dramatically
 beyond where we are now.
 Do you think it's a fundamentally mathematical pursuit of trying to
 understand like the laws of this, that govern our universe from a mathematical
 perspective?
 So almost like if it's AI, it's exploring the space of like theorems and
 those kinds of things, or is there some other, is there some other more
 computational ideas, more sort of empirical ideas?
 They're both.
 I would say it's really interesting to look out at the landscape of
 everything we call science today.
 So here you come now with this big new hammer, it says machine learning on
 it and ask, you know, where are there some nails that you can help with
 here that you can hammer?
 Ultimately, if machine learning gets the point that it can do everything
 better than us, it will be able to help across the whole space of science.
 But maybe we can anchor it by starting a little bit right now near term and
 see how we kind of move forward.
 So right now, first of all, you have a lot of big data science where, for
 example, with telescopes, we are able to collect way more data every hour
 than a grad student can just pour over like in the old times.
 Right.
 And machine learning is already being used very effectively, even at MIT,
 like to find planets around other stars, to detect exciting new signatures of
 new particle physics in the sky and to detect the ripples and the fabric of
 space time that we call gravitational waves caused by enormous black holes
 crashing into each other halfway across the observable universe.
 Machine learning is running and taking it right now, you know, doing all
 these things.
 And it's really helping all these experimental fields.
 There is a separate front of physics, computational physics, which is getting
 an enormous boost also.
 So we had to do all our computations by hand, right?
 People would have these giant books with tables of logarithms and, oh my God,
 it pains me to even think how long time it would have taken to do simple stuff.
 Then we started to get little calculators and computers that could do
 some basic math for us.
 Now what we're starting to see is kind of a shift from GoFi computational
 physics to neural network computational physics.
 What I mean by that is most computational physics would be done by humans
 programming in the intelligence of how to do the computation into the computer.
 Just as when Gary Kasparov got his posterior kicked by IBM's Deep Blue in
 chess, humans had programmed in exactly how to play chess.
 Intelligence came from the humans.
 It wasn't learned, right?
 Mu zero can be not only Kasparov in chess, but also Stockfish, which is the best
 sort of GoFi chess program by learning.
 We're seeing more of that now, that shift beginning to happen in physics.
 Let me give you an example.
 So Lattice QCD is an area of physics whose goal is basically to take the
 periodic table and just compute the whole thing from first principles.
 This is not the search for theory of everything.
 We already know the theory that's supposed to produce as output the
 periodic table, which atoms are stable, how heavy they are, all that good stuff.
 There's spectral lines.
 It's a theory, Lattice QCD.
 You can put it on your t-shirt.
 Our colleague Frank Wilczek got the Nobel Prize for working on it.
 But the math is just too hard for us to solve.
 We have not been able to start with these equations.
 And solve them to the extent that we can predict, oh yeah.
 And then there is carbon and this is what the spectrum of the carbon atom looks like.
 But awesome people are building these super computer simulations where you just
 put in these equations and you make a big cubic lattice of space, or actually
 it's a very small lattice because you're going to the subatomic scale, down to
 the subatomic scale, and you try to solve it, and you get a very, very
 big cube of space, and you're like, oh, I'm going to solve this.
 But it's just so computationally expensive that we still haven't been able to
 calculate things as accurately as we measure them in many cases.
 And now machine learning is really revolutionizing this.
 So my colleague, Fiyala Shanahan at MIT, for example, she's been using this really
 cool machine learning technique called normalizing flows, where she's realized
 she can actually speed up the calculation dramatically by having the AI
 in other area like this, where we suck up an enormous amount of super computer time
 to do physics as black hole collisions.
 So now that we've done the sexy stuff of detecting a bunch of this,
 LIGO and other experiments, we want to be able to know what we're seeing.
 And so it's a very simple conceptual problem.
 It's the two body problem.
 Newton solved it for classical gravity, hundreds of years ago.
 But the two body problem is still not fully solved.
 For black holes.
 Yes, and Einstein's gravity, because they won't just orbit each other forever
 anymore, two things.
 They give off gravitational waves and make sure they crash into each other.
 And the game, what you want to do is you want to figure out, okay,
 what kind of wave comes out as a function of the masses of the two black holes,
 as a function of how they're spinning relative to each other, et cetera.
 And that is so hard.
 It can take months of super computer time and massive numbers of cores to do it.
 Wouldn't it be great if you can use machine learning to greatly speed that up?
 Now you can use the expensive old go-fi calculation as the truth,
 and then see if machine learning can figure out a smarter,
 faster way of getting the right answer.
 Yet another area, like computational physics.
 These are probably the big three that suck up the most computer time.
 Lattice QCD, black hole collisions, and cosmological simulations,
 where you take not a subatomic thing and try to figure out the mass of the proton,
 but you take something that's enormous and try to look at how all the galaxies get formed in there.
 There again, there are a lot of very cool ideas right now about how you can use machine learning
 to do this sort of stuff better.
 The difference between this and the big data is you kind of make the data yourself.
 And then finally, we're looking over the physics landscape and saying,
 what can we hammer with machine learning?
 So we talked about experimental data, big data,
 discovering cool stuff that we humans then look more closely at.
 Then we talked about taking the expensive computations we're doing now
 and figuring out how to do them much faster and better with AI.
 And finally, let's go really theoretical.
 Things like discovering equations, having deep fundamental insights.
 This is something closest to what I've been doing in my group.
 We talked earlier about the whole AI Feynman project, where if you just have some data,
 how do you automatically discover equations that seem to describe this well,
 that you can then go back as a human and work with and test and explore?
 You asked a really good question also about if this is sort of a search problem in some sense.
 That's very deep, actually, what you said, because it is.
 Suppose I ask you to prove some mathematical theorem.
 What is a proof in math?
 It's just a long string of steps, logical steps that you can write out with symbols.
 And once you find it, it's very easy to write a program to check whether it's
 a valid proof or not.
 Why is it so hard to prove it then?
 Well, because there are ridiculously many possible candidate proofs you could write down.
 If the proof contains 10,000 symbols, even if there were only 10 options for what each symbol
 could be, that's 10 to the power of 1,000 possible proofs, which is way more than there are atoms in
 our universe.
 You could say it's trivial to prove these things.
 You just write a computer, generate all strings, and then check, is this a valid proof?
 No.
 Is this a valid proof?
 No.
 And then you just keep doing this forever.
 But it is fundamentally a search problem.
 You just want to search the space of all those all strings of symbols to find one that is the
 proof, right?
 And there's a whole area of machine learning called search.
 How do you search with some giant space to find the needle in the haystack?
 It's easier in cases where there's a clear measure of good, like you're not just right or wrong,
 but this is better and this is worse.
 You can maybe get some hints as to which direction to go in.
 That's why we talked about neural networks work so well.
 I mean, that's such a human thing of that moment of genius of figuring out the intuition of
 good, essentially.
 I mean, we thought that that was...
 Or is it?
 Maybe it's not right.
 We thought that about chess, right?
 Exactly.
 The ability to see like 10, 15, sometimes 20 steps ahead was not a calculation that humans
 were performing.
 It was some kind of weird intuition about different patterns, about board positions,
 about the relative positions, somehow stitching stuff together.
 And a lot of it is just like intuition.
 But then you have like Alpha, I guess Zero be the first one that did the self-play.
 It just came up with this.
 It was able to learn through self-play mechanism, this kind of intuition.
 But just like you said, it's so fascinating to think
 whether in the space of totally new ideas, can that be done in developing theorems?
 We know it can be done by neural networks because we did it with the neural networks
 in the craniums of the great mathematicians of humanity.
 And I'm so glad you brought up Alpha Zero because that's the counter example.
 It turned out we were flattering ourselves when we said intuition is something different.
 Only humans can do it.
 It's not information processing.
 It used to be that way.
 Again, it's really instructive, I think, to compare the chess computer Deep Blue that
 beat Kasparov with Alpha Zero that beat Lisa Doll at the go.
 Because for Deep Blue, there was no intuition.
 There was some pro, humans had programmed in some intuition.
 After humans had played a lot of games, they told the computer, you know,
 count the pawn as one point, the bishop is three points, the rook is five points,
 and so on.
 You add it all up and then you add some extra points for past pawns and subtract if the
 opponent has it and blah, blah, blah, blah.
 And then what Deep Blue did was just search.
 Just very brute force and tried many, many moves ahead and all these combinations and
 a pruned tree search.
 And it could think much faster than Kasparov and it won.
 Right.
 And that, I think, inflated our egos in a way it shouldn't have because people started
 to say, yeah, yeah, it's just brute force search, but has no intuition.
 Yeah.
 Alpha zero really popped our bubble there because what Alpha Zero does, yes, it does
 also do some of that tree search, but it also has this intuition module, which in geek speak
 is called a value function, where it just looks at the board and comes up with a number
 for how good is that position.
 The difference was no human told it how good the position is.
 It just learned it.
 And Mu Zero is the coolest or scariest of all, depending on your mood, because it, the
 same basic AI system will learn what the good board position is, regardless of whether it's
 chess or go or Shogi or Pac-Man or Lady Pac-Man or Breakout or Space Invaders or any number,
 bunch of other games.
 You don't tell anything and it gets this intuition after a while for what's good.
 So this is very hopeful for science, I think, because if it can get intuition for what's
 a good position there, maybe it can also get intuition for what are some good directions
 to go if you're trying to prove something.
 I often, one of the more most fun things in my science career is when I've been able to
 prove some theorem about something and it's very heavily intuition guided.
 Of course, I don't sit and try all random strings.
 I have a hunch that this reminds me a little bit of about this other proof I've seen for
 this thing.
 So maybe I first, what if I try this?
 Nah, that didn't work out.
 But this reminds me actually the way this failed reminds me of that.
 And so combining the intuition with all these brute force capabilities, I think it's going
 to be able to help physics too.
 Do you think there'll be a day when an AI system being the primary contributor, let's
 say 90% plus wins the Nobel Prize in physics?
 Obviously they'll give it to the humans because we humans don't like to give prizes to
 machines.
 It'll give it to the humans behind the system.
 You could argue that AI has already been involved in some Nobel prizes, probably maybe
 something with black holes and stuff like that.
 Yeah, we don't like giving prizes to other life forms.
 If someone wins a horse racing contest, they don't give the prize to the horse either.
 That's true.
 But do you think we might be able to see something like that in our lifetimes when AI...
 So the first system I would say that makes us think about a Nobel Prize seriously is
 like Alpha Fold is making us think about in medicine and physiology, a Nobel Prize, perhaps
 discoveries that are a direct result of something that's discovered by Alpha Fold.
 Do you think in physics we might be able to see that in our lifetimes?
 I think what's probably going to happen is more of a blurring of the distinctions.
 So today if somebody uses a computer to do a computation that gives them the Nobel Prize,
 nobody's going to dream of giving the prize to the computer.
 They're going to be like, that was just a tool.
 I think for these things also people are just going to for a long time view the computer
 as a tool.
 But what's going to change is the ubiquity of machine learning.
 I think at some point in my lifetime, finding a human physicist who knows nothing about
 machine learning is going to be almost as hard as it is today finding a human physicist
 who doesn't say, oh, I don't know anything about computers or I don't use math.
 It would just be a ridiculous concept.
 But the thing is, there is a magic moment though, like with Alpha Zero, when the system
 surprises us in a way where the best people in the world truly learn something from the
 system in a way where you feel like it's another entity.
 Like the way people, the way Magnus Carlsen, the way certain people are looking at the
 work of Alpha Zero, it truly is no longer a tool in the sense that it doesn't feel like
 a tool.
 It feels like some other entity.
 So there is a magic difference where you're like, if an AI system is able to come up with
 an insight that surprises everybody in some major way that's a phase shift in our understanding
 of some particular science or some particular aspect of physics, I feel like that is no
 longer a tool.
 And then you can start to say that it perhaps deserves the prize.
 So for sure, the more important, the more fundamental transformation of the 21st century
 science is exactly what you're saying, which is probably everybody will be doing machine
 learning.
 It's to some degree, like if you want to be successful at unlocking the mysteries of science,
 you should be doing machine learning.
 But it's just exciting to think about like, whether there'll be one that comes along that's
 super surprising and they'll make us question like who the real inventors are in this world.
 Yeah.
 Yeah, I think the question of isn't if it's going to happen, but when, but it's important
 also in my mind, the time when that happens is also more or less the same time when we
 get artificial general intelligence.
 And then we have a lot bigger things to worry about than whether you get the Nobel Prize
 or not.
 Right.
 Because when you have machines that can outperform our best scientists at science, they can
 probably outperform us at a lot of other stuff as well, which can at a minimum, you know,
 make them incredibly powerful agents in the world.
 And I think it's a mistake to think we only have to start worrying about loss of control
 when machines get to AGI across the board, where they can do everything, all our jobs.
 Long before that, they'll be hugely influential.
 We talked at length about how the hacking of our minds with algorithms trying to get
 us glued to our screens, right, has already had a big impact on society.
 That was an incredibly dumb algorithm in the grand scheme of things, right?
 The supervised machine learning, yet that had huge impact.
 So I just don't want us to be lulled into a false sense of security and think there won't
 be any societal impact until things reach human level because it's happening already.
 I was just thinking the other week, you know, when I see some scaremonger going,
 oh, the robots are coming.
 The implication is always that they're coming to kill us.
 And maybe you should have worried about that if you were in Nagorno-Karabakh
 during the recent war there.
 But more seriously, the robots are coming right now, but they're mainly not coming to
 kill us. They're coming to hack us.
 They're coming to hack our minds into buying things that maybe we didn't need,
 to vote for people who may not have our best interests in mind.
 And it's kind of humbling, I think, actually, as a human being to admit that it turns out
 that our minds are actually much more hackable than we thought.
 And the ultimate insult is that we are actually getting hacked by
 the machine learning algorithms that are in some objective sense much dumber than us.
 But maybe we shouldn't be so surprised because, you know,
 how do you feel about cute puppies?
 Love them.
 So, you know, you would probably argue that in some across the board measure,
 you're more intelligent than they are.
 But boy, are our cute puppies good at hacking us, right?
 Yeah.
 They move into our house, persuade us to feed them and do all these things.
 What do they ever do for us?
 Yeah.
 Other than being cute and making us feel good, right?
 So if puppies can hack us, maybe we shouldn't be so surprised if
 pretty dumb machine learning algorithms can hack us too.
 Not to speak of cats, which is another level.
 And I think we should, to counter your previous point about there,
 let us not think about evil creatures in this world.
 We can all agree that cats are as close to objective evil as we can get.
 But that's just me saying that.
 OK.
 Have you seen that?
 The cartoon, I think it's maybe the onion where this incredibly cute kitten
 and it just says underneath something that thinks about murder all day.
 Exactly.
 That's accurate.
 You've mentioned offline that there might be a link between post biological AGI and SETI.
 So last time we talked, you've talked about this intuition.
 We humans might be quite unique in our galactic neighborhood, perhaps our galaxy,
 perhaps the entirety of the observable universe, who might be the only
 intelligent civilization here, which is.
 And you argue pretty well for that thought.
 So I have a few little questions around this.
 One, the scientific question, in which way would you be if you were wrong in that intuition?
 In which way do you think you would be surprised?
 Like, why were you wrong?
 We find out that you ended up being wrong.
 In which dimension?
 So like, is it because we can't see them?
 Is it because the nature of their intelligence or the nature of their life
 is totally different than we can possibly imagine?
 Is it because the I mean, something about the great filters and surviving them?
 Or maybe because we're being protected from signals?
 So all those explanations for.
 For why we haven't heard a big, loud, like red light that says we're here.
 Yeah.
 So there are actually two separate things there that I could be wrong about.
 Two separate claims that I made, right?
 None of them.
 One, one of them is.
 I made the claim, I think.
 Most civilizations.
 When you're going from simple bacteria like things to space, space colonizing civilizations,
 they spend only a very, very tiny fraction of their other, of their life being where we are.
 That I could be wrong about.
 The other one I could be wrong about is that it's quite different states.
 It's a very, very different state.
 It's a very, very different state.
 The other one I could be wrong about is that it's quite different statements that I think
 that actually I'm guessing that we are the only civilization in our observable universe
 from which light has reached us so far that's actually gotten far enough to invent telescopes.
 So let's talk about maybe both of them in turn, because they really are different.
 The first one, if you look at the N equals one, the data point we have on this planet.
 Yeah. So we spent four and a half billion years futzing around on this planet with life.
 And most of it was pretty lame stuff from an intelligence perspective.
 Bacteria and then the dinosaurs spent, but things gradually accelerated.
 Then the dinosaurs spent over a hundred million years stomping around here without even inventing
 smartphones. And then very recently, we've only spent 400 years going from Newton to us,
 right? In terms of technology. And look what we've done even when I was a little kid,
 there was no internet even. So I think it's pretty likely in this case of this planet,
 we're either going to really get our act together and start spreading life into space the century.
 And doing all sorts of great things, or we're going to wipe out. It's a little hard.
 I could be wrong in the sense that maybe what happened on this earth is very atypical. And for
 some reason, what's more common on other planets is that they spend an enormously long time futzing
 around with the ham radio and things, but they just never really take it to the next level.
 For reasons I haven't understood, I'm humble and open to that. But I would bet at least 10 to 1,
 that our situation is more typical because the whole thing with Moore's law and accelerating
 technology, it's pretty obvious why it's happening. Everything that grows exponentially,
 we call it an explosion, whether it's a population explosion or a nuclear explosion,
 it's always caused by the same thing. It's that the next step triggers a step after that.
 Today's technology enables tomorrow's technology and that enables the next level.
 And because the technology is always better, of course, the steps can come faster and faster.
 On the other question that I might be wrong about, that's the much more controversial one,
 I think. But before we close out on this thing about the first one, if it's true that most
 civilizations spend only a very short amount of their total time in the stage, say, between
 inventing telescopes or mastering electricity and doing space travel. If that's actually generally
 true, then that should apply also elsewhere out there. So we should be very, very surprised if we
 find some random civilization and we happen to catch them exactly in that very, very short stage.
 It's much more likely that they're going to be able to do that.
 Or that we find some civilization that's already post-biological and has done some really cool
 galactic construction projects in their galaxy. Would we be able to recognize them, do you think?
 Is it possible that we just can't? I mean, this post-biological world,
 could it be just existing in some other dimension? It could be just all a virtual reality game,
 for them, or something, I don't know, that it changes completely where we won't be able to detect?
 We have to be, honestly, very humble about this. I think I said earlier the number one
 principle being scientists is you have to be humble and willing to acknowledge that everything
 we think just might be totally wrong. Of course, you could imagine some civilization where they
 all decide to become Buddhists and very inward-looking, and they're all going to
 move into their little virtual reality and not disturb the flora and fauna around them, and we
 might not notice them. But this is a numbers game, right? If you have millions of civilizations out
 there, or billions of them, all it takes is one with a more ambitious mentality that decides,
 hey, we are going to go out and settle a bunch of other things that are going to be
 in our own galaxy, and then it doesn't matter if they're a bunch of quiet Buddhists, we're still
 going to notice that expansionist one, right? And it seems like quite the stretch to assume that,
 you know, we know even in our own galaxy that there are probably a billion or more
 planets that are pretty Earth-like, and many of them were formed over a billion years before ours,
 so it had a big head start. So if you actually assume also that life happens kind of automatically
 on an Earth-like planet, I think it's quite the stretch to then go and say, okay, so we have
 billions of other civilizations out there that also have our level of tech, and they all
 decided to become Buddhists, and not a single one decided to go Hitler on the galaxy and say,
 we need to go on and colonize, or not a single one decided for more benevolent reasons to go out
 and get more resources. That seems like a bit of a stretch, frankly, and this leads into the
 second thing you challenged me that I might be wrong about, how rare or common is life?
 So Francis Drake, when he wrote down the Drake equation, multiplied together a huge number of
 factors, and then we know that life is a very, very, very, very, very, very, very, very, very
 large number of factors, and we don't know any of them, so we know even less about what you get
 when you multiply together the whole product. Since then, a lot of those factors have become
 much better known. One of his big uncertainties was how common is it that a solar system even has a
 planet? Well, now we know it's very common. Earth-like planets, we know we have better...
 There are a dime a dozen, there are many, many of them, even in our galaxy. At the same time,
 I'm a big supporter of the SETI project and its cousins, and I think we should keep doing this,
 and we've learned a lot. We've learned that so far, all we have is still unconvincing hints,
 nothing more. There are certainly many scenarios where it would be dead obvious
 if there were 100 million other human-like civilizations in our galaxy, it would not be
 that hard to notice some of them with today's technology, and we haven't. What we can say is,
 well, okay, we can rule out that there is a human-level civilization on the Moon, and
 in fact, in many nearby solar systems, where we cannot rule out, of course, that there is something
 like Earth sitting in a galaxy five billion light-years away. But we've ruled out a lot,
 and that's already kind of shocking given that there are all these planets there. So where are
 they? Where are they all? That's the classic Fermi paradox. So my argument, which might very
 be wrong, it's very simple, really, it just goes like this. Okay, we have no clue about this.
 It could be the probability of getting life on a random planet, it could be 10 to the minus one,
 a priori, or 10 to the minus 10, 10 to the minus 20, 10 to the minus 30, 10 to the minus 40.
 Basically, every order of magnitude is about equally likely. When you then do the math and
 ask how close is our nearest neighbor, it's again equally likely that it's 10 meters away, 10 to 20
 meters away, 10 to the 30 meters away. We have some nerdy ways of talking about this with Bayesian
 statistics and a uniform log prior, but that's irrelevant. This is the simple basic argument.
 And now comes the data. So we can say, okay, how many were there? All these orders of magnitude,
 10 to the 26 meters away, there's the edge of our observable universe. If it's farther than that,
 light hasn't even reached us yet. If it's less than 10 to the 16 meters away, well, it's
 within Earth's range. It's no farther away than the sun. We can definitely rule that out.
 So I think about it like this. A priori, before we looked with telescopes,
 it could be 10 to 10 meters, 10 to 20, 10 to 30, 10 to 40, 10 to 50, 10 to blah, blah,
 equally likely anywhere here. And now we've ruled out this chunk.
 Yeah. And most of it is outside.
 And here is the edge of our observable universe already. So I'm certainly not saying I don't think
 there's any life elsewhere in space. If space is infinite, then you're basically 100% guaranteed
 that there is. But the probability that there is life, that the nearest neighbor happens to be in
 this little region between where we would have seen it already and where we will never see it,
 there's actually significantly less than one, I think. And I think there's a moral lesson from
 this, which is really important, which is to be good stewards of this planet and this shot we've
 had. It can be very dangerous to say, oh, it's fine if we nuke our planet or ruin the climate or
 mess it up with unaligned AI, because I know there is this nice Star Trek fleet out there.
 They're going to swoop in and take over where we failed. Just like it wasn't the big deal that the
 Easter Island losers wiped themselves out. That's a dangerous way of lulling yourself into false
 sense of security. If it's actually the case that it might be up to us and only us, the whole future
 of intelligent life in our observable universe, then I think it really puts a lot of responsibility
 on our shoulders. It's inspiring. It's a little bit terrifying, but it's also inspiring. But it's
 empowering, I think, most of all, because the biggest problem today is, and I see this even
 when I teach, so many people feel that it doesn't matter what they do or what we do. We feel
 disempowered. Oh, it makes no difference. This is about as far from that as you can come. But we
 realize that what we do on our little spinning ball here in our lifetime could make the difference
 for the entire future of life in our universe. How empowering is that?
 Yes, survival of consciousness. A very similar kind of empowering aspect of the Drake equation is,
 say there is a huge number of intelligent civilizations that spring up everywhere,
 but because of the Drake equation, which is the lifetime of a civilization, maybe many of them
 hit a wall. And just like you said, it's clear that for us, the great filter,
 the one possible great filter seems to be coming in the next 100 years. So it's also empowering to
 say, okay, well, we have a chance to not. I mean, the way great filters work, it does just get most
 of them. Exactly. Nick Bostrom has articulated this really beautifully, too. Every time yet another
 search for life on Mars comes back negative or something, I'm like, yes, yes. Our odds for
 us surviving. You already made the argument in broad brush there, right? But just to unpack it.
 The point is, we already know there is a crap ton of planets out there that are Earth-like,
 and we also know that most of them do not seem to have anything like our kind of life on them. So
 what went wrong? There's clearly one step along the evolutionary, at least one, filter roadblock
 in going from no life to spacefaring life. And where is it? Is it in front of us or is it behind
 us? Right? If there's no filter behind us, and we keep finding all sorts of little mice on
 Mars or whatever, right? That's actually very depressing because that makes it much more likely
 that the filter is in front of us. And what actually is going on is the ultimate dark joke
 that whenever a civilization invents sufficiently powerful tech, it just sets your clock, and then
 after a little while it goes poof for one reason or another and wipes itself out. Wouldn't that be
 utterly depressing if we're actually doomed? Whereas if it turns out that there is a great
 filter early on, that for whatever reason seems to be really hard to get to the stage of
 sexually reproducing organisms or even the first ribosome or whatever, right? Or maybe you have
 lots of planets with dinosaurs and cows, but for some reason they tend to get stuck there and never
 invent smartphones. All of those are huge boosts for our own odds because been there, done that.
 You know, it doesn't matter how hard or unlikely it was that we got past that roadblock because
 we already did. And then that makes it likely that the future is in our own hands. We're not
 doomed. So that's why I think the fact that life is rare in the universe, it's not just something
 that there is some evidence for, but also something we should actually hope for.
 So that's the end, the mortality, the death of human civilization that we've been discussing
 in life, maybe prospering beyond any kind of great filter. Do you think about your own death? Does it
 make you sad that you may not witness some of the, you lead a research group on working some of the
 biggest questions in the universe actually, both on the physics and the AI side? Does it make you
 sad that you may not be able to see some of these exciting things come to fruition that we've been
 talking about?
 Of course. Of course it sucks the fact that I'm going to die. I remember once when I was much
 younger, my dad made this remark that life is fundamentally tragic. And I'm like, what are you
 talking about daddy? And then many years later, I felt, now I feel I totally understand what he
 means. You know, we grow up, we're little kids and everything is infinite and it's so complicated.
 Everything is infinite and it's so cool. And then suddenly we find out that actually, you know,
 you got to serve only this, this is the, you can get game over at some point. So of course it's,
 it's something that's sad. Are you afraid?
 No, not in the sense that I think anything terrible is going to happen after I die or
 anything like that. No, I think it's really going to be a game over, but it's more that
 it makes me very acutely aware of what a wonderful gift this is that it gets to be alive right now.
 And is a steady reminder to just live life to the fullest and really enjoy it because it is finite.
 You know, and I think actually, and we know we all get the regular reminders when someone near
 and dear to us dies that, that, you know, one day it's going to be our turn. It adds this kind of
 focus. I wonder what it would feel like actually to be an immortal being if they might even enjoy
 some of the wonderful things of life, live it less just because there isn't that.
 Finiteness. Do you think that could be a feature, not a bug, the fact that we,
 beings are finite. Maybe there's lessons for engineering and artificial intelligence systems
 as well that are conscious. Like, do you think it makes, is it possible that the reason the
 pistachio ice cream is delicious is the fact that you're going to die one day and you will not have
 all the pistachio ice cream that you could eat because of that fact.
 Well, let me say two things. First of all, it's actually quite profound what you're saying. I do
 think I appreciate the pistachio ice cream a lot more knowing that I will, there's only a finite
 number of times I get to enjoy that. And I can only remember finite number of times in the past.
 And moreover, my life is not so long that it just starts to feel like things are repeating themselves
 in general. It's so new and fresh. I also think though that death is a little bit overrated in
 the sense that it comes from a sort of outdated view of physics and what life actually is.
 Because if you ask, okay, what is it that's going to die exactly? What am I really? When I say it,
 I feel sad about the idea of myself dying. Am I really sad that this skin cell here is going to
 die? Of course not, because it's going to die next week anyway, and I'll grow a new one.
 And it's not any of my cells that I'm associating really with who I really am,
 nor is it any of my atoms or quarks or electrons. In fact, basically all of my atoms get replaced
 on a regular basis. So what is it that's really me from a more modern physics perspective? It's
 the information in processing Amy. That's my memories, that's my values, my dreams, my passion,
 my love. That's what's really fundamentally me. And frankly, not all of that will die
 when my body dies. Richard Feynman, for example, his body died of cancer, but many of his ideas
 that he felt made him very him actually live on. This is my own little personal tribute to Richard
 Feynman. I try to keep a little bit of him alive in myself. I've even quoted him today.
 Yeah, he almost came alive for a brief moment in this conversation.
 Yeah. And this honestly gives me some solace. When I work as a teacher, I feel if I can actually share
 a bit about myself, that my students feel worthy enough to copy and adopt
 as a part of things that they know or they believe or aspire to, now I live on also a little bit in
 them. Being a teacher is a little bit of what I... That's something also that contributes to making
 me a little teeny bit less mortal, because I'm not at least not all going to die all at once.
 And I find that a beautiful tribute to people we do not respect. If we can remember them and carry
 in us the things that we felt was the most awesome about them, then they live on.
 And I'm getting a bit emotional here, but it's a very beautiful idea you bring up there.
 I think we should stop this old fashioned materialism and just equate who we are with
 our quarks and electrons. There's no scientific basis for that really. And it's also very uninspiring.
 Now, if you look a little bit towards the future, one thing which really sucks about humans dying
 is that even though some of their teachings and memories and stories and ethics and so on will be
 copied by those around them, hopefully, a lot of it can't be copied and just dies with them, with
 their brain. And that really sucks. That's the fundamental reason why we find it so tragic
 when someone goes from having all this information there to the more just gone, ruined.
 With more post-biological intelligence, that's going to shift a lot. The only reason it's so
 hard to make a backup of your brain in its entirety is exactly because it wasn't built for that.
 If you have a future machine intelligence, there's no reason for why it has to die at all.
 If you want to copy it, whatever it is, into some other quark blob, you can copy not just some of
 it, but all of it. In that sense, you can get immortality because all the information can be
 copied out of any individual entity. It's not just mortality that will change if we get to
 more post-biological life. It's also, with that, very much the whole individualism we have now.
 The reason that we make such a big difference between me and you is exactly because we're a
 little bit limited in how much we can copy. I would just love to go like this and copy your
 Russian skills, Russian-speaking skills. Wouldn't it be awesome? But I can't. I have to actually
 work for years if I want to get better on it. But if we were robots, just copying and pasting freely,
 then that completely washes away the sense of what immortality is.
 And also individuality a little bit. Maybe we would feel much more collaborative with each other.
 You can give me your Russian and I'll give you whatever. Suddenly you can speak Swedish. Maybe
 that's a bad trade for you, but whatever else you want from my brain. There have been a lot of sci-fi
 stories about hive minds and so on where people, where experiences can be more broadly shared.
 And I think one, we don't, I don't pretend to know what it would feel like to be a
 super intelligent machine, but I'm quite confident that
 however it feels about mortality and individuality will be very, very different from how it is for us.
 Well, for us, mortality and finiteness seems to be pretty important at this particular moment.
 And so all good things must come to an end. Just like this conversation. I saw that coming.
 Sorry, this is the world's worst transition. I could talk to you forever. It's such a huge honor
 that you spend time with me. Honor is mine. Thank you so much for getting me essentially to start
 this podcast by doing the first conversation, making me realize falling in love with conversation
 in itself. And thank you so much for inspiring so many people in the world with your books,
 with your research, with your talking and with the other, like this ripple effect of friends,
 including Elon and everybody else that you inspire. So thank you so much for talking to me.
 Thank you. I feel so fortunate that you're doing this podcast and getting so many
 interesting voices out there into the ether and not just the five second sound bites,
 but so many of the interviews are what you do. You really let people go in
 into depth in a way which we sorely need in this day and age. And that I got to be number one,
 I feel super honored. Yeah, you started it. Thank you so much, Max. Thanks for listening to this
 conversation with Max Tegmark and thank you to our sponsors, The Jordan Harbinger Show,
 Four Sigmatic Mushroom Coffee, Better Help Online Therapy, and ExpressVPN. So the choice is wisdom,
 whether it's between sanity or privacy. Choose wisely, my friends. And if you wish, click the
 sponsor links below to get a discount and to support this podcast. And now let me leave you
 with some words from Max Tegmark. If consciousness is the way that information feels when it's
 processed in certain ways, then it must be substrate independent. It's only the structure
 of the matter. It's not the structure of the matter doing the information processing.
 Thank you for listening and hope to see you next time.