As part of MIT course 6S099 on artificial general intelligence, I got a chance to sit down with Christoph Koch, who is one of the seminal figures in neurobiology, neuroscience, and generally in the study of consciousness. He is the president, the chief scientific officer of the Allen Institute for Brain Science in Seattle. From 1986 to 2013, he was a professor at Caltech. Before that, he was at MIT, he is extremely well cited, over 100,000 citations. His research, his writing, his ideas have had big impact on the scientific community and the general public in the way we think about consciousness, in the way we see ourselves as human beings. He's the author of several books, The Quest for Consciousness and Neurobiological Approach, and a more recent book, Consciousness, Confessions of a Romantic Reductionist. If you enjoy this conversation, this course, subscribe, click the little bell icon to make sure you never miss a video, and in the comments, leave suggestions for any people you'd like to see be part of the course or any ideas that you would like us to explore. Thanks very much and I hope you enjoy. Okay, before we delve into the beautiful mysteries of consciousness, let's zoom out a little bit and let me ask, do you think there's intelligent life out there in the universe? Yes, I do believe so. We have no evidence of it, but I think the probabilities are overwhelming in favor of it. Given a universe where we have 10 to the 11 galaxies and each galaxy has between 10 to the 11, 10 to the 12 stars and we know most stars have one or more planets. So how does that make you feel? It still makes me feel special because I have experiences. I feel the world, I experience the world and independent of whether there are other creatures out there, I still feel the world and I have access to this world in this very strange compelling way and that's the core of human existence. Now, you said human, do you think if those intelligent creatures are out there, do you think they experience their world? Yes, if they are evolved, if they are a product of natural evolution as they would have to be, they will also experience their own world. The consciousness isn't just human, you're right, it's much wider. It may be spread across all of biology. The only thing that we have special is we can talk about it. Of course, not all people can talk about it. Babies and little children can talk about it. Patients who have a stroke in the left inferior frontal gyrus can talk about it, but most normal adult people can talk about it and so we think that makes us special compared to let's say monkeys or dogs or cats or mice or all the other creatures that we share the planet with, but all the evidence seems to suggest that they too experience the world and so it's overwhelmingly likely that aliens would also experience their world. Of course, differently because they have a different sensorium, they have different sensors, they have a very different environment, but the fact that I would strongly suppose that they also have experiences. They feel pain and pleasure and see in some sort of spectrum and hear and have all the other senses. Of course, their language, if they have one, would be different so we might not be able to understand their poetry about the experiences that they have. That's correct. So in a talk, in a video, I've heard you mention Siputzo, a dachshund that you came up with, that you grew up with, it was part of your family when you were young. First of all, you're technically a Midwestern boy. You just – Technically. Yes. But after that, you traveled around a bit, hence a little bit of the accent. You talked about Siputzo, the dachshund, having these elements of humanness, of consciousness that you discovered. So I just wanted to ask, can you look back in your childhood and remember when was the first time you realized you yourself, sort of from a third person perspective, are a conscious being? This idea of stepping outside yourself and seeing there's something special going on here in my brain. I can't really actually – it's a good question. I'm not sure I recall a discrete moment. I mean, you take it for granted because that's the only world you know. The only world I know and you know is the world of seeing and hearing voices and touching and all the other things. So it's only much later at early – in my underguided days when I enrolled in physics and in philosophy that I really thought about it and thought, well, this is really fundamentally very, very mysterious and there's nothing really in physics right now that explains this transition from the physics of the brain to feelings. Where do the feelings come in? So you can look at the foundational equation of quantum mechanics, general relativity. You can look at the periodic table of the elements. You can look at the endless ATGC chat in our genes and nowhere is consciousness. Yet I wake up every morning to a world where I have experiences. And so that's the heart of the ancient mind body problem. How do experiences get into the world? So what is consciousness? Experience. This is any experience. Some people call it subjective feeling. Some people call it phenomenology. Some people call it qualia of the philosopher. But they all denote the same thing. It feels like something in the famous word of the philosopher Thomas Nagel. It feels like something to be a bat or to be an American or to be angry or to be sad or to be in love or to have pain. And that is what experience is, any possible experience. Could be as mundane as just sitting in a chair. Could be as exalted as having a mystical moment in deep meditation. Those are just different forms of experiences. Experience. So if you were to sit down with maybe the next, skip a couple generations, of IBM Watson, something that won Jeopardy, what is the gap, I guess the question is, between Watson, that might be much smarter than you, than us, than any human alive, but may not have experience, what is the gap? Well, so that's a big, big question. That's occupied people for the last, certainly last 50 years since we, you know, since the advent, the birth of computers. That's a question Alan Turing tried to answer. And of course he did it in this indirect way by proposing a test, an operational test. But that's not really, that's, you know, he tried to get at what does it mean for a person to think, and then he had this test, right? You lock them away, and then you have a communication with them, and then you try to guess after a while whether that is a person or whether it's a computer system. There's no question that now or very soon, you know, Alexa or Siri or, you know, Google now will pass this test, right? And you can game it, but you know, ultimately, certainly in your generation, there will be machines that will speak with complete poise that will remember everything you ever said. They'll remember every email you ever had, like Samantha, remember in the movie Her? Yeah. There's no question it's going to happen. But of course, the key question is, does it feel like anything to be Samantha in the movie Her? Or does it feel like anything to be Watson? And there one has to very, very strongly think there are two different concepts here that we co mingle. There is the concept of intelligence, natural or artificial, and there is a concept of consciousness, of experience, natural or artificial. Those are very, very different things. Now, historically, we associate consciousness with intelligence. Why? Because we live in a world, leaving aside computers, of natural selection, where we're surrounded by creatures, either our own kin that are less or more intelligent, or we go across species. Some are more adapted to a particular environment. Others are less adapted, whether it's a whale or dog, or you go talk about a paramecium or a little worm. And we see the complexity of the nervous system goes from one cell to specialized cells, to a worm that has three nets, that has 30 percent of its cells are nerve cells, to creature like us or like a blue whale that has 100 billion, even more nerve cells. And so based on behavioral evidence and based on the underlying neuroscience, we believe that as these creatures become more complex, they are better adapted to their particular ecological niche, and they become more conscious, partly because their brain grows. And we believe consciousness, unlike the ancient, ancient people thought most, almost every culture thought that consciousness with intelligence has to do with your heart. And you still see that today. You see, honey, I love you with all my heart. But what you should actually say is, no, honey, I love you with all my lateral hypothalamus. And for Valentine's Day, you should give your sweetheart, you know, hypothalamus, a piece of chocolate and not a heart shaped chocolate. Anyway, so we still have this language, but now we believe it's a brain. And so we see brains of different complexity and we think, well, they have different levels of consciousness. They're capable of different experiences. But now we confront the world where we know where we're beginning to engineer intelligence. And it's radical unclear whether the intelligence we're engineering has anything to do with consciousness and whether it can experience anything. Because fundamentally, what's the difference? Intelligence is about function. Intelligence no matter exactly how you define it, sort of adaptation to new environments, being able to learn and quickly understand, you know, the setup of this and what's going on and who are the actors and what's going to happen next. That's all about function. Consciousness is not about function. Consciousness is about being. It's in some sense much fundamental. You can see this in several cases. You can see it, for instance, in the case of the clinic. When you're dealing with patients who are, let's say, had a stroke or had were in traffic accident, et cetera, they're pretty much immobile. Terri Schiavo, you may have heard historically, she was a person here in the 90s in Florida. Her heart stood still. She was reanimated. And then for the next 14 years, she was what's called in a vegetative state. So there are thousands of people in a vegetative state. So they're, you know, they're, you know, they're like this. Occasionally, they open their eyes for two, three, four, five, six, eight hours, and then close their eyes. They have sleep wake cycle. Occasionally, they have behaviors. They do like, you know, but there's no way that you can establish a lawful relationship between what you say or the doctor says or the mom says and what the patient does. So there isn't any behavior, yet in some of these people, there is still experience. You can design and build brain machine interfaces where you can see there's still experience something. And of course, these cases of locked in state, there's this famous book called The Diving Bell and the Butterfly, where you had an editor, a French editor, he had a stroke in the brainstem, unable to move except his vertical eyes, eye movement. He could just move his eyes up and down. And he dictated an entire book. And some people even lose this at the end. All the evidence seems to suggest that they're still in there. In this case, you have no behavior, you have consciousness. Second case is tonight, like all of us, you're going to go to sleep, close your eyes, you go to sleep, you will wake up inside your sleeping body, and you will have conscious experiences. They are different from everyday experience. You might fly, you might not be surprised that you're flying, you might meet a long dead pet, childhood dog, and you're not surprised that you're meeting them. But you have conscious experience of love, of hate, they can be very emotional. Your body during this state, typically it's REM state, sends an active signal to your motor neurons to paralyze you. It's called atonia. Because if you don't have that, like some patients, what do you do? You act out your dreams. You get, for example, REM behavioral disorder, which is bad juju to get. Okay. Third case is pure experience. So I recently had this, what some people call a mystical experience. I went to Singapore and went into a flotation tank. Yeah. All right. So this is a big tub filled with water, that's body temperature and Epsom salt. You strip completely naked, you lie inside of it, you close the lid. Darkness. Complete darkness, soundproof. So very quickly, you become bodiless because you're floating and you're naked. You have no rings, no watch, no nothing. You don't feel your body anymore. There's no sound, soundless. There's no photon, sightless, timeless, because after a while, early on you actually hear your heart, but then you sort of adapt to that and then sort of the passage of time ceases. Yeah. And if you train yourself, like in a meditation, not to think, early on you think a lot. It's a little bit spooky. You feel somewhat uncomfortable or you think, well, I'm going to get bored. And if you try to not to think actively, you become mindless. There you are, bodiless, timeless, you know, soundless, sightless, mindless, but you're in a conscious experience. You're not asleep. Yeah. You're not asleep. You are a being of pure, you're a pure being. There isn't any function. You aren't doing any computation. You're not remembering. You're not projecting. You're not planning. Yet you are fully conscious. You're fully conscious. There's something going on there. It could be just a side effect. So what is the... You mean epiphenomena. So what's the select, meaning why, what is the function of you being able to lay in this sensory free deprivation tank and still have a conscious experience? Evolutionary? Evolutionary. Obviously we didn't evolve with flotation tanks in our environment. I mean, so biology is notoriously bad at asking why question, telenormical question. Why do we have two eyes? Why don't we have four eyes like some teachers or three eyes or something? Well, no, there's probably, there is a function to that, but we're not very good at answering those questions. We can speculate endlessly where biology is very, or science is very good about mechanistic question. Why is there a charge in the universe? Right? We find a certain universe where there are positive and negative charges. Why? Why does quantum mechanics hold? You know, why doesn't some other theory hold? Quantum mechanics holding our universe is very unclear why. So telenormical question, why questions are difficult to answer. There's some relationship between complexity, brain processing power and consciousness. But however, in these cases, in these three examples I gave, one is an everyday experience at night. The other one is trauma. And third one is in principle, you can, everybody can have these sort of mystical experiences. You have a dissociation of function from, of intelligence from consciousness. You caught me asking a why question. Let me ask a question that's not a why question. You're giving a talk later today on the Turing test for intelligence and consciousness, drawing lines between the two. So is there a scientific way to say there's consciousness present in this entity or not? And to anticipate your answer, cause you, you will also, there's a neurobiological answer. So we can test the human brain, but if you take a machine brain that you don't know tests for yet, how would you even begin to approach a test if there's consciousness present in this thing? Okay. That's a really good question. So let me take it in two steps. So as you point out for, for, for, for humans, let's just stick with humans. There's now a test called the Zap and Zip is a procedure where you ping the brain using transcranial magnetic stimulation. You look at the electrical reverberations essentially using EG, and then you can measure the complexity of this brain response. And you can do this in awake people, in asleep, normal people, you can do it in awake people and then anesthetize them. You can do it in patients. And it, it, it has a hundred percent accuracy that in all those cases, when you're clear, the patient or the person is either conscious or unconscious, the complexity is either high or low. And then you can adopt these techniques to similar creatures like monkeys and dogs and, and, and mice that have very similar brains. Now of course you, you point out that may not help you because we don't have a cortex, you know, and if I send a magnetic pulse into my iPhone or my computer, it's probably going to break something. So we don't have that. So what we need ultimately, we need a theory of consciousness. We can't just rely on our intuition. Our intuition is, well, yeah, if somebody talks, they're conscious. However, then there are all these patients, children, babies don't talk, right? But we believe that, that the babies also have conscious experiences, right? And then there are all these patients I mentioned and they don't talk. When you dream, you can't talk because you're paralyzed. So what we ultimately need, we can't just rely on our intuition. We need a theory of conscience that tells us what is it about a piece of matter? What is it about a piece of highly excitable matter like the brain or like a computer that gives rise to conscious experience? We all believe, none of us believes anymore in the old story. It's a soul, right? That used to be the most common explanation that most people accept that instill a lot of people today believe, well, there's, there's God endowed only us with a special thing that animals don't have. Rene Descartes famously said, a dog, if you hit it with your carriage may yell, may cry, but it doesn't have this special thing. It doesn't have the magic, the magic soul. It doesn't have res cogitans, the soul. Now we believe that isn't the case anymore. So what is the difference between brains and, and these guys, silicon? And in particular, once their behavior matches. So if you have Siri or Alexa in 20 years from now that she can talk just as good as any possible human, what grounds do you have to say she's not conscious in particular, if she says it's of course she will, well, of course I'm conscious. You ask her how are you doing? And she'll say, well, you know, they, they'll generate some way to, of course she'll behave like a, like a person. Now there's several differences. One is, so this relates to the problem, the very hard, why is consciousness a hard problem? It's because it's subjective, right? Only I have it, for only I know I have direct experience of my own consciousness. I don't have experience in your consciousness. Now I assume as a sort of a Bayesian person who believes in probability theory and all of that, you know, I can do, I can do an abduction to the, to the best available facts. I deduce your brain is very similar to mine. If I put you in a scanner, your brain is roughly going to behave the same way as I do. If, if, if, you know, if I give you this muesli and ask you, how does it taste? You tell me things that, you know, that, that I would also say more or less, right? So I infer based on all of that, that you're conscious. Now with theory, I can't do that. So there I really need a theory that tells me what is it about, about any system, this or this, that makes it conscious. We have such a theory. Yes. So the integrated information theory, but let me first, maybe as an introduction for people who are not familiar, Descartes, can you, you talk a lot about pan, panpsychism. Can you describe what, uh, physicalism versus dualism? This you, you mentioned the soul, what, what is the history of that idea? What is the idea of panpsychism or no, the debate really, uh, out of which panpsychism can, um, emerge of, of, of, um, dualism versus, uh, physicalism or do you not see panpsychism as fitting into that? No, you can argue there's some, okay, so let's step back. So panpsychism is a very ancient belief that's been around, uh, I mean, Plato and Aristotle talks about it, uh, modern philosophers talk about it. Of course, in Buddhism, the idea is very prevalent that, I mean, there are different versions of it. One version says everything is ensouled, everything, rocks and stones and dogs and people and forest and iPhones, all of us all, right? All matter is ensouled. That's sort of one version. Another version is that all biology, all creatures, small or large, from a single cell to a giant sequoia tree feel like something. This one I think is somewhat more realistic. Um, so the different versions, what do you mean by feel like something, have, have feelings, have some kind of, it feels like something, it may well be possible that it feels like something to be a paramecium. I think it's pretty likely it feels like something to be a bee or a mouse or a dog. Sure. So, okay. So, so that you can see that's also, so panpsychism is very broad and you can, so some people, for example, Bertrand Russell, tried to advocate this, this idea, it's called Rasselian Monism, that that panpsychism is really physics viewed from the inside. So the idea is that physics is very good at describing relationship among objects like charges or like gravity, right? You know, describe the relationship between curvature and mass distribution, okay? That's the relationship among things. Physics doesn't really describe the ultimate reality itself. It's just relationship among, you know, quarks or all these other stuff from like a third person observer. Yes. Yes. Yes. And consciousness is what physics feels from the inside. So my conscious experience, it's the way the physics of my brain, particularly my cortex feels from the inside. And so if you are paramecium, you got to remember, you say paramecium, well, that's a pretty dumb creature. It is, but it has already a billion different molecules, probably, you know, 5,000 different proteins assembled in a highly, highly complex system that no single person, no computer system so far on this planet has ever managed to accurately simulate. Its complexity vastly escapes us. Yes. And it may well be that that little thing feels like a tiny bit. Now, it doesn't have a voice in the head like me. It doesn't have expectations. You know, it doesn't have all that complex things, but it may well feel like something. Yeah. So this is really interesting. Can we draw some lines and maybe try to understand the difference between life, intelligence and consciousness? How do you see all of those? If you had to define what is a living thing, what is a conscious thing and what is an intelligent thing? Do those intermix for you or are they totally separate? Okay. So A, that's a question that we don't have a full answer to. A lot of the stuff we're talking about today is full of mysteries and fascinating ones, right? For example, you can go to Aristotle, who's probably the most important scientist and philosopher who's ever lived in, certainly in Western culture. He had this idea, it's called hylomorphism. It's quite popular these days, that there are different forms of soul. The soul is really the form of something. He says, all biological creatures have a vegetative soul. That's life principle. Today, we think we understand something more than it is biochemistry and nonlinear thermodynamics. Then he said they have a sensitive soul. Only animals and humans have also a sensitive soul or a petitive soul. They can see, they can smell, and they have drives. They want to reproduce, they want to eat, et cetera. And then only humans have what he called a rational soul, okay? And that idea then made it into Christendom and then the rational soul is the one that lives forever. He was very unclear. He wasn't really, I mean, different readings of Aristotle give different, whether did he believe that rational soul was immortal or not. I probably think he didn't. But then, of course, that made it through Plato into Christianity, and then this soul became immortal and then became the connection to God. So you ask me, essentially, what is our modern conception of these three, Aristotle would have called them different forms. Life, we think we know something about it, at least life on this planet, right? Although we don't understand how to originate it, but it's been difficult to rigorously pin down. You see this in modern definitions of death. In fact, right now, there's a conference ongoing, again, that tries to define legally and medically what is death. It used to be very simple. Death is you stop breathing, your heart stops beating, you're dead, totally uncontroversial. If you're unsure, you wait another 10 minutes. If the patient doesn't breathe, he's dead. Well, now we have ventilators, we have heart pacemakers, so it's much more difficult to define what death is. Typically, death is defined as the end of life and life is defined before death. Okay, so we don't have really very good definitions. Intelligence, we don't have a rigorous definition. We know something how to measure, it's called IQ or G factors, right? And we're beginning to build it in a narrow sense, right? Like go, AlphaGo and Watson and, you know, Google cars and Uber cars and all of that, it's still narrow AI and some people are thinking about artificial general intelligence. But roughly, as we said before, it's something to do with ability to learn and to adapt to new environments. But that is, as I said, also, it's radical difference from experience. And it's very unclear if you build a machine that has AGI, it's not at all a priori, it's not at all clear that this machine will have consciousness, it may or may not. So let's ask it the other way, do you think if you were to try to build an artificial general intelligence system, do you think figuring out how to build artificial consciousness would help you get to an AGI? So or put another way, do you think intelligent requires consciousness? In human, it goes hand in hand. In human, or I think in biology, consciousness, intelligence goes hand in hand, quay is illusion because the brain evolved to be highly complex, complexity via the theory integrated information theory is sort of ultimately is what is closely tied to consciousness. Ultimately it's causal power upon itself. And so in evolved systems, they go together. In artificial system, particularly in digital machines, they do not go together. And if you ask me point blank, is Alexa 20.0 in the year 2040, when she can easily pass every Turing test, is she conscious? No, even if she claims she's conscious. In fact, you could even do a more radical version of this thought experiment. You can build a computer simulation of the human brain. You know what Henry Markham in the Blue Brain Project or the Human Brain Project in Switzerland is trying to do. Let's grant them all the success. So in 10 years, we have this perfect simulation of the human brain. Every neuron is simulated and it has a larynx and it has motor neurons. It has a Broca's area and of course they'll talk and they'll say, hi, I just woke up. I feel great. OK, even that computer simulation that can in principle map onto your brain will not be conscious. Why? Because it simulates, it's a difference between the simulated and the real. So it simulates the behavior associated with consciousness. It might be, it will, if it's done properly, will have all the intelligence that that particular person they're simulating has. But simulating intelligence is not the same as having conscious experiences. And I give you a really nice metaphor that engineers and physicists typically get. I can write down Einstein's field equation, nine or ten equations that describe the link in general relativity between curvature and mass. I can do that. I can run this on my laptop to predict that the central, the black hole at the center of our galaxy will be so massive that it will twist space time around it so no light can escape. It's a black hole. But funny, have you ever wondered why doesn't this computer simulation suck me in? It simulates gravity, but it doesn't have the causal power of gravity. That's a huge difference. So it's a difference between the real and the simulator, just like it doesn't get wet inside a computer when the computer runs code that simulates a weather storm. And so in order to have, to have artificial consciousness, you have to give it the same causal power as the human brain. You have to build so called a neuromorphic machine that has hardware that is very similar to the human brain, not a digital clocked phenomenon computer. So that's, just to clarify though, you think that consciousness is not required to create human level intelligence. It seems to accompany in the human brain, but for machine not. That's correct. So maybe just because this is AGI, let's dig in a little bit about what we mean by intelligence. So one thing is the G factor, these kind of IQ tests of intelligence. But I think if you, maybe another way to say, so in 2040, 2050, people will have Siri that is just really impressive. Do you think people will say Siri is intelligent? Yes. Intelligence is this amorphous thing. So to be intelligent, it seems like you have to have some kind of connections with other human beings in a sense that you have to impress them with your intelligence. And there feels, you have to somehow operate in this world full of humans. And for that, there feels like there has to be something like consciousness. So you think you can have just the world's best natural NLP system, natural language understanding generation, and that will be, that will get us happy and say, you know what, we've created an AGI. I don't know happy, but yes, I do believe we can get what we call high level functional intelligence, particular sort of the G, you know, this fluid like intelligence that we cherish, particularly at a place like MIT, right, in machines. I see a priori no reasons, and I see a lot of reason to believe it's going to happen very, you know, over the next 50 years or 30 years. So for beneficial AI, for creating an AI system that's, so you mentioned ethics, that is exceptionally intelligent but also does not do, does, you know, aligns its values with our values as humanity. Do you think then it needs consciousness? Yes, I think that that is a very good argument that if we're concerned about AI and the threat of AI, a la Nick Bostrom, existentialist threat, I think having an intelligence that has empathy, right, why do we find abusing a dog, why do most of us find that abhorrent, abusing any animal, right? Why do we find that abhorrent because we have this thing called empathy, which if you look at the Greek really means feeling with, I feel a path of empathy, I have feeling with you. I see somebody else suffer that isn't even my conspecific, it's not a person, it's not my wife or my kids, it's a dog, but I feel naturally most of us, not all of us, most of us will feel emphatic. And so it may well be in the long term interest of survival of homo sapiens sapiens that if we do build AGI and it really becomes very powerful that it has an emphatic response and doesn't just exterminate humanity. So as part of the full conscious experience to create a consciousness, artificial or in our human consciousness, do you think fear, maybe we're going to get into the earlier days with Nietzsche and so on, but do you think fear and suffering are essential to have consciousness? Do you have to have the full range of experience to have a system that has experience or can you have a system that only has very particular kinds of very positive experiences? Look you can have in principle, people have done this in the rat where you implant an electrode in the hypothalamus, the pleasure center of the rat and the rat stimulates itself above and beyond anything else. It doesn't care about food or natural sex or drink anymore, it just stimulates itself because it's such a pleasurable feeling. I guess it's like an orgasm just you have all day long. And so a priori I see no reason why you need a great variety. Now clearly to survive that wouldn't work, right? But if I'd engineered artificially, I don't think you need a great variety of conscious experience. You could have just pleasure or just fear. It might be a terrible existence, but I think that's possible at least on conceptual logical ground. Because any real creature whether artificially engineered, you want to give it fear, the fear of extinction that we all have. And you also want to give it positive repetitive states, states that you want the machine encouraged to do because they give the machine positive feedback. So you mentioned panpsychism, to jump back a little bit, everything having some kind of mental property. How do you go from there to something like human consciousness? So everything having some elements of consciousness, is there something special about human consciousness? So it's not everything. Like a spoon, the form of panpsychism I think about doesn't ascribe consciousness to anything like this, the spoon on my liver. However, the theory, the integrated information theory does say that the system, even one that looks from the outside relatively simple, at least if they have this internal causal power, it does feel like something. The theory a priori doesn't say anything what's special about human. Biologically we know the one thing that's special about human is we speak and we have an overblown sense of our own importance. We believe we're exceptional and we're just God's gift to the universe. But behaviorally the main thing that we have, we can plan over the long term, we have language and that gives us an enormous amount of power and that's why we are the current dominant species on the planet. So you mentioned God, you grew up a devout Roman Catholic family, so with consciousness you're sort of exploring some really deeply fundamental human things that religion also touches on. Where does religion fit into your thinking about consciousness? You've grown throughout your life and changed your views on religion as far as I understand. Yeah, I mean I'm now much closer to, I'm not a Roman Catholic anymore, I don't believe there's sort of this God, the God I was educated to believe in, sits somewhere in the fullness of time, I'll be united in some sort of everlasting bliss, I just don't see any evidence for that. Look, the world, the night is large and full of wonders, there are many things that I don't understand, I think many things that we as a cult, look we don't even understand more than 4% of all the universe, dark matter, dark energy, we have no idea what it is, maybe it's lost socks, what do I know? So all I can tell you is it's sort of my current religious or spiritual sentiment is much closer to some form of Buddhism, without the reincarnation unfortunately, there's no evidence for it than reincarnation. So can you describe the way Buddhism sees the world a little bit? Well so they talk about, so when I spent several meetings with the Dalai Lama and what always impressed me about him, he really, unlike for example let's say the Pope or some Cardinal, he always emphasized minimizing the suffering of all creatures. So they have this, from the early beginning they look at suffering in all creatures, not just in people, but in everybody, this universal and of course by degrees, an animal in general is less capable of suffering than a well developed, normally developed human and they think consciousness pervades in this universe and they have these techniques, you can think of them like mindfulness etc. and meditation that tries to access what they claim of this more fundamental aspect of reality. I'm not sure it's more fundamental, I think about it, there's the physical and then there's this inside view, consciousness and those are the two aspects that's the only thing I have access to in my life and you've got to remember my conscious experience and your conscious experience comes prior to anything you know about physics, comes prior to knowledge about the universe and atoms and super strings and molecules and all of that. The only thing you directly are acquainted with is this world that's populated with things in images and sounds in your head and touches and all of that. I actually have a question, so it sounds like you kind of have a rich life, you talk about rock climbing and it seems like you really love literature and consciousness is all about experiencing things, so do you think that has helped your research on this topic? Yes, particularly if you think about it, the various states, so for example when you do rock climbing or now I do rowing, crew rowing and a bike every day, you can get into this thing called the zone and I've always wanted about it, particularly with respect to consciousness because it's a strangely addictive state. Once people have it once, they want to keep on going back to it and you wonder what is it so addicting about it and I think it's the experience of almost close to pure experience because in this zone, you're not conscious of inner voice anymore, there's always inner voice nagging you, you have to do this, you have to do that, you have to pay your taxes, you have to fight with your ex and all of those things, they're always there. But when you're in the zone, all of that is gone and you're just in this wonderful state where you're fully out in the world, you're climbing or you're rowing or biking or doing soccer or whatever you're doing and sort of consciousness is this, you're all action or in this case of pure experience, you're not action at all but in both cases, you experience some aspect of conscious, you touch some basic part of conscious existence that is so basic and so deeply satisfying. You I think you touch the root of being, that's really what you're touching there, you're getting close to the root of being and that's very different from intelligence. So what do you think about the simulation hypothesis, simulation theory, the idea that we all live in a computer simulation? Rapture for nerds. Rapture for nerds. I think it's as likely as the hypothesis had engaged hundreds of scholars for many centuries, are we all just existing in the mind of God? And this is just a modern version of it, it's equally plausible. People love talking about these sort of things, I know they're book written about this simulation hypothesis, if that's what people want to do, that's fine, it seems rather esoteric, it's never testable. But it's not useful for you to think of in those terms, so maybe connecting to the questions of free will which you've talked about, I vaguely remember you saying that the idea that there's no free will, it makes you very uncomfortable. So what do you think about free will from a physics perspective, from a conscious perspective, what does it all fit? Okay, so from the physics perspective, leaving aside quantum mechanics, we believe we live in a fully deterministic world, right? But then comes of course quantum mechanics, so now we know that certain things are in principle not predictable, which as you said I prefer, because the idea that the initial condition of the universe and then everything else, we're just acting out the initial condition of the universe, that doesn't… It's not a romantic notion. Certainly not. Now when it comes to consciousness, I think we do have certain freedom. We are much more constrained by physics of course and by our past and by our own conscious desires and what our parents told us and what our environment tells us. We all know that, right? There's hundreds of experiments that show how we can be influenced. But finally in the final analysis, when you make a life – and I'm talking really about critical decision where you really think, should I marry, should I go to this school or that school, should I take this job or that job, should I cheat on my taxes or not? These are things where you really deliberate and I think under those conditions, you are as free as you can be. When you bring your entire being, your entire conscious being to that question and try to analyze it under all the various conditions, then you make a decision, you are as free as you can ever be. That is I think what free will is. It's not a will that's totally free to do anything it wants. That's not possible. Right. So as Jack mentioned, you actually write a blog about books you've read, amazing books from, I'm Russian, from Bulgakov, Neil Gaiman, Carl Sagan, Murakami. So what is a book that early in your life transformed the way you saw the world, something that changed your life? Nietzsche I guess did. That's Brooks R. Truster because he talks about some of these problems. He was one of the first discoverer of the unconscious. This is a little bit before Freud when he was in the air. He makes all these claims that people sort of under the guise or under the mass of charity actually are very noncharitable. So he is sort of really the first discoverer of the great land of the unconscious and that really struck me. And what do you think about the unconscious, what do you think about Freud, what do you think about these ideas? Just like dark matter in the universe, what's over there in that unconscious? A lot. I mean much more than we think. This is what a lot of last 100 years of research has shown. So I think he was a genius, misguided towards the end, but he started out as a neuroscientist. He contributed, he did the studies on the lamprey, he contributed himself to the neuron hypothesis, the idea that there are discrete units that we call nerve cells now. And then he wrote about the unconscious and I think it's true, there's lots of stuff happening. You feel this particular when you're in a relationship and it breaks asunder, right? And then you have this terrible, you can have love and hate and lust and anger and all of it's mixed in. And when you try to analyze yourself, why am I so upset? It's very, very difficult to penetrate to those basements, those caverns in your mind because the prying eyes of conscious doesn't have access to those, but they're there in the amygdala or lots of other places. They make you upset or angry or sad or depressed and it's very difficult to try to actually uncover the reason. You can go to a shrink, you can talk with your friend endlessly, you construct finally a story why this happened, why you love her or don't love her or whatever, but you don't really know whether that actually happened because you simply don't have access to those parts of the brain and they're very powerful. Do you think that's a feature or a bug of our brain? The fact that we have this deep, difficult to dive into subconscious? I think it's a feature because otherwise, look, we are like any other brain or nervous system or computer, we are severely band limited. If everything I do, every emotion I feel, every eye movements I make, if all of that had to be under the control of consciousness, I wouldn't be here. What you do early on, your brain, you have to be conscious when you learn things like typing or like riding on a bike, but then what you do, you train up routes, I think that involve basal ganglia and striatum. You train up different parts of your brain and then once you do it automatically like typing, you can show you do it much faster without even thinking about it because you've got these highly specialized, what Frans Krik and I call zombie agents, they're taking care of that while your consciousness can sort of worry about the abstract sense of the text you want to write. I think that's true for many, many things. But for the things like all the fights you had with an ex girlfriend, things that you would think are not useful to still linger somewhere in the subconscious. So that seems like a bug that it would stick to there. You think it would be better if you can analyze it and then get it out of the system. Better to get it out of the system or just forget it ever happened. That seems a very buggy kind of. Well yeah, in general we don't have, and that's probably functional, we don't have an ability unless it's extreme, there are cases, clinical dissociations, right? When people are heavily abused, when they completely repress the memory, but that doesn't happen in normal people. We don't have an ability to remove traumatic memories and of course we suffer from that. On the other hand, probably if you had the ability to constantly wipe your memory, you'd probably do it to an extent that isn't useful to you. So yeah, it's a good question to balance. So on the books, as Jack mentioned, correct me if I'm wrong, but broadly speaking, academia and the different scientific disciplines, certainly in engineering, reading literature seems to be a rare pursuit. So I'm wrong on this, but that's in my experience, most people read much more technical text and do not sort of escape or seek truth in literature. It seems like you do. So what do you think is the value, what do you think literature adds to the pursuit of scientific truth? Do you think it's good, it's useful for everybody? Gives you access to a much wider array of human experiences. How valuable do you think it is? Well if you want to understand human nature and nature in general, then I think you have to better understand a wide variety of experiences, not just sitting in a lab staring at a screen and having a face flashed onto you for a hundred milliseconds and pushing a button. That's what I used to do, that's what most psychologists do. There's nothing wrong with that, but you need to consider lots of other strange states. And literature is a shortcut for this. Well yeah, because literature, that's what literature is all about, all sorts of interesting experiences that people have, the contingency of it, the fact that women experience the world different, black people experience the world different. The one way to experience that is reading all these different literature and try to find out. You see, everything is so relative. You read a book 300 years ago, they thought about certain problems very, very differently than us today. We today, like any culture, think we know it all. That's common to every culture. Every culture believes at its heyday they know it all. And then you realize, well, there's other ways of viewing the universe and some of them may have lots of things in their favor. So this is a question I wanted to ask about time scale or scale in general. When you, with IIT or in general, try to think about consciousness, try to think about these ideas, we kind of naturally think in human time scales, and also entities that are sized close to humans. Do you think of things that are much larger and much smaller as containing consciousness? And do you think of things that take, you know, eons to operate in their conscious cause effect? That's a very good question. So I think a lot about small creatures because experimentally, you know, a lot of people work on flies and bees, right? So most people just think they are automata, they're just bugs for heaven's sake, right? But if you look at their behavior, like bees, they can recognize individual humans. They have this very complicated way to communicate. If you've ever been involved or you know your parents when they bought a house, what sort of agonizing decision that is. And bees have to do that once a year, right, when they swarm in the spring. And then they have this very elaborate way, they have free and scouts, they go to the individual sites, they come back, they have this power, this dance, literally, where they dance for several days, they try to recruit other deets, this very complicated decision rate, when they finally, once they make a decision, the entire swarm, the scouts warm up the entire swarm and then go to one location. They don't go to 50 locations, they go to one location that the scouts have agreed upon by themselves. That's awesome. If we look at the circuit complexity, it's 10 times more denser than anything we have in our brain. Now they only have a million neurons, but the neurons are amazingly complex. Complex behavior, very complicated circuitry, so there's no question they experience something, their life is very different, they're tiny, they only live, you know, for, well, workers live maybe for two months. So I think, and IIT tells you this, in principle, the substrate of consciousness is the substrate that maximizes the cause effect power over all possible spatial temporal grains. So when I think about, for example, do you know the science fiction story, The Black Cloud? Okay, it's a classic by Fred Hoyle, the astronomer. He has this cloud intervening between the earth and the sun and leading to some sort of, to global cooling, this is written in the 50s. It turns out you can, using the radio dish, they communicate with actually an entity, it's actually an intelligent entity, and they sort of, they convince it to move away. So here you have a radical different entity, and in principle, IIT says, well, you can measure the integrated information, in principle at least, and yes, if the maximum of that occurs at a time scale of months, rather than in assets for a fraction of a second, yes, then they would experience life where each moment is a month rather than, or microsecond, right, rather than a fraction of a second in the human case. And so there may be forms of consciousness that we simply don't recognize for what they are because they are so radical different from anything you and I are used to. Again, that's why it's good to read or to watch science fiction movies, well, to think about this. Do you know Stanislav Lem, this Polish science fiction writer, he wrote Solaris and was turned into a Hollywood movie? Yes. His best novel is in the 60s, a very engineer, he's an engineer in background. His most interesting novel is called The Victorious, where human civilization, they have this mission to this planet and everything is destroyed and they discover machines, humans got killed and then these machines took over and there was this machine evolution, Darwinian evolution, he talks about this very vividly. And finally, the dominant machine intelligence organism that survived were gigantic clouds of little hexagonal universal cellular automata. This was written in the 60s, so typically they're all lying on the ground individual by themselves, but in times of crisis, they can communicate, they assemble into gigantic nets into clouds of trillions of these particles and then they become hyper intelligent and they can beat anything that humans can throw at it. It's very beautiful and compelling where you have an intelligence where finally the humans leave the planet, they're simply unable to understand and comprehend this creature. They can say, well, either we can nuke the entire planet and destroy it or we just have to leave because fundamentally it's an alien, it's so alien from us and our ideas that we cannot communicate with them. Yeah, actually in conversation, so you're talking to us, Steven Wolf from Brought Up is that there could be ideas that you already have these artificial general intelligence like super smart or maybe conscious beings in the cellular automata, we just don't know how to talk to them. So it's the language of communication, but you don't know what to do with it. So that's one sort of view is consciousness is only something you can measure. So it's not conscious if you can't measure it. So you're making an ontological and an epistemic statement. One is it's just like seeing the multiverses, that might be true, but I can't communicate with them. I can't have any knowledge of them. That's an epistemic argument. Right? So those are two different things. So it may well be possible. Look, in another case that's happening right now, people are building these mini organoids. Do you know what this is? So you can take stem cells from under your arm, put it in a dish, add four transcription factors and then you can induce them to grow into large, well, large, they're a few millimeters. They're like a half a million neurons that look like nerve cells in a dish called mini organoids at Harvard, at Stanford, everywhere they're building them. It may well be possible that they're beginning to feel like something, but we can't really communicate with them right now. So people are beginning to think about the ethics of this. So yes, he may be perfectly right, but it's one question, are they conscious or not? It's a totally separate question. How would I know? Those are two different things. If you could give advice to a young researcher, sort of dreaming of understanding or creating human level intelligence or consciousness, what would you say? Just follow your dreams. Read widely. No, I mean, I suppose with discipline, what is the pursuit that they should take on? Is it neuroscience? Is it computational cognitive science? Is it philosophy? Is it computer science or robotics? No, in a sense that, okay, so the only known system that have high level of intelligence is homo sapiens. So if you wanted to build it, it's probably good to continue to study closely what humans do. So cognitive neuroscience, you know, somewhere between cognitive neuroscience on the one hand and some philosophy of mind and then AI, AI computer science. You can look at all the original ideas in your network, they all came from neuroscience, right? Reinforce whether it's Snarky, Minsky building is Snarky or whether it's, you know, the early Hubel and Wiesel experiments at Harvard that then gave rise to networks and then multi layer networks. So it may well be possible, in fact, some people argue that to make the next big step in AI once we realize the limits of deep convolutional networks, they can do certain things, but they can't really understand. They don't, they can't really, I can't really show them one image. I can show you a single image of somebody, a pickpocket who steals a wallet from a purse. You immediately know that's a pickpocket. Now computer system would just say, well, it's a man, it's a woman, it's a purse, right? Unless you train this machine on showing it a hundred thousand pickpockets, right? So it doesn't have this easy understanding that you have, right? So some people make the argument in order to go to the next step or you really want to build machines that understand in a way you and I, we have to go to psychology. We need to understand how we do it and how our brains enable us to do it. And so therefore being on the cusp, it's also so exciting to try to understand better our nature and then to build, to take some of those inside and build them. So I think the most exciting thing is somewhere in the interface between cognitive science, neuroscience, AI, computer science and philosophy of mind. Beautiful. Yeah. I'd say if there is from the machine learning, from our, from the computer science, computer vision perspective, many of the researchers kind of ignore the way the human brain works or even psychology or literature or studying the brain, I would hope Josh Tenenbaum talks about bringing that in more and more. And that's, yeah, so you've worked on some amazing stuff throughout your life. What's the thing that you're really excited about? What's the mystery that you would love to uncover in the near term beyond, beyond all the mysteries that you're already surrounded by? Well, so there's a structure called the claustrum. This is a structure, it's underneath our cortex, it's yay big. You have one on the left, on the right, underneath this, underneath the insula, it's very thin, it's like one millimeter, it's embedded in, in wiring, in white matter, so it's very difficult to image. And it has, it has connection to every cortical region. And Francis Crick, the last paper he ever wrote, he dictated corrections the day he died in hospital on this paper. You know, we hypothesize, well, because it has this unique anatomy, it gets input from every cortical area and projects back to every, every cortical area. That the function of this structure is similar, it's just a metaphor to the role of a conductor in a symphony orchestra. You have all the different cortical players. You have some that do motion, some that do theory of mind, some that infer social interaction and color and hearing and all the different modules in cortex. But of course, what consciousness is, consciousness puts it all together into one package, right? The binding problem, all of that. And this is really the function because it has relatively few neurons compared to cortex, but it talks, it receives input from all of them and it projects back to all of them. And so we're testing that right now. We've got this beautiful neuronal reconstruction in the mouse called crown of thorns, crown of thorns neurons that are in the claustrum that have the most widespread connection of any neuron I've ever seen. They're very, you have individual neurons that sit in the claustrum tiny, but then they have this single neuron, they have this huge axonal tree that cover both ipsy and contralateral cortex and trying to turn using, you know, fancy tools like optogenetics, trying to turn those neurons on or off and study it, what happens in the, in the mouse. So this thing is perhaps where the parts become the whole. Perhaps it's one of the structures, it's a very good way of putting it, where the individual parts turn into the whole of the whole of the conscious experience. Well, with that, thank you very much for being here today. Thank you very much. Thank you very much. All right, thank you very much.

Christof Koch: Consciousness | Lex Fridman Podcast #2


You've studied the human mind, cognition, language, vision, evolution, psychology, from child to adult, from the level of individual to the level of our entire civilization. So I feel like I can start with a simple multiple choice question. What is the meaning of life? Is it A. to attain knowledge as Plato said, B. to attain power as Nietzsche said, C. to escape death as Ernest Becker said, D. to propagate our genes as Darwin and others have said, E. there is no meaning as the nihilists have said, F. knowing the meaning of life is beyond our cognitive capabilities as Stephen Pinker said, based on my interpretation 20 years ago, and G. none of the above. I'd say A. comes closest, but I would amend that to C. to attaining not only knowledge but fulfillment more generally, that is life, health, stimulation, access to the living cultural and social world. Now this is our meaning of life. It's not the meaning of life if you were to ask our genes. Their meaning is to propagate copies of themselves, but that is distinct from the meaning that the brain that they lead to sets for itself. So to you knowledge is a small subset or a large subset? It's a large subset, but it's not the entirety of human striving because we also want to interact with people. We want to experience beauty. We want to experience the richness of the natural world, but understanding what makes the universe tick is way up there. For some of us more than others, certainly for me that's one of the top five. So is that a fundamental aspect? Are you just describing your own preference or is this a fundamental aspect of human nature is to seek knowledge? In your latest book you talk about the power, the usefulness of rationality and reason and so on. Is that a fundamental nature of human beings or is it something we should just strive for? Both. We're capable of striving for it because it is one of the things that make us what we are, homo sapiens, wise men. We are unusual among animals in the degree to which we acquire knowledge and use it to survive. We make tools. We strike agreements via language. We extract poisons. We predict the behavior of animals. We try to get at the workings of plants. And when I say we, I don't just mean we in the modern West, but we as a species everywhere, which is how we've managed to occupy every niche on the planet, how we've managed to drive other animals to extinction. And the refinement of reason in pursuit of human wellbeing, of health, happiness, social richness, cultural richness is our main challenge in the present. That is using our intellect, using our knowledge to figure out how the world works, how we work in order to make discoveries and strike agreements that make us all better off in the long run. Right. And you do that almost undeniably and in a data driven way in your recent book, but I'd like to focus on the artificial intelligence aspect of things and not just artificial intelligence, but natural intelligence too. So 20 years ago in a book you've written on how the mind works, you conjecture again, am I right to interpret things? You can correct me if I'm wrong, but you conjecture that human thought in the brain may be a result of a massive network of highly interconnected neurons. So from this interconnectivity emerges thought compared to artificial neural networks, which we use for machine learning today, is there something fundamentally more complex, mysterious, even magical about the biological neural networks versus the ones we've been starting to use over the past 60 years and have become to success in the past 10? There is something a little bit mysterious about the human neural networks, which is that each one of us who is a neural network knows that we ourselves are conscious. Conscious not in the sense of registering our surroundings or even registering our internal state, but in having subjective first person, present tense experience. That is when I see red, it's not just different from green, but there's a redness to it that I feel. Whether an artificial system would experience that or not, I don't know and I don't think I can know. That's why it's mysterious. If we had a perfectly lifelike robot that was behaviorally indistinguishable from a human, would we attribute consciousness to it or ought we to attribute consciousness to it? And that's something that it's very hard to know. But putting that aside, putting aside that largely philosophical question, the question is, is there some difference between the human neural network and the ones that we're building in artificial intelligence will mean that we're on the current trajectory, not going to reach the point where we've got a lifelike robot indistinguishable from a human because the way their so called neural networks are organized are different from the way ours are organized. I think there's overlap, but I think there are some big differences that current neural networks, current so called deep learning systems are in reality not all that deep. That is, they are very good at extracting high order statistical regularities, but most of the systems don't have a semantic level, a level of actual understanding of who did what to whom, why, where, how things work, what causes what else. Do you think that kind of thing can emerge as it does? So artificial neural networks are much smaller, the number of connections and so on than the current human biological networks, but do you think sort of to go to consciousness or to go to this higher level semantic reasoning about things, do you think that can emerge with just a larger network with a more richly weirdly interconnected network? Separate it in consciousness because consciousness is even a matter of complexity. A really weird one. Yeah, you could sensibly ask the question of whether shrimp are conscious, for example, they're not terribly complex, but maybe they feel pain. So let's just put that part of it aside. But I think sheer size of a neural network is not enough to give it structure and knowledge, but if it's suitably engineered, then why not? That is, we're neural networks, natural selection did a kind of equivalent of engineering of our brains. So I don't think there's anything mysterious in the sense that no system made out of silicon could ever do what a human brain can do. I think it's possible in principle. Whether it'll ever happen depends not only on how clever we are in engineering these systems, but whether we even want to, whether that's even a sensible goal. That is, you can ask the question, is there any locomotion system that is as good as a human? Well, we kind of want to do better than a human ultimately in terms of legged locomotion. There's no reason that humans should be our benchmark. They're tools that might be better in some ways. It may be that we can't duplicate a natural system because at some point it's so much cheaper to use a natural system that we're not going to invest more brainpower and resources. So for example, we don't really have an exact substitute for wood. We still build houses out of wood. We still build furniture out of wood. We like the look. We like the feel. It has certain properties that synthetics don't. It's not that there's anything magical or mysterious about wood. It's just that the extra steps of duplicating everything about wood is something we just haven't bothered because we have wood. Likewise, say cotton. I'm wearing cotton clothing now. It feels much better than polyester. It's not that cotton has something magic in it. It's not that we couldn't ever synthesize something exactly like cotton, but at some point it's just not worth it. We've got cotton. Likewise, in the case of human intelligence, the goal of making an artificial system that is exactly like the human brain is a goal that we probably know is going to pursue to the bitter end, I suspect, because if you want tools that do things better than humans, you're not going to care whether it does something like humans. So for example, diagnosing cancer or predicting the weather, why set humans as your benchmark? But in general, I suspect you also believe that even if the human should not be a benchmark and we don't want to imitate humans in their system, there's a lot to be learned about how to create an artificial intelligence system by studying the human. Yeah, I think that's right. In the same way that to build flying machines, we want to understand the laws of aerodynamics, including birds, but not mimic the birds, but they're the same laws. You have a view on AI, artificial intelligence, and safety that, from my perspective, is refreshingly rational or perhaps more importantly, has elements of positivity to it, which I think can be inspiring and empowering as opposed to paralyzing. For many people, including AI researchers, the eventual existential threat of AI is obvious, not only possible, but obvious. And for many others, including AI researchers, the threat is not obvious. So Elon Musk is famously in the highly concerned about AI camp, saying things like AI is far more dangerous than nuclear weapons, and that AI will likely destroy human civilization. Human civilization. So in February, he said that if Elon was really serious about AI, the threat of AI, he would stop building self driving cars that he's doing very successfully as part of Tesla. Then he said, wow, if even Pinker doesn't understand the difference between narrow AI, like a car and general AI, when the latter literally has a million times more compute power and an open ended utility function, humanity is in deep trouble. So first, what did you mean by the statement about Elon Musk should stop building self driving cars if he's deeply concerned? Not the last time that Elon Musk has fired off an intemperate tweet. Well, we live in a world where Twitter has power. Yes. Yeah, I think there are two kinds of existential threat that have been discussed in connection with artificial intelligence, and I think that they're both incoherent. One of them is a vague fear of AI takeover, that just as we subjugated animals and less technologically advanced peoples, so if we build something that's more advanced than us, it will inevitably turn us into pets or slaves or domesticated animal equivalents. I think this confuses intelligence with a will to power, that it so happens that in the intelligence system we are most familiar with, namely homo sapiens, we are products of natural selection, which is a competitive process, and so bundled together with our problem solving capacity are a number of nasty traits like dominance and exploitation and maximization of power and glory and resources and influence. There's no reason to think that sheer problem solving capability will set that as one of its goals. Its goals will be whatever we set its goals as, and as long as someone isn't building a megalomaniacal artificial intelligence, then there's no reason to think that it would naturally evolve in that direction. Now, you might say, well, what if we gave it the goal of maximizing its own power source? That's a pretty stupid goal to give an autonomous system. You don't give it that goal. I mean, that's just self evidently idiotic. So if you look at the history of the world, there's been a lot of opportunities where engineers could instill in a system destructive power and they choose not to because that's the natural process of engineering. Well, except for weapons. I mean, if you're building a weapon, its goal is to destroy people, and so I think there are good reasons to not build certain kinds of weapons. I think building nuclear weapons was a massive mistake. You do. So maybe pause on that because that is one of the serious threats. Do you think that it was a mistake in a sense that it should have been stopped early on? Or do you think it's just an unfortunate event of invention that this was invented? Do you think it's possible to stop? I guess is the question. It's hard to rewind the clock because of course it was invented in the context of World War II and the fear that the Nazis might develop one first. Then once it was initiated for that reason, it was hard to turn off, especially since winning the war against the Japanese and the Nazis was such an overwhelming goal of every responsible person that there's just nothing that people wouldn't have done then to ensure victory. It's quite possible if World War II hadn't happened that nuclear weapons wouldn't have been invented. We can't know, but I don't think it was by any means a necessity, any more than some of the other weapon systems that were envisioned but never implemented, like planes that would disperse poison gas over cities like crop dusters or systems to try to create earthquakes and tsunamis in enemy countries, to weaponize the weather, weaponize solar flares, all kinds of crazy schemes that we thought the better of. I think analogies between nuclear weapons and artificial intelligence are fundamentally misguided because the whole point of nuclear weapons is to destroy things. The point of artificial intelligence is not to destroy things. So the analogy is misleading. So there's two artificial intelligence you mentioned. The first one I guess is highly intelligent or power hungry. Yeah, it's a system that we design ourselves where we give it the goals. Goals are external to the means to attain the goals. If we don't design an artificially intelligent system to maximize dominance, then it won't maximize dominance. It's just that we're so familiar with homo sapiens where these two traits come bundled together, particularly in men, that we are apt to confuse high intelligence with a will to power, but that's just an error. The other fear is that will be collateral damage that will give artificial intelligence a goal like make paper clips and it will pursue that goal so brilliantly that before we can stop it, it turns us into paper clips. We'll give it the goal of curing cancer and it will turn us into guinea pigs for lethal experiments or give it the goal of world peace and its conception of world peace is no people, therefore no fighting and so it will kill us all. Now I think these are utterly fanciful. In fact, I think they're actually self defeating. They first of all assume that we're going to be so brilliant that we can design an artificial intelligence that can cure cancer, but so stupid that we don't specify what we mean by curing cancer in enough detail that it won't kill us in the process and it assumes that the system will be so smart that it can cure cancer, but so idiotic that it can't figure out that what we mean by curing cancer is not killing everyone. I think that the collateral damage scenario, the value alignment problem is also based on a misconception. So one of the challenges, of course, we don't know how to build either system currently or are we even close to knowing? Of course, those things can change overnight, but at this time, theorizing about it is very challenging in either direction. So that's probably at the core of the problem is without that ability to reason about the real engineering things here at hand is your imagination runs away with things. Exactly. But let me sort of ask, what do you think was the motivation, the thought process of Elon Musk? I build autonomous vehicles, I study autonomous vehicles, I study Tesla autopilot. I think it is one of the greatest currently large scale application of artificial intelligence in the world. It has potentially a very positive impact on society. So how does a person who's creating this very good quote unquote narrow AI system also seem to be so concerned about this other general AI? What do you think is the motivation there? What do you think is the thing? Well, you probably have to ask him, but there, and he is notoriously flamboyant, impulsive to the, as we have just seen, to the detriment of his own goals of the health of the company. So I don't know what's going on in his mind. You probably have to ask him, but I don't think the, and I don't think the distinction between special purpose AI and so called general AI is relevant that in the same way that special purpose AI is not going to do anything conceivable in order to attain a goal. All engineering systems are designed to trade off across multiple goals. When we build cars in the first place, we didn't forget to install brakes because the goal of a car is to go fast. It occurred to people, yes, you want it to go fast, but not always. So you would build in brakes too. Likewise, if a car is going to be autonomous and program it to take the shortest route to the airport, it's not going to take the diagonal and mow down people and trees and fences because that's the shortest route. That's not what we mean by the shortest route when we program it. And that's just what an intelligence system is by definition. It takes into account multiple constraints. The same is true, in fact, even more true of so called general intelligence. That is, if it's genuinely intelligent, it's not going to pursue some goal singlemindedly, omitting every other consideration and collateral effect. That's not artificial and general intelligence. That's artificial stupidity. I agree with you, by the way, on the promise of autonomous vehicles for improving human welfare. I think it's spectacular. And I'm surprised at how little press coverage notes that in the United States alone, something like 40,000 people die every year on the highways, vastly more than are killed by terrorists. And we spent a trillion dollars on a war to combat deaths by terrorism, about half a dozen a year. Whereas year in, year out, 40,000 people are massacred on the highways, which could be brought down to very close to zero. So I'm with you on the humanitarian benefit. Let me just mention that as a person who's building these cars, it is a little bit offensive to me to say that engineers would be clueless enough not to engineer safety into systems. I often stay up at night thinking about those 40,000 people that are dying. And everything I tried to engineer is to save those people's lives. So every new invention that I'm super excited about, in all the deep learning literature and CVPR conferences and NIPS, everything I'm super excited about is all grounded in making it safe and help people. So I just don't see how that trajectory can all of a sudden slip into a situation where intelligence will be highly negative. You and I certainly agree on that. And I think that's only the beginning of the potential humanitarian benefits of artificial intelligence. There's been enormous attention to what are we going to do with the people whose jobs are made obsolete by artificial intelligence, but very little attention given to the fact that the jobs that are going to be made obsolete are horrible jobs. The fact that people aren't going to be picking crops and making beds and driving trucks and mining coal, these are soul deadening jobs. And we have a whole literature sympathizing with the people stuck in these menial, mind deadening, dangerous jobs. If we can eliminate them, this is a fantastic boon to humanity. Now granted, you solve one problem and there's another one, namely, how do we get these people a decent income? But if we're smart enough to invent machines that can make beds and put away dishes and handle hospital patients, I think we're smart enough to figure out how to redistribute income to apportion some of the vast economic savings to the human beings who will no longer be needed to make beds. Okay. Sam Harris says that it's obvious that eventually AI will be an existential risk. He's one of the people who says it's obvious. We don't know when the claim goes, but eventually it's obvious. And because we don't know when, we should worry about it now. This is a very interesting argument in my eyes. So how do we think about timescale? How do we think about existential threats when we don't really, we know so little about the threat, unlike nuclear weapons perhaps, about this particular threat, that it could happen tomorrow, right? So, but very likely it won't. Very likely it'd be a hundred years away. So how do we ignore it? How do we talk about it? Do we worry about it? How do we think about those? What is it? A threat that we can imagine. It's within the limits of our imagination, but not within our limits of understanding to accurately predict it. But what is the it that we're afraid of? Sorry. AI being the existential threat. AI. How? Like enslaving us or turning us into paperclips? I think the most compelling from the Sam Harris perspective would be the paperclip situation. Yeah. I mean, I just think it's totally fanciful. I mean, that is don't build a system. Don't give a, don't, first of all, the code of engineering is you don't implement a system with massive control before testing it. Now, perhaps the culture of engineering will radically change. Then I would worry, but I don't see any signs that engineers will suddenly do idiotic things, like put a electric power plant in control of a system that they haven't tested first. Or all of these scenarios, not only imagine almost a magically powered intelligence, including things like cure cancer, which is probably an incoherent goal because there's so many different kinds of cancer or bring about world peace. I mean, how do you even specify that as a goal? But the scenarios also imagine some degree of control of every molecule in the universe, which not only is itself unlikely, but we would not start to connect these systems to infrastructure without testing as we would any kind of engineering system. Now, maybe some engineers will be irresponsible and we need legal and regulatory and legal responsibility implemented so that engineers don't do things that are stupid by their own standards. But the, I've never seen enough of a plausible scenario of existential threat to devote large amounts of brain power to, to forestall it. So you believe in the sort of the power on mass of the engineering of reason, as you argue in your latest book of Reason and Science, to sort of be the very thing that guides the development of new technology so it's safe and also keeps us safe. You know, granted the same culture of safety that currently is part of the engineering mindset for airplanes, for example. So yeah, I don't think that that should be thrown out the window and that untested all powerful systems should be suddenly implemented, but there's no reason to think they are. And in fact, if you look at the progress of artificial intelligence, it's been, you know, it's been impressive, especially in the last 10 years or so, but the idea that suddenly there'll be a step function that all of a sudden before we know it, it will be all powerful, that there'll be some kind of recursive self improvement, some kind of fume is also fanciful. We, certainly by the technology that we, that we're now impresses us, such as deep learning, where you train something on hundreds of thousands or millions of examples, they're not hundreds of thousands of problems of which curing cancer is a typical example. And so the kind of techniques that have allowed AI to increase in the last five years are not the kind that are going to lead to this fantasy of exponential sudden self improvement. I think it's kind of a magical thinking. It's not based on our understanding of how AI actually works. Now give me a chance here. So you said fanciful, magical thinking. In his TED talk, Sam Harris says that thinking about AI killing all human civilization is somehow fun, intellectually. Now I have to say as a scientist engineer, I don't find it fun, but when I'm having beer with my non AI friends, there is indeed something fun and appealing about it. Like talking about an episode of Black Mirror, considering if a large meteor is headed towards Earth, we were just told a large meteor is headed towards Earth, something like this. And can you relate to this sense of fun? And do you understand the psychology of it? Yes. Good question. I personally don't find it fun. I find it kind of actually a waste of time because there are genuine threats that we ought to be thinking about like pandemics, like cyber security vulnerabilities, like the possibility of nuclear war and certainly climate change. You know, this is enough to fill many conversations. And I think Sam did put his finger on something, namely that there is a community, sometimes called the rationality community, that delights in using its brainpower to come up with scenarios that would not occur to mere mortals, to less cerebral people. So there is a kind of intellectual thrill in finding new things to worry about that no one has worried about yet. I actually think, though, that it's not only is it a kind of fun that doesn't give me particular pleasure, but I think there can be a pernicious side to it, namely that you overcome people with such dread, such fatalism, that there are so many ways to die, to annihilate our civilization, that we may as well enjoy life while we can. There's nothing we can do about it. If climate change doesn't do us in, then runaway robots will. So let's enjoy ourselves now. We've got to prioritize. We have to look at threats that are close to certainty, such as climate change, and distinguish those from ones that are merely imaginable but with infinitesimal probabilities. And we have to take into account people's worry budget. You can't worry about everything. And if you sow dread and fear and terror and fatalism, it can lead to a kind of numbness. Well, these problems are overwhelming, and the engineers are just going to kill us all. So let's either destroy the entire infrastructure of science, technology, or let's just enjoy life while we can. So there's a certain line of worry, which I'm worried about a lot of things in engineering. There's a certain line of worry when you cross, you're allowed to cross, that it becomes paralyzing fear as opposed to productive fear. And that's kind of what you're highlighting. Exactly right. And we've seen some, we know that human effort is not well calibrated against risk in that because a basic tenet of cognitive psychology is that perception of risk and hence perception of fear is driven by imaginability, not by data. And so we misallocate vast amounts of resources to avoiding terrorism, which kills on average about six Americans a year with one exception of 9 11. We invade countries, we invent entire new departments of government with massive, massive expenditure of resources and lives to defend ourselves against a trivial risk. Whereas guaranteed risks, one of them you mentioned traffic fatalities and even risks that are not here, but are plausible enough to worry about like pandemics, like nuclear war, receive far too little attention. In presidential debates, there's no discussion of how to minimize the risk of nuclear war. Lots of discussion of terrorism, for example. And so I think it's essential to calibrate our budget of fear, worry, concern, planning to the actual probability of harm. Yep. So let me ask this question. So speaking of imaginability, you said it's important to think about reason and one of my favorite people who likes to dip into the outskirts of reason through fascinating exploration of his imagination is Joe Rogan. Oh yes. So who has through reason used to believe a lot of conspiracies and through reason has stripped away a lot of his beliefs in that way. So it's fascinating actually to watch him through rationality kind of throw away the ideas of Bigfoot and 9 11. I'm not sure exactly. Kim Trails. I don't know what he believes in. Yes. Okay. But he no longer believed in. No, that's right. No, he's become a real force for good. Yep. So you were on the Joe Rogan podcast in February and had a fascinating conversation, but as far as I remember, didn't talk much about artificial intelligence. I will be on his podcast in a couple of weeks. Joe is very much concerned about existential threat of AI. I'm not sure if you're, this is why I was hoping that you would get into that topic. And in this way, he represents quite a lot of people who look at the topic of AI from 10,000 foot level. So as an exercise of communication, you said it's important to be rational and reason about these things. Let me ask, if you were to coach me as an AI researcher about how to speak to Joe and the general public about AI, what would you advise? Well, the short answer would be to read the sections that I wrote in enlightenment now about AI, but a longer reason would be I think to emphasize, and I think you're very well positioned as an engineer to remind people about the culture of engineering, that it really is safety oriented, that another discussion in enlightenment now, I plot rates of accidental death from various causes, plane crashes, car crashes, occupational accidents, even death by lightning strikes. And they all plummet because the culture of engineering is how do you squeeze out the lethal risks, death by fire, death by drowning, death by asphyxiation, all of them drastically declined because of advances in engineering that I got to say, I did not appreciate until I saw those graphs. And it is because exactly, people like you who stay up at night thinking, oh my God, is what I'm inventing likely to hurt people and to deploy ingenuity to prevent that from happening. Now, I'm not an engineer, although I spent 22 years at MIT, so I know something about the culture of engineering. My understanding is that this is the way you think if you're an engineer. And it's essential that that culture not be suddenly switched off when it comes to artificial intelligence. So, I mean, that could be a problem, but is there any reason to think it would be switched off? I don't think so. And one, there's not enough engineers speaking up for this way, for the excitement, for the positive view of human nature, what you're trying to create is positivity. Like everything we try to invent is trying to do good for the world. But let me ask you about the psychology of negativity. It seems just objectively, not considering the topic, it seems that being negative about the future makes you sound smarter than being positive about the future, irregardless of topic. Am I correct in this observation? And if so, why do you think that is? Yeah, I think there is that phenomenon that, as Tom Lehrer, the satirist said, always predict the worst and you'll be hailed as a prophet. It may be part of our overall negativity bias. We are as a species more attuned to the negative than the positive. We dread losses more than we enjoy gains. And that might open up a space for prophets to remind us of harms and risks and losses that we may have overlooked. So I think there is that asymmetry. So you've written some of my favorite books all over the place. So starting from Enlightenment Now to The Better Ages of Our Nature, Blank Slate, How the Mind Works, the one about language, Language Instinct. Bill Gates, big fan too, said of your most recent book that it's my new favorite book of all time. So for you as an author, what was a book early on in your life that had a profound impact on the way you saw the world? Certainly this book, Enlightenment Now, was influenced by David Deutsch's The Beginning of Infinity, a rather deep reflection on knowledge and the power of knowledge to improve the human condition. And with bits of wisdom such as that problems are inevitable but problems are solvable given the right knowledge and that solutions create new problems that have to be solved in their turn. That's I think a kind of wisdom about the human condition that influenced the writing of this book. There are some books that are excellent but obscure, some of which I have on a page on my website. I read a book called The History of Force, self published by a political scientist named James Payne on the historical decline of violence and that was one of the inspirations for The Better Angels of Our Nature. What about early on? If you look back when you were maybe a teenager? I loved a book called One, Two, Three, Infinity. When I was a young adult I read that book by George Gamow, the physicist, which had very accessible and humorous explanations of relativity, of number theory, of dimensionality, high multiple dimensional spaces in a way that I think is still delightful 70 years after it was published. I like the Time Life Science series. These are books that would arrive every month that my mother subscribed to, each one on a different topic. One would be on electricity, one would be on forests, one would be on evolution and then one was on the mind. I was just intrigued that there could be a science of mind and that book I would cite as an influence as well. Then later on... That's when you fell in love with the idea of studying the mind? Was that the thing that grabbed you? It was one of the things I would say. I read as a college student the book Reflections on Language by Noam Chomsky. I spent most of his career here at MIT. Richard Dawkins, two books, The Blind Watchmaker and The Selfish Gene, were enormously influential, mainly for the content but also for the writing style, the ability to explain abstract concepts in lively prose. Stephen Jay Gould's first collection, Ever Since Darwin, also an excellent example of lively writing. George Miller, a psychologist that most psychologists are familiar with, came up with the idea that human memory has a capacity of seven plus or minus two chunks. That's probably his biggest claim to fame. But he wrote a couple of books on language and communication that I read as an undergraduate. Again, beautifully written and intellectually deep. Wonderful. Stephen, thank you so much for taking the time today. My pleasure. Thanks a lot, Lex.

Steven Pinker: AI in the Age of Reason | Lex Fridman Podcast #3