Dmitri Dolgov: Waymo and the Future of Self-Driving Cars | Lex Fridman Podcast #147
dmitri dolgov artificial intelligence agi ai ai podcast artificial intelligence podcast lex fridman lex podcast lex mit lex ai lex jre mit ai
Dmitri Dolgov is the CTO of Waymo, an autonomous vehicle company. Please support this podcast by checking out our sponsors:
- Tryolabs: https://tryolabs.com/lex
- Blinkist: https://blinkist.com/lex and use code LEX to get 25% off premium
- BetterHelp: https://betterhelp.com/lex to get 10% off
- Cash App: https://cash.app/ and use code LexPodcast to get $10

EPISODE LINKS:
Waymo's Twitter: https://twitter.com/waymo
Waymo's Website: https://waymo.com

PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
Full episodes playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
2:16 - Computer games
7:23 - Childhood
9:55 - Robotics
10:44 - Moscow Institute of Physics and Technology
12:56 - DARPA Urban Challenge
23:16 - Waymo origin story
38:58 - Waymo self-driving hardware
47:31 - Connected cars
53:23 - Waymo fully driverless service in Phoenix
57:45 - Getting feedback from riders
1:05:58 - Creating a product that people love
1:11:49 - Do self-driving cars need to break the rules like humans do?
1:18:33 - Waymo Trucks
1:24:11 - Future of Waymo
1:37:23 - Role of lidar in autonomous driving
1:50:23 - Machine learning is essential for autonomous driving
1:54:25 - Pedestrians
2:01:02 - Trolley problem
2:05:30 - Book recommendations
2:16:56 - Meaning of life

CONNECT:
- Subscribe to this YouTube channel
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/LexFridmanPage
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman
 The following is a conversation with Dmitry Dolgov, the CTO of Waymo, which is an autonomous driving company that started as Google self-driving car project in 2009 and became Waymo in 2016. Dmitry was there all along. Waymo is currently leading in the fully autonomous vehicle space and that they actually have an at-scale deployment of publicly accessible autonomous vehicles, driving passengers around with no safety driver, with nobody in the driver's seat. This, to me, is an incredible accomplishment of engineering on one of the most difficult and exciting artificial intelligence challenges of the 21st century. Quick mention of a sponsor, followed by some thoughts related to the episode. Thank you to Trial Labs, a company that helps businesses apply machine learning to solve real world problems. Blinkist, an app I use for reading through summaries of books, BetterHelp, online therapy with a licensed professional, and CashApp, the app I use to send money to friends. Please check out the sponsors in the description to get a discount at the support this podcast. As a side note, let me say that autonomous and semi-autonomous driving was the focus of my work at MIT and is a problem space that I find fascinating and full of open questions from both a robotics and a human psychology perspective. There's quite a bit that I could say here about my experiences in academia on this topic that revealed to me, let's say, the less admirable sides of human beings. But I choose to focus on the positive, on solutions. On brilliant engineers like Dimitri and the team at Waymo, who work tirelessly to innovate and to build amazing technology that will define our future. Because of Dimitri and others like him, I'm excited for this future. And who knows, perhaps I too will help contribute something of value to it. If you enjoy this thing, subscribe on YouTube, review it with five stars on a podcast, follow on Spotify, support on Patreon, or connect with me on Twitter at Lex Friedman. And now here's my conversation with Dimitri Dolgov. When did you first fall in love with robotics or even computer science more in general? Computer science first at a fairly young age. Robotics happened much later. I think my first interesting introduction to computers was in the late 80s, when we got our first computer. I think it was an IBM, I think IBM AT. Do you remember those things that had like a turbo button in the front? Turbo button, yeah. The radio would press it and make the thing go faster. Did they already have floppy disks? Yeah, yeah, yeah. The 5.4 inch ones. I think there was a bigger inch. So it went something, then five inches, then three inches. Yeah, I think that was the five. Maybe before that was the giant plates, then I didn't get that. But it was definitely not the first computer. But it was definitely not the three inch ones. Anyway, so that computer, I spent the first few months just playing video games, as you would expect. I got bored of that. So I started messing around and trying to figure out how to make the thing do other stuff. Got into exploring programming. And a couple of years later, I got to a point where I actually wrote a game, a little game, and a game developer, a Japanese game developer actually offered to buy it for me for a few hundred bucks, but for a kid in Russia. That's a big deal. That's a big deal, yeah. I did not take the deal. Wow, integrity. Yeah, I instead... Stability. Yes, that was not the most acute financial move that I made in my life. Looking back at it now, I instead put it... Well, I had a reason. I put it online. It was... What did you call it back in the day? It was a freeware thing, right? It was not open source, but you could upload the binaries, you would put the game online. And the idea was that people like it and then they contribute and they send you a little donations, right? So I did my quick math of like, of course, thousands and millions of people are going to play my game, send me a couple of bucks apiece, should definitely do that. As I said, not the best financial move. You're already playing business models at that age. Remember what language it was? What program it was? Pascal. What? Pascal. And they had a graphical component, so it's not text-based. Yeah, it was like, I think 320 by 200, whatever it was. I think that kind of the earlier version was VGA resolution, right? And I actually think the reason why this company wanted to buy it is not like the fancy graphics or the implementation. It was maybe the idea of the actual game. The idea of the game. Okay. Well, one of the things, it's so funny, I used to play this game called Golden Axe. And the simplicity of the graphics and something about the simplicity of the music, like it still haunts me. I don't know if that's a childhood thing. I don't know if that's the same thing for Call of Duty these days for young kids. But I still think that the simple, when the games are simple, that simple purity makes for, like allows your imagination to take over and thereby creating a more magical experience. Like now with better and better graphics, it feels like your imagination doesn't get to create worlds, which is kind of interesting. It could be just an old man on a porch, like waving at kids these days that have no respect. But I still think that graphics almost get in the way of the experience. I don't know. Flippy Bird. Yeah. Well, I don't know if the imagination gets closed. I don't. Yeah. But that that's more about games that like that's more like Tetris World, where they optimally masterfully like create a fun, short term dopamine experience versus I'm more referring to like role playing games where there's like a story. You can live in it for months or years like there's an Elder Scrolls series, which is probably my favorite set of games. That was a magical experience. And the graphics are terrible. The characters were all randomly generated, but I don't know. It pulls you in. There's a story. It's like an interactive version of an Elder Scrolls Tolkien world. And you get to live in it. I don't know. I miss it. It's one of the things that suck about being an adult is there's no you have to live in the real world as opposed to the Elder Scrolls world. You know, whatever brings you joy, right? Minecraft, right? Minecraft is a great example. You create like it's not the fancy graphics, but it's the creation of your own worlds. Yeah, that one is crazy. You know, one of the pitches for being a parent that people tell me is that you can like use the excuse of parenting to to go back into the video game world. And like that's like, you know, father, son, father, daughter time. But really, you just get to play video games with your kids. So anyway, at that time, did you have any ridiculous ambitious dreams of where as a creator you might go as an engineer? Did you what did you think of yourself as as an engineer, as a tinker? Or did you want to be like an astronaut or something like that? You know, I'm tempted to make something up about, you know, robots, engineering or, you know, mysteries of the universe. But that's not the actual memory that pops into my mind when you when you ask me about childhood dreams. So I actually share the real thing. When I was maybe four or five years old, I as well do, I thought about what I wanted to do when I grow up. And I had this dream of being a traffic control cop. They don't have those today's I think. But back in the 80s and in Russia, you probably are familiar with that. Like they had these police officers that would stand in the middle of intersection all day and they would have their like striped back and white batons that they would use to control the flow of traffic. And for whatever reason, I was strangely infatuated with this whole process. And like that, that was my dream. That's what I wanted to do when I grew up. And, you know, my parents, both physics profs, by the way, I think were a little concerned with that level of ambition coming from their child. Yeah, that age. Well, it's an interesting I don't know if you can relate, but I very much love that idea. I have OCD nature that I think lends itself very close to the engineering mindset, which is you want to kind of optimize, you know, solve a problem by creating an automated solution, like a set of rules, that set of rules that you follow and thereby make it ultra efficient. I don't know if that's it was of that nature. I certainly have that. There's like fact like SimCity and factory building games, all those kinds of things kind of speak to that engineering mindset. Or did you just like the uniform? I think it was more of the latter. I think it was the uniform and the striped baton that made cars go in the right directions. But I guess, you know, it is I did end up, I guess, you know, working on the transportation industry one way or another. No uniform, no. But that's right. Maybe it was my deep inner infatuation with the traffic control batons that led to this career. OK, what when did you when was the leap from programming to robotics? That happened later. That was after grad school after and actually the most seldom cars was, I think, my first real hands on introduction to robotics. But I never really had that much hands on experience in school and training. I worked on applied math and physics. Then in college, I did more abstract computer science and it was after grad school that I really got involved in robotics. It was actually self-driving cars. And, you know, that was a big flip. What what grad school? So I went to grad school in Michigan and then I did a postdoc at Stanford, which is that was the postdoc where I got to play with self-driving cars. Yeah, so we'll return there. Let's go back to to Moscow. So, you know, for episode 100, I talked to my dad and also I grew up with my dad, I guess. So I had to put up with them for many years. And he he went to the Fistia or M.I.P.T. It's weird to say in English because I've heard all of this in Russian, Moscow Institute of Physics and Technology. And to me, that was like I met some super interesting as a child, I met some super interesting characters. It felt to me like the greatest university in the world, the most elite university in the world. And just the people that I met that came out of there were like. Not only brilliant, but also special humans, it seems like that place really tested the soul, both in terms of technically and spiritually. So that could be just the romanticization of that place. I'm not sure. But so maybe you can speak to it. But is it correct to say that you spent some time at Fistia? Yeah, that's right. Six years. I got my bachelor's and master's and physics and math there. And it's interesting because my my dad and actually both my parents went there. And I think all the stories that I heard, like just like you, Alex, growing up, about the place and how interesting and special and magical it was, I think that was a significant, maybe the main reason I wanted to go there for college. Enough so that I actually went back to Russia from the US. I graduated high school in the US and you went back there. I went back there. Yeah. Wow. Exactly the reaction most of my peers in college had. But perhaps a little bit stronger that point me out as this crazy kid. Were your parents supportive of that? Yeah. Yeah. I came to your previous question. They they supported me in letting me pursue my passions and the things that I was interested in. That's a bold move. Wow. What was it like there? It was interesting. Definitely fairly hardcore on the fundamentals of math and physics and lots of good memories from from those times. So, OK. So Stanford, how did you get into autonomous vehicles? I had the great fortune and great honor to join Stanford's DARPA Urban Challenge team in 2006. This was a third in the sequence of the DARPA challenges. There were two grand challenges prior to that. And then in 2007, they held the DARPA Urban Challenge. So I was doing my postdoc. I joined the team and worked on motion planning for that competition. So, OK, so for people who might not know, I know from from a certain point, autonomous vehicles is a funny world in a certain circle of people. Everybody knows everything. And in a certain circle, nobody knows anything in terms of general public. So it's interesting. It's a good question what to talk about. But I do think that the Urban Challenge is worth revisiting. It's a fun little challenge. One that first like sparked so much. So many incredible minds to focus on one of the hardest problems of our time in artificial intelligence. So that's it's a success from a perspective of a single little challenge. But can you talk about like, what did the challenge involve? So were there pedestrians? Were there other cars? What was the goal? Who was on the team? How long did it take? Any fun, fun sort of specs? Sure, sure, sure. So the way the challenge was constructed, just a little bit of background. As I mentioned, this was the third competition in that series. The first two were the Grand Challenge, called the Grand Challenge. The goal there was to just drive in a completely static environment. You had to drive in a desert. That was very successful. So then DARPA followed with what they called the Urban Challenge, where the goal was to build vehicles that could operate in more dynamic environments and share them with other vehicles. There were no pedestrians there. But what DARPA did is they took over an abandoned Air Force base. And it was kind of like a little fake city that they built out there. And they had a bunch of robots, cars that were autonomous in there, all at the same time, mixed in with other vehicles driven by professional drivers. And each car had a mission. And so there's a crude map that they received at the beginning. And they had a mission, you know, go here and then there and over here. And they kind of all were sharing this environment. At the same time, they had to interact with each other. They had to interact with the human drivers. So it's this very first, very rudimentary version of a self-driving car that could operate in an environment shared with other dynamic actors that, as you said, really, in many ways, kick-started this whole industry. OK, so who was on the team? And how did you do? I forget. I came in second. Perhaps that was my contribution to the team. I think the Stanford team came in first in the DARPA Challenge. But then I joined the team. You were the one with the bug in the code. I mean, do you have sort of memories of some particularly challenging things or... You know, one of the cool things is not, you know, this isn't a product. This isn't the thing that, you know, you have a little bit more freedom to experiment so you can take risks. And there's... So you can make mistakes. Is there interesting mistakes? Is there interesting challenges that stand out to you as something like taught you a good technical lesson or a good philosophical lesson from that time? Yeah, definitely a very memorable time. Not really a challenge, but one of the most vivid memories that I have from that time. And I think that was actually one of the days that really got me hooked on this whole field was the first time I got to run my software on the car. And I was working on a part of our planning algorithm that had to navigate in parking lots. So it was something that called free space motion planning. So the very first version of that was, you know, we tried on the car. I was on Stanford's campus in the middle of the night and had this little course constructed with cones in the middle of a parking lot. So we were there like 3 a.m. By the time we got the code to, you know, compile and turn over. And, you know, it drove. I could actually did something quite reasonable. And I was, of course, very buggy at the time and had all kinds of problems. But it was pretty darn magical. I remember going back and, you know, later trying to fall asleep and just being unable to fall asleep for the rest of the night. Just my mind was blown. And that's what I've been doing ever since for more than a decade. In terms of challenges and interesting memories, like on the day of the competition, it was pretty nerve wrecking. I remember standing there with Mike Montemariello, who was the software lead and wrote most of the code. I think I did one little part of the planner. Mike, you know, incredibly did pretty much the rest of it with a bunch of other incredible people. But I remember standing on the day of the competition, you know, watching the car with Mike. And cars are completely empty, right? They're all there lined up in the beginning of the race. And then, you know, DARPA sends them on their mission one by one. So they leave and like you just they had these sirens. They all had their different silence, right? Each siren had its own personality, if you will. So, you know, off they go and you don't see them. And then every once in a while, they come a little bit closer to where the audience is. And you can kind of hear the sound of your car. And it seems to be moving along. So that gives you hope. And then, you know, it goes away and you can't hear it for too long. You start getting anxious, right? So it's a little bit like, you know, sending your kids to college and like, you know, kind of you invested in them. You hope you build it properly, but it's still anxiety inducing. So that was an incredibly fun few days in terms of, you know, bugs. As you mentioned, you know, one that was my bug that caused us the loss of the first place. It's still a debate that occasionally have with people on the CMU team. CMU came first, I should mention that. CMU haven't heard of them, but yeah, it's a small school. It's really a glitch that, you know, they haven't succeeded something robotics related. Very scenic, though. Most people go there for the scenery. Yeah, it's a beautiful campus. I apologize. Unlike Stanford. So for people, yeah, that's true. Unlike Stanford. For people who don't know, CMU is one of the great robotics and sort of artificial intelligence universities in the world. CMU Carnegie Ballon University. OK, sorry. Go ahead. Good, good PC. So in the part that I contributed to, which was navigating parking lots and the way that part of the mission work is you in a parking lot, you would get from DARPA an outline of the map. You basically get this giant polygon that define the perimeter of the parking lot. And there would be an entrance and maybe multiple entrances or exits to it. And then you would get a goal within that open space, XY heading where the car had to park. It had no information about the obstacles that the car might encounter there. So it had to navigate a completely free space from the entrance to the parking lot into that parking space. And then once it parked there, it had to exit the parking lot. While, of course, in counting and reasoning about all the obstacles that it encounters in real time. So our interpretation, or at least my interpretation of the rules, was that you had to reverse out of the parking spot. And that's what our cars did, even if there's no obstacle in front. That's not what CMU's car did. And it just kind of drove right through. So there's still a debate. And of course, you stop and then reverse out and go out the different way that cost you some time. So there's still a debate whether it was my poor implementation that cost us extra time or whether it was CMU violating an important rule of the competition. And I have my own opinion here in terms of other bugs. And I have to apologize to Mike Montemariella for sharing this on air. But it is actually one of the more memorable ones. And it's something that's kind of become a bit of a metaphor in the industry since then, I think, at least in some circles. It's called the victory lap. And our cars did that. So in one of the missions in the Urban Challenge and one of the courses, there was this big oval right by the start and finish of the race. So the ARPA had a lot of the missions would finish in that same location. And it was pretty cool because you could see the cars come by, kind of finish that part leg of the trip, that leg of the mission, and then go on and finish the rest of it. And other vehicles would come, hit their waypoint and exit the oval and off they would go. Our car, on the other hand, which hit the checkpoint, and then it would do an extra lap around the oval and only then leave and go on its merry way. So over the course of the full day, it accumulated some extra time. And the problem was that we had a bug where it wouldn't start reasoning about the next waypoint and plan a route to get to that next point until it hit a previous one. And in that particular case, by the time you hit that one, it was too late for us to consider the next one and kind of make a lane change. So every time we would do like an extra lap. So that's the Stanford Victory Lap. I feel like there's something philosophically profound in there somehow. But I mean, ultimately, everybody is a winner in that kind of competition. And it led to sort of famously to the creation of Google Self-Driving Car Project and now Waymo. So can we give an overview of how is Waymo born? How is the Google Self-Driving Car Project born? What is the mission? What is the hope? What is it is the engineering kind of set of milestones that it seeks to accomplish? There's a lot of questions in there. Yeah. I don't know. Kind of the DARPA Urban Challenge and the previous DARPA Grand Challenges kind of led, I think, to a very large degree to that next step. And then Larry and Sergey, Larry Page and Sergey Bren, Google founders, I saw that competition and believed in the technology. The Google Self-Driving Car Project was born. You know, at that time, and we started in 2009, it was a pretty small group of us, about a dozen people who came together to work on this project at Google. At that time, we saw an incredible early result in the DARPA Urban Challenge. I think we're all incredibly excited about where we got to. And we believed in the future of the technology, but we still had a very rudimentary understanding of the problem space. So the first goal of this project in 2009 was to really better understand what we're up against. And with that goal in mind, when we started the project, we created a few milestones for ourselves that maximized learnings. Well, the two milestones were, one was to drive 100,000 miles in autonomous mode, which was at that time orders of magnitude that more than anybody has ever done. And the second milestone was to drive 10 routes. Each one was 100 miles long. They were specifically chosen to become extra spicy, extra complicated, and sample the full complexity of that domain. And you had to drive each one from beginning to end with no intervention, no human intervention. So you would get to the beginning of the course, you would press the button that engaged in autonomy, and you had to go for 100 miles, beginning to end with no interventions. And it sampled, again, the full complexity of driving conditions. Some were on freeways. We had one route that went all through all the freeways and all the bridges in the Bay Area. We had some that went around Lake Tahoe and kind of mountainous roads. We had some that drove through dense urban environments like in downtown Palo Alto and through San Francisco. So it was incredibly interesting to work on. It took us just under two years, about a year and a half or a bit more, to finish both of these milestones. And in that process, it was an incredible amount of fun, probably the most fun I had in my professional career. And you're just learning so much. The goal here is to learn a prototype. You're not yet starting to build a production system. So you're just working 24-7 and hacking things together. And you also don't know how hard this is. I mean, that's the point. So that's an ambitious, if I put myself in that mindset, even still, that's a really ambitious set of goals. Like just those two, picking 10 different difficult, spicy challenges and then having zero interventions. So like not saying gradually, we're going to like, you know, over a period of 10 years, we're going to have a bunch of roots and gradually reduce the number of interventions. That literally says, as soon as possible, we want to have zero and on hard roads. So to me, if I was facing that, it's unclear whether that takes two years or whether that takes 20 years. It took us under two. I guess that speaks to a really big difference between doing something once and having a prototype where you are going after learning about the problem versus how you go about engineering a product that where you look at, you properly do evaluation, you look at metrics, you drive down, and you're confident that you can do that. And I guess that's why it took a dozen people 16 months or a little bit more than that back in 2009 and 2010 with the technology of more than a decade ago, that amount of time to achieve that milestone of 10 routes, 100 miles each and no interventions. And it took us a little bit longer to get to a full driverless product that customers use. That's another really important moment. Is there some memories of technical lessons or just one? What did you learn about the problem of driving from that experience? I mean, we can now talk about what you learned from modern day Waymo, but I feel like you may have learned some profound things in those early days, even more so, because it feels like what Waymo is now is to trying to know how to do scale, how to make sure you create a product, how to make sure it's safe, those things, which is all fascinating challenges. But like you were facing the more fundamental philosophical problem of driving in those early days. Like, what the hell is driving as an autonomous? Or maybe I'm again romanticizing it, but is there some valuable lessons you picked up over there at those two years? A ton. The most important one is probably that we believe that it's doable. And we've gotten far enough into the problem that we had, I think, only a glimpse of the true complexity of the domain. It's a little bit like climbing a mountain where you kind of see the next peak and you think that's kind of the summit. But then you get to that and you kind of see that this is just the start of the journey. But we've tried, we've sampled enough of the problem space and we've made enough rapid success, even with technology of 2009, 2010, that it gave us confidence to then pursue this as a real product. So, OK, so the next step, you mentioned the milestones that you had in those two years. What are the next milestones that then led to the creation of Waymo and beyond? It was a really interesting journey. And Waymo came a little bit later than we completed those milestones in 2010. That was the pivot when we decided to focus on actually building a product using this technology. The initial couple of years after that, we were focused on a freeway, what you would call a driver assist, maybe an L3 driver assist program. Then around 2013, we've learned enough about the space and have thought more deeply about the product that we wanted to build that we pivoted. We pivoted towards this vision of building a driver and deploying it fully driverless vehicles without a person. And that's the path that we've been on since then. And it was exactly the right decision for us. So there was a moment where you also considered like, what is the right trajectory here? What is the right role of automation in the task of driving? There was still it wasn't from the early days, obviously, you want to go fully autonomous. From the early days, it was not. I think it was in around 2013, maybe that we've that became very clear and we made the pivot and also became very clear. And that it's either the way you go building a driver assist system is fundamentally different from how you go building a fully driverless vehicle. So we've pivoted towards the latter. And that's what we've been working on ever since. And so that was around 2013. Then there's a sequence of really meaningful for us, really important defining milestones since then. And 2015, we had our first actually the world's first fully driverless trade on public roads. It was in a custom built vehicle that we had. I must have seen those. We called them the Firefly, that funny looking marshmallow looking thing. And we put a passenger, his name was Steve Mann, a great friend of our project from the early days. The man happens to be blind. So we put him in that vehicle. The car had no steering wheel, no pedals. It was an uncontrolled environment. No lead or chase cars, no police escorts. And we did that trip a few times in Austin, Texas. So that was a really big milestone. But that was in Austin. Yeah. OK. And we only, but at that time, it took a tremendous amount of engineering. It took a tremendous amount of validation to get to that point. But we only did it a few times. We only did that. It was a fixed route. It was not kind of a controlled environment, but it was a fixed route. And we only did it a few times. Then in 2016, end of 2016, beginning of 2017, is when we founded Waymo, the company. That's when we kind of that was the next phase of the project, where I wanted we believed in the commercial vision of this technology. And it made sense to create an independent entity within that alphabet umbrella to pursue this product at scale. Beyond that, in 2017, later in 2017, was another really huge step for us, this really big milestone where we started, I think it was October of 2017, where when we started regular driverless operations on public roads, that first day of operations, we drove in one day, in that first day, 100 miles in driverless fashion. And then the most important thing about that milestone was not that 100 miles in one day, but that it was the start of kind of regular ongoing driverless operations. Can we say driverless? It means no driver. That's exactly right. So on that first day, we actually had a mix. And we didn't want to be on YouTube and Twitter that same day. So in many of the rides, we had somebody in the driver's seat, but they could not disengage. The car could not disengage. But actually, on that first day, some of the miles were driven in just completely empty driver's seat. This is the key distinction that I think people don't realize. Oftentimes, when you talk about autonomous vehicles, there's often a driver in the seat that's ready to take over, what's called a safety driver. And then Waymo is really one of the only companies, at least, that I'm aware of, or at least as boldly and carefully and all that actually has cases, and now we'll talk about more and more, where there is literally no driver. So that's another interesting case of where the driver is not supposed to disengage. That's like a nice middle ground. They're still there, but they're not supposed to disengage. But really, there's the case when there's no. OK, there's something magical about there being nobody in the driver's seat. Just like to me, you mentioned the first time you wrote some code for free space navigation of the parking lot. That was like a magical moment. To me, just as an observer of robots, the first magical moment is seeing an autonomous vehicle turn, like make a left turn, apply sufficient torque to the steering wheel to where there's a lot of rotation. And for some reason, and there's nobody in the driver's seat, for some reason that communicates that here's a being with power that makes a decision. There's something about the steering wheel, because we perhaps romanticize the notion of the steering wheel. It's so essential to our conception, our 20th century conception of a car. And it turning the steering wheel with nobody in the driver's seat that to me, I think maybe to others, it's really powerful. Like this thing is in control. And then there's this leap of trust that you give. Like I'm going to put my life in the hands of this thing that's in control. So in that sense, when there's no driver in the driver's seat, that's a magical moment for robots. So I got a chance last year to take a ride in a Waymo vehicle. And that was the magical moment. There's like nobody in the driver's seat. It's like the little details. You would think it doesn't matter whether there's a driver or not. But if there's no driver and the steering wheel is turning on its own, I don't know. It's magical. It's absolutely magical. I've taken many of these rides in a completely empty car. No human in the car pulls up. You call it on your cell phone, it pulls up. You get in, it takes you on its way. There's no driver in the car. There's nobody in the car but you. That's something called fully driverless, our rider only mode of operation. Yeah, it is magical. It is transformative. This is what we hear from our riders. It really changes your experience. And that really is what unlocks the real potential of this technology. But coming back to our journey, that was 2017 when we started truly driverless operations. Then in 2018, we've launched our public commercial service that we called Waymo One in Phoenix. In 2019, we started offering truly driverless rider only rides to our early rider population of users. And then 2020 has also been a pretty interesting year. One of the first ones, less about technology but more about the maturing and the growth of Waymo as a company. We raised our first round of external financing this year. We were part of Alphabet so obviously we have access to significant resources but on the journey of Waymo maturing as a company, it made sense for us to partially go externally in this round. So we raised about $3.2 billion from that round. We've also started putting our fifth generation of our driver, our hardware that is on the new vehicle but it's also a qualitatively different set of self-driving hardware that is now on the JLR PACE so that was a very important step for us. The hardware specs, fifth generation, I think it would be fun to maybe, I apologize if I'm interrupting but maybe talk about maybe the generations with a focus on what we're talking about in the fifth generation in terms of hardware specs, like what's on this car. Sure. So we separated out the actual car that we are driving from the self-driving hardware we put on it. Right now we have, so this is as I mentioned the fifth generation. We've gone through, we started building our own hardware many, many years ago and that Firefly vehicle also had the hardware suite that was mostly design, engineered and built in-house. Lighters are one of the more important components that we design and build from the ground up. So on the fifth generation of our drivers, of our self-driving hardware that we're switching to right now, we have as with previous generations in terms of sensing, we have lighters, cameras and radars and we have a pretty beefy computer that processes all that information and makes new decisions in real time on board the car. So in all of the, and it's really a qualitative jump forward in terms of the capabilities and the various parameters and specs of the hardware compared to what we had before and compared to what you can kind of get off the shelf in the market today. Meaning from fifth to fourth or from fifth to first? Definitely from first to fifth, but also from the fourth. That was the world's dumbest question. Definitely from fourth to fifth, as well as that last step is a big step forward. So everything's in-house, so like lighters built in-house and cameras are built in-house? It's different. We work with partners and there's some components that we get from our manufacturing and supply chain partners. What exactly is in-house is a bit different. We do a lot of custom design on all of our sensing models. There's lighters, radars, cameras. Exactly there's lighters are almost exclusively in-house and some of the technologies that we have, some of the fundamental technologies there are completely unique to Waymo. That is also largely true about radars and cameras. It's a little bit more of a mix in terms of what we do ourselves versus what we get from partners. Is there something super sexy about the computer that you can mention that's not top secret? Like for people who enjoy computers. There's a lot of machine learning involved, but there's a lot of just basic compute. You have to probably do a lot of signal processing on all the different sensors. You have to integrate everything. It has to be in real time. There's probably some kind of redundancy type of situation. Is there something interesting you can say about the computer for the people who love hardware? It does have all of the characteristics, all the properties that you just mentioned. Redundancy, very beefy compute for general processing as well as inference and ML models. It is some of the more sensitive stuff that I don't want to get into for IP reasons, but we've shared a little bit in terms of the specs of the sensors that we have on the car. We actually shared some videos of what our lighter see in the world. We have 29 cameras. We have five lighters. We have six radars on these vehicles. You can kind of get a feel for the amount of data that they're producing. That all has to be processed in real time to do perception, to do complex reasoning. That kind of gives you some idea of how beefy those computers are, but I don't want to get into specifics of exactly how we build them. Okay. Well, let me try some more questions that you can get into the specifics of, like GPU-wise. Is that something you can get into? I know that Google works with GPUs and so on. I mean, for machine learning folks, it's kind of interesting. Or is there no... How do I ask it? I've been talking to people in the government about UFOs and they don't answer any questions. So this is how I feel right now asking about GPUs. But is there something interesting that you could reveal or leave it up to our imagination, some of the compute? Is there any, I guess, is there any fun trickery? Like I talked to Chris Latner for a second time and he was a key person about GPUs. And there's a lot of fun stuff going on in Google in terms of hardware that optimizes for machine learning. Is there something you can reveal in terms of how much you mentioned customization, how much customization there is for hardware for machine learning purposes? I'm going to be like that government, you know, you've got guy versus body of foes. But I guess I will say that it's really compute. It's really important. We have very data hungry and compute hungry ML models of all over our stack. And this is where both being part of Alphabet as well as designing our own sensors and the entire hardware suite together, where on one hand, you get access to like really rich raw sensor data that you can pipe from your sensors into your compute platform and build like build the whole pipe from sensor, raw sensor data to the big compute as then have the massive compute to process all that data. And this is where we're finding that having a lot of control of that hardware part of the stack is really advantageous. One of the fascinating magical places to me, again, might not be able to speak to the details, but it is the other compute, which is like, you know, this we're just talking about a single car, but the, you know, the driving experience is a source of a lot of fascinating data and you have a huge amount of data coming in on the car and on the car and, you know, the infrastructure of storing some of that data to then train or to analyze or so on. That's a fascinating like piece of it that that I understand a single car. I don't understand how you pull it all together. In a nice way, is that something that you could speak to in terms of the challenges of, of seeing the network of cars and then bringing the data back and analyzing things that went like, like edge cases of driving, be able to learn on them, to improve the system, to, to see where things went wrong, where things went right and analyze all that kind of stuff. Is there something interesting there in the, from an engineering perspective? Oh, there's an incredible, incredible, incredible, incredible amount of really interesting work that's happening there, both in the, you know, the real time operation of the fleet of cars and the information that they exchange with each other in real time to make better decisions, as well as on the kind of the off board component where you have to deal with massive amounts of data for training your ML models, evaluating the ML models, for simulating the entire system and for evaluating your entire system. And this is where being part of Alphabet has once again been tremendously advantageous. I think we consume an incredible amount of compute for ML infrastructure. We build a lot of custom frameworks to get good at, on data mining, finding the interesting edge cases for training and for evaluation of the system, for both training and evaluating some components and your sub parts of the system in various ML models, as well as the evaluating the entire system and simulation. Okay, that first piece that you mentioned that cars communicating to each other, essentially, I mean, through perhaps through a centralized point, but what, that's fascinating too. How much does that help you? Like if you imagine, you know, right now the number of Waymo vehicles is whatever X, I don't know if you can talk to what that means. I don't know what that number is, but it's not in the hundreds of millions yet. And imagine if the whole world is Waymo vehicles, like that changes potentially the power of connectivity. Like the more cars you have, I guess, actually, if you look at Phoenix, because there's enough vehicles, there's enough, when there's like some level of density, you can start to probably do some really interesting stuff with the fact that cars can negotiate, can communicate with each other and thereby make decisions. Is there something interesting there that you can talk to about like, how does that help with the driving problem from as compared to just a single car solving the driving problem by itself? Yeah, it's a spectrum. I first say that, you know, it helps and it helps in various ways, but it's not required right now. The way we build our system, like each cars can operate independently, they can operate with no connectivity. So I think it is important that you have a fully autonomous, fully capable driver, computerized driver that each car has. Then, you know, they do share information and they share information in real time. It really, really helps. All right. So the way we do this today is, you know, whenever one car encounters something interesting in the world, whether it might be an accident or a new construction zone, that information immediately gets, you know, uploaded over the air and it's propagated to the rest of the fleet. So that's kind of how we think about maps as priors in terms of the knowledge of our drivers, of our fleet of drivers that is distributed across the fleet and it's updated in real time. So that's one use case you can imagine as the density of these vehicles go up that they can exchange more information in terms of what they're planning to do and start influencing how they interact with each other, as well as potentially sharing some observations to help with if you have enough density of these vehicles where one car might be seeing something that another is relevant to another car that is very dynamic. It's not part of kind of your updating your static prior of the map of the world, but it's more of a dynamic information that could be relevant to the decisions that another car is making in real time. So you can see them exchanging that information and you can build on that. But again, I see that as an advantage, but it's not a requirement. So what about the human in the loop? So when I got a chance to drive with a ride in a Waymo, you know, there's customer service, so like there is somebody that's able to dynamically like tune in and help you out. What role does the human play in that picture? That's a fascinating like, you know, the idea of teleoperation, be able to remotely control a vehicle. So here what we're talking about is like, like frictionless, like a human being able to in a frictionless way sort of help you out. I don't know if they're able to actually control the vehicle. Is that something you could talk to? Yes. Okay. To be clear, we don't do teleoperation. I'm going to believe in teleoperation for a reason. That's not what we have in our cars. We do, as you mentioned, have a version of customer support. We call it life health. In fact, we find it that it's very important for our rider experience, especially if it's your first trip. You've never been in a fully driverless rider only way more vehicle. You get in, there's nobody there. And so you can imagine having all kinds of questions in your head, like how this thing works. So we've put a lot of thought into kind of guiding our riders, our customers through that experience, especially for the first time, they get some information on the phone. If the fully driverless vehicle is used to service their trip, when you get into the car, we have an in-car screen and audio that kind of guides them and explains what to expect. They also have a button that they can push that will connect them to a real life human being that they can talk to about this whole process. So that's one aspect of it. I should mention that there is another function that humans provide to our cars, but it's not teleoperation. You can think of it a little bit more like fleet assistance, kind of like traffic control that you have where our cars, again, they're responsible on their own for making all of the driving decisions that don't require connectivity. Anything that is safety or latency critical is done purely autonomously by our onboard system. But there are situations where if connectivity is available and a car encounters a particularly challenging situation, you can imagine like a super hairy scene of an accident, the cars will do their best. They will recognize that it's an off-nominal situation. They will do their best to come up with the right interpretation and the best course of action in that scenario. But if connectivity is available, they can ask for confirmation from a human assistant to kind of confirm those actions and perhaps provide a little bit of kind of contextual information and guidance. So October 8th was when you're talking about the, was Waymo launched the public version of its fully driverless, that's the right term, I think, service in Phoenix. Is that October 8th? That's right. Yeah, it was the introduction of fully driverless, right, or only vehicles into our public Waymo One service. Okay. So that's amazing. So it's like anybody can get into a Waymo in Phoenix? That's right. So we previously had early people in our early rider program taking fully driverless rides in Phoenix. And just a little while ago, we opened on October 8th, we opened that mode of operation to the public. So I can download the app and go on a ride. There is a lot more demand right now for that service than we have capacity. So we're kind of managing that, but that's exactly the way you described it. Yeah, well, that's interesting. So there's more demand than you can handle. Like what has been the reception so far? Like what, I mean, okay, so, you know, this is a product, right? That's a whole nother discussion of like how compelling of a product it is. Great. But it's also like one of the most kind of transformational technologies of the 21st century. So it's also like a tourist attraction. Like it's fun to, you know, to be a part of it. So it'd be interesting to see like, what do people say? What do people, what have been the feedback so far? Still early days, but so far the feedback has been incredible, incredibly positive. They, you know, we asked them for feedback during the ride. We asked them for feedback after the ride as part of their trip. We asked them some questions. We asked them to rate the performance of our driver. Most by far, you know, most of our drivers give us five stars in our app. Which is absolutely great to see. And, you know, that's, and they're also giving us feedback on, you know, things we can improve. And, you know, that's, that's one of the main reasons we're doing this is Phoenix. And, you know, over the last couple of years and every day today, we are just learning a tremendous amount of new stuff from our users. There's no substitute for actually doing the real thing, actually having a fully driverless product out there in the field with, you know, users that are actually paying us money to get from point A to point B. So this is a legitimate, like there's a paid service. That's right. And the idea is you use the app to go from point A to point B. And then what, what are the A's, what are the, what's the freedom of the, of the starting and ending places? It's an area of geography where that service is enabled. It's a decent size of geography of territory. It's actually larger than, you know, the size of San Francisco. And, you know, within that you have full freedom of, you know, selecting where you want to go. You know, of course there's some, and you, on your app, you get a map, you tell the car where you want to be picked up, you know, and where you want the car to pull over and pick you up. And then you tell it where you want to be dropped off. All right. And of course there are some exclusions, right? You don't want to be, you know, you, where, in terms of where the car is allowed to pull over, right? So, you know, that you can't do, but besides that, it's amazing. It's not like a fixed, just would be very exciting. I guess, I don't know, maybe that's what's the question behind your question, but it's not a, you know, preset set of, uh, Yes, I guess. So within the geographic constraints with that, within that area, anywhere else, it can be, you can be picked up and dropped off anywhere. That's right. And, you know, people use them on like all kinds of trips they, we have, and we have an incredible spectrum of riders. I think the youngest actually have car seats, them and we have, you know, people taking their kids and rides. I think the youngest riders we had on cars are, you know, one or two years old, you know, and the full spectrum of use cases. People can take them to, you know, schools, uh, to, you know, go grocery stock shopping, to restaurants, to bars, you know, run errands, you know, go shopping, et cetera, et cetera. You can go to your office, right? Uh, like the full spectrum of use cases and, uh, people, you know, use them in their daily lives to get around. Uh, and we see all kinds of really interesting use cases and that, that, that's providing us incredibly valuable experience, uh, that we then, you know, use to improve our product. So as somebody who's been on, done a few long rants with Joe Rogan and others about the toxicity of the internet and the comments and the negativity in the comments, I'm fascinated by feedback. I, I believe that most people are good and kind and intelligent and can provide like, uh, even in disagreement, really fascinating ideas. So on a product side, it's fascinating to me, like, how do you get the richest possible user feedback? Like to improve what's, what are the channels that you use to measure? Cause like you're, you're no longer, that's one of the magical things about autonomous vehicles is it's not like it's frictionless interaction with the human. So like you don't get to, you know, it's just giving a ride. So like, how do you get feedback from people to, in order to improve? Uh, yeah, great question. Various mechanisms. So as part of the normal flow, we ask people for feedback. They, as the car is driving around, we have on the phone and in the car, and we have a touchscreen in the car, uh, you can actually click some buttons and provide, uh, real-time feedback on how the car is doing, uh, and how the car is handling a particular situation, you know, both positive and negative. Uh, so that's one channel. Uh, we have, as we discussed, customer support or life help where, you know, if a customer wants to, has a question, uh, or he has some sort of concern, they can talk to a person in real time. So that, that is another mechanism that gives us, uh, feedback. Uh, at the end of a trip, you know, we also ask them how things went. They give us, uh, comments and, you know, star rating and, you know, if it's, uh, we also, you know, ask them, uh, to, uh, uh, explain what, you know, went well and, you know, what could be improved. And, uh, we, we have, uh, our writers are providing, you know, very rich, uh, feedback there, uh, a lot, a large fraction is, uh, very passionate and very excited about this technology. So we get really good feedback. Uh, we also run, uh, UXR studies, right? You know, specific, uh, that are kind of more, you know, go more in depth and we will run both kind of, uh, lateral and longitudinal studies. Um, where we have deeper engagement, uh, with our customers, you know, we have our user experience research team tracking over time. That's things about legitude. No, that's cool. That's exactly right. And, you know, that's another really valuable, uh, feedback, uh, source of feedback. And we were just covering a tremendous amount, right? Uh, people go grocery shopping and they like want to load, you know, 20 bags of groceries in our cars and like that. That's one workflow that you maybe don't think about, uh, you know, getting just right when you're building the driverless product. I have people like, you know, who, uh, bike as part of their trip. So they, you know, bike somewhere, then they get on our cars, they take a path, part of their bike, they load into our vehicle. Then they go and that's, you know, how they, you know, where we want to pull over and how that, you know, uh, get in and get out, um, uh, process works, uh, provides us very, uh, useful feedback in terms of, you know, what makes a good, uh, pick up and drop off location. Uh, we get really valuable feedback. Yeah. And in fact, we had to, um, uh, do some really interesting work with, uh, high definition maps and, uh, thinking about walking directions. If you imagine you're in a store, right? In some giant space and then, you know, you want to be picked up somewhere. I could, if you just drop a pin at a current location, which is maybe in the middle of a shopping mall, like what's the best location for the car to come pick you up. And you can have simple heuristics where you're just going to take your, you know, you clean in distance, uh, and find the nearest, uh, spot where the car can pull over that's closest to you. But oftentimes that's not the most convenient one. You know, I have many anecdotes where that heuristic breaks in horrible ways. Uh, one example, uh, that I often mentioned is, uh, somebody wanted to be, you know, uh, dropped off, uh, and Phoenix, uh, and, you know, we, our car picked a location, uh, that was close, the closest to their, you know, where the pin was dropped on the map in terms of, you know, latitude and longitude. Uh, but it happened to be on the other side of a parking lot that had this row of cacti and the poor person had to like walk all around the parking lot to get to where they wanted to be in 110 degree heat. So that, you know, that was a buck. So then, you know, we took all, take all of these, um, all of that feedback from our users and, uh, incorporate it into our system and, uh, improve it. Yeah. I feel like that's like requires AGI to solve the problem of like when you're, which is a very common case when you're in a big space of some kind, like apartment building, it doesn't matter. It's not some large space. And then you call the, like the Waymo from there, right? It's like, so any, whatever it doesn't matter, ride share vehicle. And like, where's the pin supposed to drop? I feel like that's, I, you don't think, I think that requires AGI. I'm going to, okay, the alternative, which I think the Google search engine is taught is like, there's something really valuable about the perhaps slightly dumb answer, but a really powerful one, which is like what was done in the past by others, like what was the choice made by others? That seems to be like in terms of Google search, when you have like billions of searches that you could, you could see which, like when they recommend what you might possibly mean, they suggest based on not some machine learning thing, which they also do, but like on what was successful for others in the past and finding a thing that they were happy with, is that integrated at all the way more like what, what pickups worked for others? It is. I think you're exactly right. So there's a real, it's an interesting problem. Naive solutions have interesting failure modes. So there's definitely lots of things that can be done to improve and both learning from, you know, what works, what doesn't work in actual hail from, you know, getting richer data and getting more information about the environment and, you know, richer maps. But you're absolutely right that there's something, I think there's some properties of solutions that in terms of the effect that they have on users, so much, much, much better than others, right? And predictability and understandability is important. So you can have maybe something that is not quite as optimal, but is very natural and predictable to the user and kind of works the same way all the time. And that matters. That matters a lot for the user experience. But, you know, to get to the basics, the pretty fundamental property is that the car actually arrives where you told it to arrive. Like you can always, you know, change it, see it on the map and you can move it around if you don't like it. But like that property that the car actually shows up on pin is critical, which, you know, where compared to some of the human driven analogs, I think, you know, you can have more predictability. It's actually the fact, if I have a little bit of a detour here, I think the fact that it's your phone and the car, two computers talking to each other, can lead to some really interesting things we can do in terms of the user interfaces, both in terms of function, like the car actually shows up exactly where you told it you want it to be, but also some really interesting things on the user interface. Like as the car is driving, as you call it, and it's on the way to come and pick you up, and of course you get the position of the car and the route on the map, but and they actually follow that route, of course, but it can also share some really interesting information about what it's doing. So, you know, our cars, as they are coming to pick you up, if it's come, if a car is coming up to a stop sign, it will actually show you that like it's there sitting because it's at a stop sign or a traffic light will show you that it's got sitting at a red light. So, you know, like little things, right? But it's I find those little touch touches really interesting, really magical. And it's just, you know, little things like that that you can do to kind of delight your users. You know, this makes me think of there's some products that I just love. Like there's a there's a company called Rev Rev.com where I like for this podcast, for example, I can just drag and drop a video and then they do all the captioning. It's humans doing the captioning, but they connect. They automate everything of connecting you to the humans and they do the captioning and transcription. It's all effortless. And it like I remember when I first started using them, it was like, life's good. Like because it was so painful to figure that out earlier. The same thing with something called the iZotope RX. This company I use for cleaning up audio, like the sound cleanup they do. It's like drag and drop and it just cleans everything up very nicely. Another experience like that I had with Amazon one click purchase. First time. I mean, other places do that now, but just the effortlessness of purchasing, making it frictionless. It kind of communicates to me like I'm a fan of design. I'm a fan of products that you can just create a really pleasant experience. The simplicity of it, the elegance just makes you fall in love with it. So on the, do you think about this kind of stuff? I mean, it's exactly what we've been talking about. It's like the little details that somehow make you fall in love with the product. Is that we went from like urban challenge days where, where love was not part of the conversation probably. And to this point where there's a, where there's human beings and you want them to fall in love with the experience. Is that something you're trying to optimize for? Try to think about like, how do you, how do you create experience that people love? Absolutely. That's the vision is removing any friction or complexity from getting our users, our writers to where they want to go. And making that as simple as possible. And then, you know, beyond that, just transportation. Making, you know, things and goods get to their destination as seamlessly as possible. And you talked about a drag and drop experience where I kind of express your intent and then you know, it just magically happens. And for our writers, that's what we're trying to get to is you download an app and you click and car shows up. It's the same car. It's very predictable. It's a safe and high quality experience. And then it gets you in a very reliable, very convenient, frictionless way to where you want to be. And along the journey, I think we also want to do little things to delight our users. Like the ride sharing companies, because they don't control the experience, I think, they can't make people fall in love necessarily with the experience. Or maybe they haven't put in the effort, but I think if I were to speak to the ride sharing experience I currently have, it's just very convenient. But there's a lot of room for like falling in love with it. Like we can speak to sort of car companies. Car companies do this well. You can fall in love with a car, right? And be like a loyal car person, like whatever. Like I like badass hot rods. I guess 69 Corvette. And at this point, you know, you can't really... Cars are so... Owning a car is so 20th century, man. But is there something about the Waymo experience where you hope that people will fall in love with it? Because is that part of it? Or is it just about making a convenient ride, not ride sharing, I don't know what the right term is, but just a convenient A to B autonomous transport? Or like, do you want them to fall in love with Waymo? So maybe elaborate a little bit. I mean, almost like from a business perspective, I'm curious. Like how... Do you want to be in the background invisible or do you want to be like a source of joy that's in very much in the foreground? I want to provide the best, most enjoyable transportation solution. And that means building it, building our product and building our service in a way that people do use in a very seamless, frictionless way in their day-to-day lives. And I think that does mean, you know, in some way falling in love in that product, right? It just kind of becomes part of your routine. It comes down, in my mind, to safety, predictability of the experience, and privacy, I think, aspects of it, right? Our cars, you get the same car, you get very predictable behavior, and that is important, I think, if you're going to use it in your daily life. Privacy, and when you're in a car, you can do other things. You're spending a bunch, just another space where you're spending a significant part of your life, right? So not having to share it with other people who you don't want to share it with, I think, is a very nice property. Maybe you want to take a phone call or do something else in the vehicle. And, you know, safety on the quality of the driving, as well as the physical safety of not having to share that ride is important to a lot of people. What about the idea that when there's somebody, like a human driving, and they do a rolling stop on a stop sign, like sometimes, like, you know, you get a new bro lift or whatever, like human driver, and, you know, they can be a little bit aggressive as drivers. It feels like there's not all aggression is bad. Now, that may be a wrong, again, 20th century conception of driving. Maybe it's possible to create a driving experience. Like, if you're in the back busy doing something, maybe aggression is not a good thing. It's a very different kind of experience, perhaps. But it feels like in order to navigate this world, you need to, how do I phrase this? You need to kind of bend the rules a little bit, or at least like test the rules. I don't know what language politicians use to discuss this, but whatever language they use, you like flirt with the rules. I don't know. But like you sort of have a bit of an aggressive way of driving that asserts your presence in this world, thereby making other vehicles and people respect your presence, and thereby allowing you to sort of navigate through intersections in a timely fashion. I don't know if any of that made sense, but like how does that fit into the experience of driving autonomously? Is that- Makes a lot of sense. This is, you're hitting on a very important point of a number of behavioral components and, you know, parameters that make your driving feel assertive and natural, comfortable, predictable. Now, our cars will follow rules, right? They will do the safest thing possible in all situations, let me be clear on that. But if you think of really, really good drivers, just think about professional limo drivers, right? They will follow the rules. They're very, very smooth, and yet they're very efficient. But they're assertive. They're comfortable for the people in the vehicle. They're predictable for the other people outside the vehicle that they share the environment with. And that's the kind of driver that we want to build. And you just need to think if, you know, maybe there's a sport analogy there, right? You can do in very many sports, the true professionals are very efficient in their movements, right? They don't do like, you know, hectic flailing, right? They're, you know, smooth and precise, right? And they get the best results. So that's the kind of driver that we want to build. In terms of, you know, aggressiveness, yeah, you can like, you know, roll through the stop signs, you can do crazy lane changes. Typically, it doesn't get you to your destination faster. Typically not the safest or most predictable, most comfortable thing to do. And but there is a way to do both. And that's what we're doing. We're trying to build a driver that is safe, comfortable, smooth, and predictable. Yeah, that's a really interesting distinction. I think in the early days of autonomous vehicles, the vehicles felt cautious as opposed to efficient. And still probably. But when I rode in the Waymo, I mean, there was, it was quite assertive. It moved pretty quickly. Like, yeah, then he's one of the surprising feelings was that it actually, it went fast. And it didn't feel like awkwardly cautious than autonomous vehicles. Like, like, so I've also programmed autonomous vehicles and everything I've ever built was felt awkwardly either overly aggressive. Okay. Especially when it was my code or like awkwardly cautious is the way I would put it. And Waymo's vehicle felt like assertive and I think efficient is like the right terminology here. They wasn't. And I also like the professional limo driver because we often think like, you know, an Uber driver or a bus driver or taxi. This is the funny thing is people think they track taxi drivers are professionals. I mean, it's, it's like that. That's like saying I'm a professional walker just because I've been walking all my life. I think there's an art to it. Right. And if you take it seriously as an art form, then there's a certain way that mastery looks like. It's interesting to think about what is mastery look like in driving and perhaps what we associate with like aggressive driving. That's like a necessary fluff that efficiency you could. You can be, you can create a good driving experience within the rules. That's a year, the first person to tell me this, so it's kind of interesting. I need to think about this, but that's exactly what it felt like with Waymo. I kind of like the way it felt like. I think that's the way it felt like. I think it's kind of like an idea. I think it's like a, I'm not sure if I'm going to think about this, but that's exactly what it felt like with Waymo. I kind of had this intuition. Maybe it's the Russian thing. I don't know that you have to break the rules in life to get anywhere, but maybe, maybe it's possible that that's not the case in driving. I have to think about that, but it certainly felt that way on the streets of Phoenix when I was there in Waymo. It wasn't frustrating in that like, come on, move already kind of feeling. That wasn't there. Yeah, I mean, that's what we're going after. I don't think you have to pick one. I think truly good driving gives you both efficiency, assertiveness, but also comfort and predictability and safety. And that's what fundamental improvements in the core capabilities truly unlock. And you can kind of think of it as, you know, precision and recall trade off. You have certain capabilities of your model, and then it's very easy when you have some curve of precision and recall, you can move things around and can choose your operating point. And you're trading off precision versus recall, false positives versus false negatives. Right. But then, and you know, you can tune things on that curve and be kind of more cautious or more aggressive, but then aggressive is bad or, you know, cautious is bad. But true capabilities come from actually moving the whole curve up. And then you are kind of on a very different plane of those trade-offs. And that's what, you know, we're trying to do here is to move the whole curve up. Before I forget, let's talk about trucks a little bit. So I also got a chance to check out some of the Waymo trucks. I'm not sure if we want to go too much into that space, but it's a fascinating one. So maybe we can mention at least briefly, you know, Waymo is also now doing autonomous, autonomous, autonomous, autonomous, autonomous. Trucking and how different like philosophically and technically is that whole space of problems? It's one of our two big products and commercial applications of our driver, right? Right. Healing and deliveries. We have Waymo One and Waymo Via, moving people and moving goods. Trucking is an example of moving goods. We've been working on trucking since 2017. It is a very interesting space. And you're a question of how different is it? It has this really nice property that the first order challenges like the science, the hard engineering, whether it's hardware or onboard software or off-board software, all of the systems that you build for training your ML models for evaluating your entire system. Those fundamentals carry over the true challenges of driving, perception, semantic understanding, prediction, decision-making, planning, evaluation, the simulator, ML infrastructure. Those carry over the data and the application and the domains might be different, but the most difficult problems, all of that carries over between the domains. So that's very nice. So that's how we approach it. We're kind of building investing in the core, the technical core. And then there is specialization of that core technology to different product lines, to different commercial applications. So just to tease it apart a little bit, on trucks, starting with the hardware, the configuration of the sensors is different. They're different physically, geometrically, different vehicles. So for example, we have two of our main laser on the trucks on both sides so that we have the blind spots, whereas on the GLR I-PACE, we have one of it sitting at the very top, but the actual sensors are almost the same or largely the same. So all of the investment that over the years we've put into building our custom lighters, custom radars, putting the whole system together, that carries over very nicely. Then on the perception side, the fundamental challenges of seeing, understanding the world, whether it's object detection, classification, tracking, semantic understanding, all that carries over. Yes, there's some specialization when you're driving on freeways, range becomes more important. The domain is a little bit different, but again, the fundamentals carry over very, very nicely. Same, and I guess you get into prediction or decision making. The fundamentals of what it takes to predict what other people are going to do, to find the long tail, to improve your system in that long tail of behavior, prediction and response, that carries over, right? And so on and so on. So I mean, that's pretty exciting. By the way, does Waymovia include using the smaller vehicles for transportation of goods? That's an interesting distinction. So I would say there's three interesting modes of operation. So one is moving humans, one is moving goods, and one is like moving nothing, zero occupancy, meaning like you're going to the destination, your empty vehicle. The third is the less of it, that's the entirety of it, it's the less exciting from the commercial perspective. Well, I mean, in terms of like, if you think about what's inside a vehicle as it's moving, because it does, some significant fraction of the vehicle's movement has to be empty. I mean, it's kind of fascinating, maybe just on that small point, is there different control and like policies that are applied for zero occupancy vehicle? So a vehicle with nothing in it, or is it just move as if there is a person inside? What was with some subtle differences? As a first order approximation, there are no differences. And if you think about safety and comfort and quality of driving, only part of it has to do with the people or the goods inside of the vehicle. But you want to drive smoothly, as we discussed, not for the purely for the benefit of whatever you have inside the car, right? It's also for the benefit of the people outside kind of feeding, fitting naturally and predictably into that whole environment, right? So yes, there are some second order things you can do. It's going to change your route and optimize maybe kind of your fleet things at the fleet scale. And you would take into account whether some of your cars are actually serving a useful trip, whether with people or with goods, whereas other cars are driving completely empty to that next valuable trip that they're going to provide. But those are mostly second order effects. OK, cool. So Phoenix is an incredible place. And what you've announced in Phoenix is kind of amazing. But that's just like one city. How do you take over the world? I mean, I'm asking for a friend. One step at a time. Cartoon pinky in the brain. But gradually is a true answer. So I think the heart of your question is... Can you ask a better question than I asked? You're asking a great question. Answer that one. I'm just going to phrase it in the terms that I want to answer. That's exactly right. Brilliant. Where are we today and what happens next? And what does it take to go beyond Phoenix and what does it take to get this technology to more places and more people around the world? All right. So our next big area of focus is exactly that. Larger scale commercialization and scaling up. If I think about the main... And Phoenix gives us that platform and gives us that foundation upon which we can build. And there are few really challenging aspects of this whole problem that you have to pull together in order to build the technology that we need. To build the technology in order to deploy it into the field. To go from a driverless car to a fleet of cars that are providing a service. And then all the way to commercialization. And this is what we have in Phoenix. We've taken the technology from a proof point to an actual deployment. And we've taken our driver from one car to a fleet that can provide a service. Beyond that, if I think about what it will take to scale up and deploy in more places with more customers, I tend to think about three main dimensions, three main axes of scale. One is the core technology, the hardware and software, core capabilities of our driver. The second dimension is evaluation and deployment. And the third one is the product, commercial and operational excellence. So you can talk a bit about where we are along each one of those three dimensions, about where we are today and what will happen next. On the core technology, on the hardware and software together, comprised of driver, we obviously have that foundation that is providing fully driverless trips to our customers as we speak, in fact. And we've learned a tremendous amount from that. So now what we're doing is we are incorporating all those lessons into some pretty fundamental improvements in our core technology, both on the hardware side and on the software side, to build a more general, more robust solution that then will enable us to massively scale in the young Phoenix. So on the hardware side, all of those lessons are now incorporated into this fifth generation hardware platform that is being deployed right now. And that's the platform, the fourth generation, the thing that we have right now driving in Phoenix, it's good enough to operate fully driverless, night and day, at various speeds and various conditions. But the fifth generation is the platform upon which we want to go to massive scale. We've really made qualitative improvements in terms of the capability of the system, the simplicity of the architecture, the reliability of the redundancy. It is designed to be manufacturable at very large scale and provides the right unit economics. So that's the next big step for us on the hardware side. That's already there for scale, the version five. That's right. And is that coincidence or should we look into a conspiracy theory that it's the same version as the Pixel phone? Is that what's the hardware? We can neither confirm nor deny it looks. All right, cool. So sorry. So that's that axis. What else? So similarly, hardware is a very discreet jump, but similarly to how we're making that change from the fourth generation hardware to the fifth, we're making similar improvements on the software side to make it more robust and more general and allow us to quickly scale beyond Phoenix. So that's the first dimension of core technology. The second dimension is evaluation and deployment. Now, how do you measure your system? How do you evaluate it? How do you build the release and deployment process where, you know, with confidence, you can regularly release new versions of your driver into a fleet? How do you get good at it so that it is not, you know, a huge tax on your researchers and engineers? That, you know, so you can kind of how do you build all these processes, the frameworks, the simulation, the evaluation, the data science, the validation so that, you know, people can focus on improving the system and kind of the releases just go out the door and get deployed across the fleet. So we've gotten really good at that in Phoenix. That's been a tremendously difficult problem, but that's what we have in Phoenix right now. That gives us that foundation. And now we're working on kind of incorporating all the lessons that we've learned to make it more efficient to go to new places, you know, scale up and just kind of stamp things out. So that's that second dimension of evaluation and deployment. And the third dimension is product, commercial and operational excellence. And again, Phoenix there is providing an incredibly valuable platform. You know, that's why we're doing things end to end in Phoenix. We're learning, as we discussed a little earlier today, tremendous amount of really valuable lessons from our users getting really incredible feedback. And we'll continue to iterate on that and incorporate all those lessons into making our product, you know, even better and more convenient for our users. So you're converting this whole process of Phoenix in Phoenix into something that could be copied and pasted elsewhere. So like, perhaps you didn't think of it that way when you were doing the experimentation in Phoenix, but so how long did you basically, I mean, you can correct me, but you've been taking, I mean, it's still early days, but you've taken the full journey in Phoenix, right? As you were saying of like what it takes to basically automate. I mean, it's not the entirety of Phoenix, right? But I imagine it can encompass the entirety of Phoenix. That's some near term date, but that's not even perhaps important. Like as long as it's a large enough geographic area. So what, how copy pasteable is that process currently? And how, like, you know, like when you copy and paste in Google docs, I think, no, in or in word, you can like apply source formatting or apply destination formatting. So when you copy and paste the Phoenix into like, say Boston, how do you apply the destination formatting? Like how much of the core of the entire process of bringing an actual public transportation, autonomous transportation service to a city is there in Phoenix that you understand enough to copy and paste into Boston or wherever? So we're not quite there yet. We're not at a point where we're kind of massively copying pasting all over the place. But Phoenix, what we did in Phoenix, and we very intentionally have chosen Phoenix as our first full deployment area, you know, exactly for that reason to kind of tease the problem apart, look at each dimension, focus on the fundamentals of complexity and de-risking those dimensions, and then bringing the entire thing together to get all the way and force ourselves to learn all those hard lessons on this technology, hardware and software, on the evaluation deployment, on operating a service, operating a business using, actually serving our customers all the way so that we're fully informed about the most difficult, most important challenges to get us to that next step of massive copy and pasting, as you said. And that's what we're doing right now. We're incorporating all those things that we learned into that next system that then will allow us to kind of copy and paste all over the place and to massively scale to more users and more locations. And we just talked a little bit about what does that mean along those different dimensions. So on the hardware side, for example, again, it's that switch from the fourth to the fifth generation. And the fifth generation is designed to kind of have that property. Can you say what other cities you're thinking about? Like I'm thinking about, sorry, we're in San Francisco now. I thought I want to move to San Francisco, but I'm thinking about moving to Austin. I don't know why people are not being very nice about San Francisco currently. Maybe it's a small, maybe it's in vogue right now, but Austin seems, I visited there and it was, I was in a Walmart. It's funny these moments like turn your life. There's this very nice woman with kind eyes just like stopped and said, he looks so handsome in that tie honey to me. This has never happened to me in my life, but just the sweetness of this woman is something I've never experienced, certainly in the streets of Boston, but even in San Francisco where people wouldn't, that's just not how they spend their time. Or think, I don't know. There's a warmth to Austin that love. And since Waymo does have a little bit of a history there, is that a possibility? Is this your version of asking the question of like, you know, Demetri, I know you can't share your commercial and deployment roadmap, but I'm thinking about moving to San Francisco of Austin, like in a blink twice. If you think I should move to that's true. That's true. You got me. We've been testing in all over the place. I think we've been testing in more than 25 cities. We drive in San Francisco, we drive in Michigan for snow. We are doing significant amount of testing in the Bay Area, including San Francisco. Which is not like, because we're talking about the very different thing, which is like a full on large geographic area public service. You can't share. Okay. What about Moscow? When is that happening? Take on Yandex. I'm not paying attention to those folks. They're doing, you know, there's a lot of fun. I mean, maybe as a way of a question, you didn't speak to sort of like policy or like, is there tricky things with government and so on? Like, is there other friction that you've encountered except sort of technological friction of solving this very difficult problem? Is there other stuff that you have to overcome when deploying a public service in a city? That's interesting. It's very important. So we put significant effort in creating those partnerships and those relationships with governments at all levels, local governments, municipalities, state level, federal level. We've been engaged in very deep conversations from the earliest days of our projects. At all of these levels, whenever we go to test or operate in a new area, we always lead with a conversation with the local officials. But the result of that investment is that no, it's not challenges we have to overcome, but it is very important that we continue to have this conversation. Oh, yeah. I love politicians too. Okay. So Mr. Elon Musk said that LIDAR is a crutch. What are your thoughts? I wouldn't characterize it exactly that way. Okay. I know. I think LIDAR is very important. It is a key sensor that we use just like other modalities. As we discussed, our cars use cameras, LIDARs, and radars. They are all very important. They are at the kind of the physical level. They are very different. They have very different physical characteristics. Cameras are passive. LIDARs and radars are active. Use different wavelengths. That means they complement each other very nicely. And together, combined, they can be used to build a much safer and much more capable system. To me, it's more of a question, why the heck would you handicap yourself and not use one or more of those sensing modalities when they undoubtedly just make your system more capable and safer? Now, what might make sense for one product or one business might not make sense for another one. So if you're talking about driver assist technologies, you make certain design decisions and you make certain trade-offs and make different ones. If you are building a driver that you deploy in fully driverless vehicles. And LIDARs specifically, when this question comes up, typically the criticisms that I hear are the counterpoints that cost and aesthetics. And I don't find either of those, honestly, very compelling. So on the cost side, there's nothing fundamentally prohibitive about the cost of LIDARs. Radars used to be very expensive before people made certain advances in technology and started to manufacture them at massive scale and deploy them in vehicles. Similar with LIDARs. And this is where the LIDARs that we have in our cars, especially the fifth generation, we've been able to make some pretty qualitative discontinuous jumps in terms of the fundamental technology that allow us to manufacture those things at very significant scale and add a fraction of the cost of both our previous generation as well as a fraction of the cost of what might be available on the market off the shelf right now. And that improvement will continue. So I think cost is not a real issue. Second one is aesthetics. I don't think that's a real issue either. You can make LIDAR sexy again. I think you're exactly right. I think it is sexy. Honestly, I think form, whole function. I was actually, somebody brought this up to me. All forms of LIDAR, even the ones that are big, you can make look beautiful. There's no sense in which you can't integrate it into design. There's all kinds of awesome designs. I don't think small and humble is beautiful. It could be brutalism or it could be harsh corner. Like I said, hot rods. I don't necessarily like, oh man, I'm going to start so much controversy with this. I don't like Porsches. Okay. The Porsche 911, everyone says, oh, it's the most beautiful. No, it's like a baby car. It doesn't make any sense. But everyone, it's beauty is in the eye of the beholder. You're already looking at me like, what's this kid talking about? I'm happy to talk about- You're digging your own hole. The form and function and my take on the beauty of the hardware that we put on our vehicles, I will not comment on your Porsche monologue. Okay. All right. But aesthetics, fine. But there's an underlying philosophical question behind the LIDAR question. It's like how much of the problem can be solved with computer vision, with machine learning? So I think without sort of disagreements and so on, it's nice to put it on the spectrum because Waymo's doing a lot of machine learning as well. It's interesting to think how much of driving, if we look at five years, 10 years, 50 years down the road, can be learned in almost more and more and more end to end way. If we look at what Tesla is doing as a machine learning problem, they're doing a multitask learning thing where it's just they break up driving into a bunch of learning tasks and they have one single neural network and they're just collecting a huge amount of data that's training that. I've recently hung out with George Hotz. I don't know if you know George. I love him so much. He's just an entertaining human being. We were off mic talking about Hunter S. Thompson. He's the Hunter S. Thompson of what time I was driving. Okay. So I didn't realize this with Comma AI, but they're like really trying to do end to end. They're looking at the machine learning problem. They're really not doing multitask learning, but it's computing the drivable area as a machine learning task and hoping that down the line, this level two system, this driver assistance will eventually lead to allowing you to have a fully autonomous vehicle. Okay. There's an underlying deep philosophical question there, technical question of how much of driving can be learned. So LiDAR is an effective tool today for actually deploying a successful machine learning for actually deploying a successful service in Phoenix, right? That's safe, that's reliable, et cetera, et cetera. But the question, and I'm not saying you can't do machine learning on LiDAR, but the question is that like how much of driving can be learned eventually? Can we do fully autonomous that's learned? Yeah. Learning is all over the place and plays a key role in every part of our system. As you said, I would decouple the sensing modalities from the ML and the software parts of it. LiDAR, radar, cameras, it's all machine learning, all of the object detection classification, of course, that's what these modern deep nets and countenance are very good at. You feed them raw data, massive amounts of raw data, and that's actually what our custom build LiDARs and radars are really good at. And radars, they don't just give you point estimates of objects in space, they give you raw physical observations. And then you take all of that raw information, whether it's colors of the pixels, whether it's LiDARs returns and some auxiliary information, it's not just distance, right? And angle and distance is much richer information that you get from those returns, plus really rich information from the radars. You fuse it all together and you feed it into those massive ML models that then lead to the best results in terms of object detection, classification, state estimation. So there's a side to interrupt, but there is a fusion. I mean, that's something that people didn't do for a very long time, which is like at the sensor fusion level, I guess, like early on fusing the information together, whether so that the sensory information that the vehicle receives from the different modalities, or even from different cameras is combined before it is fed into the machine learning models. Yeah. So I think this is one of the trends. You're seeing more of that. You mentioned end-to-end. There's different interpretations of end-to-end. There is kind of the purest interpretation of I'm going to have one model that goes from raw sensor data to like steering torque and gas breaks. That's too much. I don't think that's the right way to do it. There's more smaller versions of end-to-end where you're kind of doing more end-to-end learning or core training or depropagation of kind of signals back and forth across the different stages of your system. There's really good ways it gets into some fairly complex design choices where on one hand you want modularity and decomposability, the composability of your system. But on the other hand, you don't want to create interfaces that are too narrow or too brittle, too engineered where you're giving up on the generality of the solution, or you're unable to properly propagate signal, reach signal forward and losses and back so you can optimize the whole system jointly. I would decouple, and I guess what you're seeing in terms of the fusion of the sensing data from different modalities as well as kind of fusion in the temporal level going more from frame by frame where you would have one net that would do frame by frame detection and camera, and then something that does frame by frame and lighter and then radar. Then you fuse it in a weaker engineered way later. The field over the last decade has been evolving in more kind of joint fusion, more end-to-end models that are solving some of these tasks jointly. There's tremendous power in that. That's the progression that our stack has been on as well. Now, so I would decouple the sensing and how that information is used from the role of ML and the entire stack. There's trade-offs in modularity and how do you inject inductive bias into your system. There's tremendous power in being able to do that. There's no part of our system that does not heavily leverage data-driven development or state of the art ML, but there's mapping, there's a simulator, or there's perception, object level perception, whether it's semantic understanding, prediction, decision-making, and so forth and so on. Of course, object detection and classification, like you're finding pedestrians and cars and cyclists and cones and signs and vegetation and being very good at estimating detection, classification, and state estimation, there's just stable stakes. That's step zero of this whole stack. You can be incredibly good at that, whether you use cameras or lighters or radar, but that's just stable stakes. That's just step zero. Beyond that, you get into the really interesting challenges of semantic understanding, the perception level. You get into scene-level reasoning. You get into very deep problems that have to do with prediction and joint prediction and interaction, so the interaction between all of the actors in the environment, pedestrians, cyclists, other cars, and you get into decision making. How do you build a lot of systems? We leverage ML very heavily in all of these components. I do believe that the best results you achieve by using a hybrid approach and having different types of ML, having different models with different degrees of inductive bias that you can have, and combining model-based approaches and some rule-based, physics-based systems. One example I can give you is traffic lights. There's a problem of the detection of traffic light state, and obviously, that's a great problem for computer vision. Confidants, that's their bread and butter. That's how you build that. But then the interpretation of a traffic light, you don't need to learn that. You don't need to build some complex ML model that infers with some precision and recall that red means stop. It's a very clear engineered signal with very clear semantics. You want to induce that bias, how you induce that bias and that whether it's a constraint or a cost function in your stack. But it is important to be able to inject that clear semantic signal into your stack. That's what we do. That's when you apply it to yourself when you are making decisions whether you want to stop for a red light or not. But if you think about how other people treat traffic lights, we're back to the ML version of that. You know they're supposed to stop for a red light, but that doesn't mean they will. Then you're back in the very heavy ML domain where you're picking up on very subtle cues that have to do with the behavior of objects, pedestrians, cyclists, cars, and the whole entire configuration of the scene that allow you to make accurate predictions on whether they will in fact stop or run a red light. So it sounds like already for Waymo, machine learning is a huge part of the stack. So it's a huge part of like not just so obviously the first level zero or whatever you said which is like just object detection of things that you know with know that machine learning can do but also starting to do prediction behavior and so on to model the what other or the other parties in the scene entities in the scene are going to do. So machine learning is more and more playing a role in that as well. Of course, absolutely. I think we've been going back to the earliest days like DARPA ERB, DARPA Grand Challenge. Our team was leveraging machine learning. I was like pre-Imagenet and it was a very different type of ML. And I think actually it was before my time but the Stanford team during the Grand Challenge had a very interesting machine learned system that would use lighter and camera when driving in the desert and it would kind of extend the range of free space reasoning. We get a clear signal from lighter and then it had a model that said, hey, like this stuff in camera kind of sort of looks like this stuff in lighter and I know this stuff that I've seen in lighter, I'm very confident this free space. So let me extend that free space zone into the camera range that would allow the vehicle to drive faster. And then we've been building on top of that and kind of staying and pushing the state of the art in ML in all kinds of different ML over the years. And in fact, we've been building on top of that in all kinds of different ML over the years. And in fact, from the earliest days, I think 2010 is probably the year where Google, maybe 2011 probably, got pretty heavily involved in machine learning, kind of deep nuts. And at that time it was probably the only company that was very heavily investing in kind of state of the art ML and self-driving cars. And they go hand in hand. And we've been on that journey ever since. We're doing pushing a lot of these areas in terms of research at Waymo and we collaborate very heavily with the researchers in alphabet and all kinds of ML, supervised ML, unsupervised ML, published some interesting research papers in the space, especially recently. It's just super active learning as well. Yeah, super, super active. Of course, there's kind of the more mature stuff like, you know, convnets for object detection, but there's some really interesting, really active work that's happening in kind of more, you know, in bigger models and, you know, models that have more structure to them, you know, not just large bitmaps and reason about temporal sequences. And some of the interesting breakthroughs that we've seen in language models, right? You know, transformers, you know, GPT-3 inference. There's some really interesting applications of some of the core breakthroughs to those problems of behavior prediction as well as decision-making and planning, right? You can think about it, kind of the behavior, how, you know, the path, the trajectories, the how people drive. They have kind of a share a lot of the fundamental structure, you know, this problem. There's, you know, sequential nature. There's a lot of structure in this representation. There is a strong locality, kind of like in sentences, you know, words that follow each other. They're strongly connected, but there are also kind of larger context that doesn't have that locality. And you also see that in driving, right? What's happening in the scene as a whole has very strong implications on, you know, the kind of the next step in that sequence where whether you're predicting what other people are going to do, whether you're making your own decisions or whether in the simulator you're building generative models of, you know, humans walking, cyclists riding, the cars driving. That's all really fascinating. Like how it's fascinating to think that transforming models and all the breakthroughs in language and NLP that might be applicable to like driving at the higher level, at the behavioral level, that's kind of fascinating. Let me ask about pesky little creatures called pedestrians and cyclists. They seem, so humans are a problem. If we can get rid of them, I would. But unfortunately, they're also a source of joy and love and beauty. So let's keep them around. They're also our customers. Oh, for your perspective. Yes, yes, for sure. They're so some money. Very good. But I don't even know where I was going. Oh, yes, pedestrians and cyclists. I, you know, they're a fascinating injection into the system of uncertainty of, of like a game theoretic dance of what to do. And also they have perceptions of their own and they can tweet about your product. So you don't want to run them over from that perspective. I mean, I don't know. I'm joking a lot, but I think in seriousness, like, you know, pedestrians are complicated, computer vision problem, a complicated behavioral problem. Is there something interesting you could say about what you've learned from a machine learning perspective, from also an autonomous vehicle and a product perspective about just interacting with the humans in this world? Yeah, just, you know, just state on record, we care deeply about the safety of pedestrians, you know, even the ones that don't have Twitter accounts. Thank you. All right. But, you know, not me, but yes, I'm glad, I'm glad somebody does. Okay. But, you know, in all seriousness, safety of vulnerable road users, pedestrians or cyclists is one of our highest priorities. We do a tremendous amount of testing and validation and put a very significant emphasis on the capabilities of our systems that have to do with safety around those unprotected vulnerable road users. Cars, as we discussed earlier in Phoenix, we have completely empty cars, completely driverless cars driving in this very large area. And some people use them to go to school, so they will drive through school zones. Right. So kids are kind of the very special class of those vulnerable user road users, right? You want to be super, super safe and super, super cautious around those. We take it very, very, very seriously. And, you know, what does it take to be good at it? You know, an incredible amount of performance across your whole stack, you know, starts with hardware. And again, you want to use all sun signal modalities available to you. Imagine driving on a residential road at night and kind of making a turn and you don't have, you know, headlights covering some part of the space and like, you know, a kid might run out. And, you know, lighters are amazing at that. They see just as well in complete darkness as they do during the day. Right. So just again, it gives you that extra, you know, margin in terms of, you know, capability and performance and safety and quality. And in fact, we oftentimes in these kinds of situations, we have our system detect something in some cases even earlier than our trained operators in the car might do, especially in conditions like very dark nights. So starts with sensing, then, you know, perception has to be incredibly good. And you have to be very, very good at kind of detecting pedestrians in all kinds of situations and all kinds of environments, including, you know, people in weird poses, people kind of running around and, you know, being partially occluded. So, you know, that's stuff number one. Then you have to have in very high accuracy and very low latency in terms of your reactions to, you know, what these actors might do. Right. And we've put a tremendous amount of engineering and tremendous amount of validation into make sure our system performs properly. And oftentimes it does require a very strong reaction to do the safe thing. And we actually see a lot of cases like that. That's the long tail of really rare, you know, really crazy events that contribute to the safety around pedestrians. One example that comes to mind that we actually happened in Phoenix, where we were driving along and I think it was a 45 mile per hour road. So, you know, pretty high speed traffic and there was a sidewalk next to it. And there was a cyclist on the sidewalk. And as we were in the right lane, right next to the side, so it was a multi-lane road. So, as we got close to the cyclist on the sidewalk, it was a woman. She tripped and fell. Just, you know, fell right into the path of our vehicle. Right. And our, you know, car, you know, this was actually with a test driver, our test drivers did exactly the right thing. They kind of reacted and came to stop. It requires both very strong steering and, you know, strong application of the brake. And then we simulated what our system would have done in that situation. And it did exactly the same thing. And that speaks to all of those components of really good state estimation and tracking. And you can imagine, you know, a person on a bike and they're falling over and they're doing that right in front of you. Right. So you have to be really like, things are changing. The appearance of that whole thing is changing, right? And a person goes one way, they're falling on the road, they're, you know, being flat on the ground in front of you. You know, the bike goes flying the other direction. Like the two objects that used to be one are now, you know, are splitting apart and the car has to, like, detect all of that. Like milliseconds matter. And it doesn't, you know, it's not good enough to just brake. You have to like steer and brake and there's traffic around you. So, like, it all has to come together. And it was really great to see in this case and other cases like that, that we're actually seeing in the wild, that our system is, you know, performing exactly the way that we would have liked and is able to, you know, avoid collisions like this. It's such an exciting space for robotics. Like in that split second to make decisions of life and death. I don't know. The stakes are high in a sense, but it's also beautiful that for somebody who loves artificial intelligence, the possibility that an AI system might be able to save a human life, that's kind of exciting as a problem. Like to wake up. It's terrifying probably for an engineer to wake up and to think about, but it's also exciting because it's like it's in your hands. Let me try to ask a question that's often brought up about autonomous vehicles. And it might be fun to see if you have anything interesting to say, which is about the trolley problem. So the trolley problem is an interesting philosophical construct of that highlights, and there's many others like it, of the difficult ethical decisions that we humans have before us in this complicated world. So the specifically is the choice between if you are forced to choose to kill a group X of people versus a group Y of people, like one person, if you didn't, if you did nothing, you would kill one person, but if you would kill five people and if you decide to swerve out of the way, you would only kill one person. Do you do nothing or you choose to do something and you can construct all kinds of sort of ethical experiments of this kind that I think at least on a positive note, inspire you to think about like introspect, what are the, the physics of our morality? And there's usually not good answers there. I think people love it because it's just an exciting thing to think about. I think people who build autonomous vehicles usually roll their eyes because this is not this one as constructed, this like literally never comes up in reality. You never have to choose between killing one or like one of two groups of people, but I wonder if you can speak to, is there some, something interesting to you as an engineer of autonomous vehicles that's within the trolley problem or maybe more generally, are there, are there, are there, are there, are there, are there, are there, are there difficult ethical decisions that you find that algorithm must make? On the specific version of the trolley problem, which one would you do if you're driving? The question itself is a profound question because we humans ourselves cannot answer it. And that's the very point. I would kill both. Yeah. Humans, I think you're exactly right. And that, you know, humans are not particularly good. I think they kind of phrased it like what would a computer do, but humans, you know, are not very good. And actually oftentimes I think that, you know, freezing and kind of not doing anything because like you've taken a few extra milliseconds to just process and then you end up like doing the worst possible outcomes. Right. So I do think that as you've pointed out, it can be a bit of a distraction and it can be a bit of a kind of a red herring. I think it's an interesting discussion in the realm of philosophy, right. But in terms of what, you know, how that affects the actual engineering and deployment of self-driving vehicles, it's not how you go about building a system, right. We've talked about how you engineer a system, how you go about evaluating the different components and the, you know, the safety of the entire thing. How do you kind of inject the, you know, various model-based safety-based arguments and you're like, yes, you reason it parts of the system, you know, you reason about the probability of a collision, the severity of that collision, right. And that is incorporated and there's, you know, you have to properly reason about the uncertainty that flows through the system. Right. So, you know, those, you know, factors definitely play a role in how the cars then behave, but they tend to be more of like the immersion behavior. And what you see like you're absolutely right that these clear theoretical problems that they, you know, you don't occur to that in system and really kind of being back to our previous discussion of like what, you know, which one do you choose? Well, you know, oftentimes, like you made a mistake earlier, like you shouldn't be in that situation in the first place, right. And in reality, the system comes up. If you build a very good, safe and capable driver, you have enough, you know, clues in the environment that you drive defensively. So you don't put yourself in that situation, right. And again, you know, it has, you know, this, if you go back to that analogy of, you know, precision and recall, like, okay, you can make a very hard trade off of, but like neither answer is really good. But what instead you focus on is kind of moving the whole curve up. And then you focus on building the right capability and the right defensive driving so that, you know, you don't put yourself in a situation like this. I don't know if you have a good answer for this, but people love it when I asked this question about books. Are there books in your life that you've enjoyed? Philosophical, fiction, technical that had a big impact on you as an engineer or as a human being, you know, everything from science fiction to a favorite textbook. Is there three books that stand out that you can think of? Three books. So I would, you know, that impacted me, I would say, this one is, you probably know it well, but not generally well known. I think in the US or kind of internationally, The Master and Margarita. It's one of actually my favorite books. It is, you know, by Russian, it's a novel by Russian author, Mikhail Bulgakov. And it's just, it's a great book. It's one of those books that you can like reread your entire life and it's very accessible. You can read it as a kid. And like, you know, the plot is interesting. It's, you know, the devil, you know, visiting the Soviet Union. But like you read it, reread it at different stages of your life and you enjoy it for different, very different reasons. And you keep finding like deeper and deeper meaning and, you know, kind of affected, you know, definitely had an imprint on me, mostly from the probably kind of the cultural stylistic aspect. Like it makes you one of those books that, you know, is good and makes you think, but also has like this really, you know, silly, quirky, dark sense of humor. It captures the Russian soul more than perhaps many other books. On that like slight note, just out of curiosity, one of the saddest things is I've read that book in English. Did you by chance read it in English or in Russian? In Russian, only in Russian. And actually that is a question I had, kind of posed to myself every once in a while. I wonder how well it translates, if it translates at all. And there's the language aspect of it. And then there's the cultural aspect. So I, and actually I'm not sure if either of those would work well in English. Now I forget their names, but so when the COVID lifts a little bit, I'm traveling to Paris for several reasons. One is just, I've never been to Paris. I want to go to Paris, but there's a, the most famous translators of Dostoevsky, Tolstoy, of most of Russian literature, live there. There's a couple, they're famous, a man and a woman. And I'm going to sort of have a series of conversations with them. And in preparation for that, I'm starting to read Dostoevsky in Russian. So I'm really embarrassed to say that I've read, everything I've read in Russian literature of like serious depth has been in English. Even though I can also read, I mean, obviously in Russian, but for some reason it seemed in the optimization of life, it seemed the improper decision to do it, to read in Russian. Like, you know, like I don't need to opt, I need to think in English, not in Russian, but now I'm changing my mind on that. And so the question of how well it translates, it's a really fundamental one, like yet even with Dostoevsky. So from what I understand, Dostoevsky translates easier. Others don't as much. Obviously the poetry doesn't translate as well. I'm also the music, big fan of Vladimir Vysotsky. He doesn't obviously translate well. People have tried. But mastermind, I don't know, I don't know about that one. I just know it in English, you know, it's fun as hell in English. So, but it's a curious question and I want to study rigorously from both the machine learning aspect and also because I want to do a couple of interviews in Russia that I'm still unsure of how to properly conduct an interview across a language barrier. It's a fascinating question that ultimately communicates to an American audience. There's a few Russian people that I think are truly special human beings. And I feel like I sometimes encounter this with some incredible scientists and maybe you encounter this as well at some point in your life that it feels like because of the language barrier, their ideas are lost to history. It's a sad thing. I think about like Chinese scientists or even authors that like, that we don't in English speaking world, don't get to appreciate some like the depth of the culture because it's lost in translation. And I feel like I would love to show that to the world. Like I'm just some idiot, but because I have this like at least some semblance of skill in speaking Russian, I feel like, and I know how to record stuff on a video camera. I feel like I want to catch like Grigori Perlman, who's a mathematician. I'm not sure if you're familiar with him. I want to talk to him. Like he's a fascinating mind and to bring him to a wider audience in English speaking will be fascinating, but that requires to be rigorous about this question of how well Bulgakov translates. I mean, I know it's a silly concept, but it's a fundamental one because how do you translate? And that's the thing that Google Translate is also facing as a more machine learning problem, but I wonder as a more bigger problem for AI, how do we capture the magic that's there in the language? I think that's really interesting, really challenging problem. If you do read it, Master and Margarita in English, sorry, in Russian, I'd be curious to get your opinion. And I think part of it is language, but part of it is just centuries of culture that the cultures are different. So it's hard to connect that. Okay, so that was my first one. You had two more. The second one, I would probably pick the science fiction by the Strogatsky brothers. It's up there with Isaac Asimov and Ray Bradbury and company. The Strogatsky brothers kind of appealed more to me. I think it made more of an impression on me growing up. I, you know, I am a person. I apologize if I'm showing my complete ignorance. I'm so weak on sci-fi. What are they, right? Oh, roadside picnic, hard to be a god, beetle in an anthill, Monday starts on Saturday. Like it's not just science fiction. It's also like has very interesting, you know, interpersonal and societal questions. And some of the language is just completely hilarious. That's the one. That's interesting. Monday starts on Saturday. So I need to read. Okay. Oh, boy. You put that in the category of science fiction. That one is, I mean, this was more of a silly, you know, humorous work. I mean, there is kind of, but it's profound too, right? Science fiction, right? It's about, you know, this research institute and like it has deep parallels to like serious research, but the setting of course is that they're working on, you know, magic, right? And there's a lot of stuff. So I, and that's their style, right? And, you know, other books are very different, right? You know, hard to be a god, right? It's about kind of this higher society being injected into this primitive world and how they operate there. Some of the very deep ethical questions there, right? And like they've got this full spectrum, some is more about kind of more adventure style, but like I enjoy all of their books. There's probably a couple, actually one, I think that they consider it their most important work. I think it's the snail on a hill. I'm not exactly sure how it translates. I tried reading a couple of times. I still don't get it, but everything else I fully enjoyed. And like for one of my birthdays as a kid, I got like their entire collection, like occupied a giant shelf in my room. And then like over the holidays, I just like, you know, my parents couldn't drag me out of the room and I read the whole thing cover to cover and it, I really enjoyed it. And that's one more. For the third one, maybe a little bit darker, but you know, comes to mind is Orwell's 1984. And I, you know, I asked what made an impression on me and the books that people should read. That one, I think falls in the category of both. Definitely it's one of those books that you read and you just kind of put it down and you stare in space for a while. That kind of work. I think there's lessons there. People should not ignore. And you know, nowadays with like everything that's happening in the world, I can't help it, but you know, I have my mind jump to some parallels with what Orwell described. And like this whole concept of double think and ignoring logic and holding completely contradictory opinions in your mind and not have that not bother you and stick into the party line at all costs. Like, you know, there's something there. If anything, 2020 has taught me and I'm a huge fan of Animal Farm, which is a kind of friendly as a friend of 1984 by Orwell. It's kind of another thought experiment of how our society may go in directions that we wouldn't like it to go. But if, if anything that's been kind of heartbreaking to an optimist about 2020 is that that society is kind of fragile. Like we have this, this is a special little experiment we have going on and not, it's not unbreakable. Like we should be careful to like preserve whatever special thing we have going on. I mean, I think 1984 and these books, Brave New World, they, they're helpful in thinking like stuff can go wrong in non-obvious ways. And it's like, it's up to us to make sure it's up to us to preserve it. And it's like, it's a responsibility. It's been weighing heavy on me because like, for some reason, like more than my mom follows me on Twitter. And I feel like I have, I have like now somehow responsibility to, to this world. And it dawned on me that like me and millions of others are like the little ants that maintain this little colony. Right. So we have a responsibility not to be, I don't know what the right analogy is, but I'll put a flame thrower to the place. We want to not do that. And there's interesting, complicated ways of doing that. As 1984 shows, it could be through bureaucracy, it could be through incompetence. It could be through misinformation. It could be through division and toxicity. I'm a huge believer in like that love will be the, somehow the solution. So love and robots, robots. Yeah. I think you're exactly right. Unfortunately, I think it's less of a flame thrower type of an answer. I think it's more of a, in many cases can be more of a slow boil and that that's the danger. Let me ask. It's a fun thing to make a world-class roboticist, engineer and leader uncomfortable with the ridiculous question about life. What is the meaning of life, Dmitri, from a robotics and a human perspective? You only have a couple of minutes or one minute to answer. So I don't know if that makes it more difficult or easier. Actually, yeah. You know, they're very tempted to quote one of the stories, stories by Isaac Asimov actually, actually titled appropriately titled the last question, short story where the plot is that humans build this supercomputer, this AI intelligence and once it gets powerful enough, they pose this question to it. How can the entropy in the universe be reduced? So the computer replies as of yet insufficient information to give a meaningful answer. And then thousands of years go by and they keep posing the same question and computer gets more and more powerful and keeps giving the same answer as of yet insufficient information to give a meaningful answer or something along those lines. And then it keeps happening and happening. You fast forward millions of years into the future and billions of years and at some point it's just the only entity in the universe. It's absorbed all humanity and all knowledge in the universe and keeps posing the same question to itself. And finally, it gets to the point where it is able to answer that question. But of course, at that point, yeah, there's, you know, the heat death of the universe has occurred and that's the only entity and there's nobody else to provide that answer to. So the only thing it can do is to answer it by demonstration. So it like, you know, recreates the big bang, right? And resets the clock, right? But I can try to give kind of a different version of the answer, you know, maybe not on the behalf of all humanity. I think that that might be a little presumptuous for me to speak about the meaning of life on the behalf of all humans, but at least, you know, personally, it changes, right? I think if you think about kind of what gives you and your life meaning and purpose and kind of what drives you, it seems to change over time, right? And the lifespan of your existence. When you just enter this world, right, it's all about kind of new experiences, right? You get like new smells, new sounds, new emotions, right? And like, that's what's driving you, right? You're experiencing new amazing things, right? And that's magical, right? That's pretty awesome, right? That gives you kind of a meaning. Then you get a little bit older, you start more intentionally learning about things. Right? I guess actually, before you start intentionally learning, probably fun. Fun is a thing that gives you kind of meaning and purpose and purpose and the thing you optimize for, right? And like fun is good. Then you get, you know, start learning. And I guess that this joy of comprehension and discovery is another thing that, you know, gives you meaning and purpose and drives you, right? Then, you know, you learn enough stuff and it, you want to give some of it back, right? And so impact and contributions back to technology or society, people, you know, local or more globally, becomes a new thing that drives a lot of kind of your behavior and something that gives you purpose and that you derive positive feedback from, right? Then you go and so on and so forth. You go through various stages of life. If you have kids, like that definitely changes your perspective on things. You know, I have three that definitely flips some bits in your head in terms of kind of what you care about and what you optimize for and you know, what matters, what doesn't matter, right? So, you know, and so on and so forth, right? And I, it seems to me that, you know, it's all of those things. And as you kind of go through life, you know, you want these to be additive, right? New experiences, fun, learning, information, learning impact, like you want, you want to, you know, be accumulating. I don't want to, you know, stop having fun or experiencing new things. And I think it's important that, you know, it just kind of becomes additive as opposed to a replacement or subtraction. But, you know, those few as far as I got, but, you know, ask me in a few years, I might have one or two more to add to the list. And before you know it, time is up just like it is for this conversation. But hopefully it was a fun ride. It was a huge honor to meet you. As you know, I've been a fan of yours and a fan of Google self-driving car and Waymo for a long time. I can't wait. I mean, it's one of the most exciting. If we look back in the 21st century, I truly believe it will be one of the most exciting things we descendants of apes have created on this earth. So I'm a huge fan and I can't wait to see what you do next. Thanks so much for talking to me. Thanks. Thanks for having me. And it's a also a huge fan. It doesn't work. Honestly, and I really enjoyed it. Thank you. Thank you for listening and hope to see you next time.
 The following is a conversation with Dmitry Dolgov, the CTO of Waymo,
 which is an autonomous driving company that started as Google
 self-driving car project in 2009 and became Waymo in 2016.
 Dmitry was there all along.
 Waymo is currently leading in the fully autonomous vehicle space
 and that they actually have an at-scale deployment of publicly accessible
 autonomous vehicles, driving passengers around with no safety driver,
 with nobody in the driver's seat.
 This, to me, is an incredible accomplishment of engineering
 on one of the most difficult and exciting artificial intelligence
 challenges of the 21st century.
 Quick mention of a sponsor, followed by some thoughts related to the episode.
 Thank you to Trial Labs, a company that helps businesses apply
 machine learning to solve real world problems.
 Blinkist, an app I use for reading through summaries of books,
 BetterHelp, online therapy with a licensed professional, and CashApp,
 the app I use to send money to friends.
 Please check out the sponsors in the description to get a discount
 at the support this podcast.
 As a side note, let me say that autonomous and semi-autonomous driving
 was the focus of my work at MIT
 and is a problem space that I find fascinating and full of open questions
 from both a robotics and a human psychology perspective.
 There's quite a bit that I could say here about my experiences
 in academia on this topic that revealed to me, let's say,
 the less admirable sides of human beings.
 But I choose to focus on the positive, on solutions.
 On brilliant engineers like Dimitri and the team at Waymo,
 who work tirelessly to innovate and to build amazing technology
 that will define our future.
 Because of Dimitri and others like him, I'm excited for this future.
 And who knows, perhaps I too will help contribute something of value to it.
 If you enjoy this thing, subscribe on YouTube, review it with five stars
 on a podcast, follow on Spotify, support on Patreon,
 or connect with me on Twitter at Lex Friedman.
 And now here's my conversation with Dimitri Dolgov.
 When did you first fall in love with robotics or even computer science more in general?
 Computer science first at a fairly young age.
 Robotics happened much later.
 I think my first interesting introduction to computers
 was in the late 80s, when we got our first computer.
 I think it was an IBM, I think IBM AT.
 Do you remember those things that had like a turbo button in the front?
 Turbo button, yeah.
 The radio would press it and make the thing go faster.
 Did they already have floppy disks?
 Yeah, yeah, yeah.
 The 5.4 inch ones.
 I think there was a bigger inch.
 So it went something, then five inches, then three inches.
 Yeah, I think that was the five.
 Maybe before that was the giant plates, then I didn't get that.
 But it was definitely not the first computer.
 But it was definitely not the three inch ones.
 Anyway, so that computer, I spent the first few months just playing video games,
 as you would expect.
 I got bored of that.
 So I started messing around and trying to figure out how to make the thing do other stuff.
 Got into exploring programming.
 And a couple of years later, I got to a point where I actually wrote a game,
 a little game, and a game developer, a Japanese game developer actually offered to buy it for me
 for a few hundred bucks, but for a kid in Russia.
 That's a big deal.
 That's a big deal, yeah.
 I did not take the deal.
 Wow, integrity.
 Yeah, I instead...
 Stability.
 Yes, that was not the most acute financial move that I made in my life.
 Looking back at it now, I instead put it...
 Well, I had a reason.
 I put it online.
 It was...
 What did you call it back in the day?
 It was a freeware thing, right?
 It was not open source, but you could upload the binaries, you would put the game online.
 And the idea was that people like it and then they contribute and they send you a little donations, right?
 So I did my quick math of like, of course, thousands and millions of people are going to play my game,
 send me a couple of bucks apiece, should definitely do that.
 As I said, not the best financial move.
 You're already playing business models at that age.
 Remember what language it was?
 What program it was?
 Pascal.
 What?
 Pascal.
 And they had a graphical component, so it's not text-based.
 Yeah, it was like, I think 320 by 200, whatever it was.
 I think that kind of the earlier version was VGA resolution, right?
 And I actually think the reason why this company wanted to buy it is not like the fancy graphics or the implementation.
 It was maybe the idea of the actual game.
 The idea of the game.
 Okay.
 Well, one of the things, it's so funny, I used to play this game called Golden Axe.
 And the simplicity of the graphics and something about the simplicity of the music, like it still haunts me.
 I don't know if that's a childhood thing.
 I don't know if that's the same thing for Call of Duty these days for young kids.
 But I still think that the simple, when the games are simple,
 that simple purity makes for, like allows your imagination to take over and thereby creating a more magical experience.
 Like now with better and better graphics, it feels like your imagination doesn't get to create worlds, which is kind of interesting.
 It could be just an old man on a porch, like waving at kids these days that have no respect.
 But I still think that graphics almost get in the way of the experience.
 I don't know.
 Flippy Bird.
 Yeah.
 Well, I don't know if the imagination gets closed.
 I don't. Yeah.
 But that that's more about games that like that's more like Tetris World, where they optimally masterfully like create a fun, short term dopamine experience
 versus I'm more referring to like role playing games where there's like a story.
 You can live in it for months or years like there's an Elder Scrolls series, which is probably my favorite set of games.
 That was a magical experience.
 And the graphics are terrible.
 The characters were all randomly generated, but I don't know.
 It pulls you in.
 There's a story.
 It's like an interactive version of an Elder Scrolls Tolkien world.
 And you get to live in it.
 I don't know.
 I miss it.
 It's one of the things that suck about being an adult is there's no you have to live in the real world as opposed to the Elder Scrolls world.
 You know, whatever brings you joy, right?
 Minecraft, right?
 Minecraft is a great example.
 You create like it's not the fancy graphics, but it's the creation of your own worlds.
 Yeah, that one is crazy.
 You know, one of the pitches for being a parent that people tell me is that you can like use the excuse of parenting to to go back into the video game world.
 And like that's like, you know, father, son, father, daughter time.
 But really, you just get to play video games with your kids.
 So anyway, at that time, did you have any ridiculous ambitious dreams of where as a creator you might go as an engineer?
 Did you what did you think of yourself as as an engineer, as a tinker?
 Or did you want to be like an astronaut or something like that?
 You know, I'm tempted to make something up about, you know, robots, engineering or, you know, mysteries of the universe.
 But that's not the actual memory that pops into my mind when you when you ask me about childhood dreams.
 So I actually share the real thing.
 When I was maybe four or five years old, I as well do, I thought about what I wanted to do when I grow up.
 And I had this dream of being a traffic control cop.
 They don't have those today's I think.
 But back in the 80s and in Russia, you probably are familiar with that.
 Like they had these police officers that would stand in the middle of intersection all day and they would have their like striped back and white batons that they would use to control the flow of traffic.
 And for whatever reason, I was strangely infatuated with this whole process.
 And like that, that was my dream.
 That's what I wanted to do when I grew up.
 And, you know, my parents, both physics profs, by the way, I think were a little concerned with that level of ambition coming from their child.
 Yeah, that age.
 Well, it's an interesting I don't know if you can relate, but I very much love that idea.
 I have OCD nature that I think lends itself very close to the engineering mindset,
 which is you want to kind of optimize, you know, solve a problem by creating an automated solution, like a set of rules, that set of rules that you follow and thereby make it ultra efficient.
 I don't know if that's it was of that nature.
 I certainly have that. There's like fact like SimCity and factory building games, all those kinds of things kind of speak to that engineering mindset.
 Or did you just like the uniform?
 I think it was more of the latter. I think it was the uniform and the striped baton that made cars go in the right directions.
 But I guess, you know, it is I did end up, I guess, you know, working on the transportation industry one way or another.
 No uniform, no.
 But that's right. Maybe it was my deep inner infatuation with the traffic control batons that led to this career.
 OK, what when did you when was the leap from programming to robotics?
 That happened later. That was after grad school after and actually the most seldom cars was, I think, my first real hands on introduction to robotics.
 But I never really had that much hands on experience in school and training.
 I worked on applied math and physics.
 Then in college, I did more abstract computer science and it was after grad school that I really got involved in robotics.
 It was actually self-driving cars. And, you know, that was a big flip.
 What what grad school?
 So I went to grad school in Michigan and then I did a postdoc at Stanford, which is that was the postdoc where I got to play with self-driving cars.
 Yeah, so we'll return there. Let's go back to to Moscow.
 So, you know, for episode 100, I talked to my dad and also I grew up with my dad, I guess.
 So I had to put up with them for many years.
 And he he went to the Fistia or M.I.P.T.
 It's weird to say in English because I've heard all of this in Russian, Moscow Institute of Physics and Technology.
 And to me, that was like I met some super interesting as a child, I met some super interesting characters.
 It felt to me like the greatest university in the world, the most elite university in the world.
 And just the people that I met that came out of there were like.
 Not only brilliant, but also special humans, it seems like that place really tested the soul,
 both in terms of technically and spiritually.
 So that could be just the romanticization of that place.
 I'm not sure. But so maybe you can speak to it.
 But is it correct to say that you spent some time at Fistia?
 Yeah, that's right. Six years.
 I got my bachelor's and master's and physics and math there.
 And it's interesting because my my dad and actually both my parents went there.
 And I think all the stories that I heard, like just like you, Alex, growing up,
 about the place and how interesting and special and magical it was,
 I think that was a significant, maybe the main reason I wanted to go there for college.
 Enough so that I actually went back to Russia from the US.
 I graduated high school in the US and you went back there.
 I went back there. Yeah. Wow.
 Exactly the reaction most of my peers in college had.
 But perhaps a little bit stronger that point me out as this crazy kid.
 Were your parents supportive of that?
 Yeah. Yeah. I came to your previous question.
 They they supported me in letting me pursue my passions and the things that I was interested in.
 That's a bold move. Wow. What was it like there?
 It was interesting. Definitely fairly hardcore on the fundamentals of math and physics and
 lots of good memories from from those times.
 So, OK. So Stanford, how did you get into autonomous vehicles?
 I had the great fortune and great honor to join Stanford's DARPA Urban Challenge team in 2006.
 This was a third in the sequence of the DARPA challenges.
 There were two grand challenges prior to that.
 And then in 2007, they held the DARPA Urban Challenge.
 So I was doing my postdoc. I joined the team and worked on motion planning
 for that competition.
 So, OK, so for people who might not know, I know from from a certain point,
 autonomous vehicles is a funny world in a certain circle of people.
 Everybody knows everything.
 And in a certain circle, nobody knows anything in terms of general public.
 So it's interesting. It's a good question what to talk about.
 But I do think that the Urban Challenge is worth revisiting.
 It's a fun little challenge.
 One that first like sparked so much.
 So many incredible minds to focus on one of the hardest problems of our time
 in artificial intelligence.
 So that's it's a success from a perspective of a single little challenge.
 But can you talk about like, what did the challenge involve?
 So were there pedestrians? Were there other cars?
 What was the goal? Who was on the team?
 How long did it take?
 Any fun, fun sort of specs? Sure, sure, sure.
 So the way the challenge was constructed, just a little bit of background.
 As I mentioned, this was the third competition in that series.
 The first two were the Grand Challenge, called the Grand Challenge.
 The goal there was to just drive in a completely static environment.
 You had to drive in a desert.
 That was very successful.
 So then DARPA followed with what they called the Urban Challenge,
 where the goal was to build vehicles that could operate in more dynamic environments
 and share them with other vehicles.
 There were no pedestrians there.
 But what DARPA did is they took over an abandoned Air Force base.
 And it was kind of like a little fake city that they built out there.
 And they had a bunch of robots, cars that were autonomous in there,
 all at the same time, mixed in with other vehicles driven by professional drivers.
 And each car had a mission.
 And so there's a crude map that they received at the beginning.
 And they had a mission, you know, go here and then there and over here.
 And they kind of all were sharing this environment.
 At the same time, they had to interact with each other.
 They had to interact with the human drivers.
 So it's this very first, very rudimentary version of a self-driving car
 that could operate in an environment shared with other dynamic actors
 that, as you said, really, in many ways, kick-started this whole industry.
 OK, so who was on the team?
 And how did you do? I forget.
 I came in second.
 Perhaps that was my contribution to the team.
 I think the Stanford team came in first in the DARPA Challenge.
 But then I joined the team.
 You were the one with the bug in the code.
 I mean, do you have sort of memories of some particularly challenging things or...
 You know, one of the cool things is not, you know, this isn't a product.
 This isn't the thing that, you know,
 you have a little bit more freedom to experiment so you can take risks.
 And there's... So you can make mistakes.
 Is there interesting mistakes?
 Is there interesting challenges that stand out to you as something like taught you
 a good technical lesson or a good philosophical lesson from that time?
 Yeah, definitely a very memorable time.
 Not really a challenge, but one of the most vivid memories that I have from that time.
 And I think that was actually one of the days that really got me hooked on this whole field
 was the first time I got to run my software on the car.
 And I was working on a part of our planning algorithm that had to navigate in parking lots.
 So it was something that called free space motion planning.
 So the very first version of that was, you know, we tried on the car.
 I was on Stanford's campus in the middle of the night and had this little course
 constructed with cones in the middle of a parking lot.
 So we were there like 3 a.m.
 By the time we got the code to, you know, compile and turn over.
 And, you know, it drove.
 I could actually did something quite reasonable.
 And I was, of course, very buggy at the time and had all kinds of problems.
 But it was pretty darn magical.
 I remember going back and, you know, later trying to fall asleep and just being unable
 to fall asleep for the rest of the night.
 Just my mind was blown.
 And that's what I've been doing ever since for more than a decade.
 In terms of challenges and interesting memories, like on the day of the competition,
 it was pretty nerve wrecking.
 I remember standing there with Mike Montemariello, who was the software lead and wrote most of the code.
 I think I did one little part of the planner.
 Mike, you know, incredibly did pretty much the rest of it with a bunch of other incredible people.
 But I remember standing on the day of the competition, you know, watching the car with Mike.
 And cars are completely empty, right?
 They're all there lined up in the beginning of the race.
 And then, you know, DARPA sends them on their mission one by one.
 So they leave and like you just they had these sirens.
 They all had their different silence, right?
 Each siren had its own personality, if you will.
 So, you know, off they go and you don't see them.
 And then every once in a while, they come a little bit closer to where the audience is.
 And you can kind of hear the sound of your car.
 And it seems to be moving along.
 So that gives you hope.
 And then, you know, it goes away and you can't hear it for too long.
 You start getting anxious, right?
 So it's a little bit like, you know, sending your kids to college and like, you know, kind of you invested in them.
 You hope you build it properly, but it's still anxiety inducing.
 So that was an incredibly fun few days in terms of, you know, bugs.
 As you mentioned, you know, one that was my bug that caused us the loss of the first place.
 It's still a debate that occasionally have with people on the CMU team.
 CMU came first, I should mention that.
 CMU haven't heard of them, but yeah, it's a small school.
 It's really a glitch that, you know, they haven't succeeded something robotics related.
 Very scenic, though.
 Most people go there for the scenery.
 Yeah, it's a beautiful campus.
 I apologize.
 Unlike Stanford.
 So for people, yeah, that's true.
 Unlike Stanford.
 For people who don't know, CMU is one of the great robotics and sort of artificial intelligence universities in the world.
 CMU Carnegie Ballon University.
 OK, sorry. Go ahead.
 Good, good PC.
 So in the part that I contributed to, which was navigating parking lots and the way that part of the mission work is you in a parking lot,
 you would get from DARPA an outline of the map.
 You basically get this giant polygon that define the perimeter of the parking lot.
 And there would be an entrance and maybe multiple entrances or exits to it.
 And then you would get a goal within that open space, XY heading where the car had to park.
 It had no information about the obstacles that the car might encounter there.
 So it had to navigate a completely free space from the entrance to the parking lot into that parking space.
 And then once it parked there, it had to exit the parking lot.
 While, of course, in counting and reasoning about all the obstacles that it encounters in real time.
 So our interpretation, or at least my interpretation of the rules, was that you had to reverse out of the parking spot.
 And that's what our cars did, even if there's no obstacle in front.
 That's not what CMU's car did.
 And it just kind of drove right through.
 So there's still a debate.
 And of course, you stop and then reverse out and go out the different way that cost you some time.
 So there's still a debate whether it was my poor implementation that cost us extra time or whether it was CMU violating an important rule of the competition.
 And I have my own opinion here in terms of other bugs.
 And I have to apologize to Mike Montemariella for sharing this on air.
 But it is actually one of the more memorable ones.
 And it's something that's kind of become a bit of a metaphor in the industry since then, I think, at least in some circles.
 It's called the victory lap.
 And our cars did that.
 So in one of the missions in the Urban Challenge and one of the courses, there was this big oval right by the start and finish of the race.
 So the ARPA had a lot of the missions would finish in that same location.
 And it was pretty cool because you could see the cars come by, kind of finish that part leg of the trip, that leg of the mission, and then go on and finish the rest of it.
 And other vehicles would come, hit their waypoint and exit the oval and off they would go.
 Our car, on the other hand, which hit the checkpoint, and then it would do an extra lap around the oval and only then leave and go on its merry way.
 So over the course of the full day, it accumulated some extra time.
 And the problem was that we had a bug where it wouldn't start reasoning about the next waypoint and plan a route to get to that next point until it hit a previous one.
 And in that particular case, by the time you hit that one, it was too late for us to consider the next one and kind of make a lane change.
 So every time we would do like an extra lap.
 So that's the Stanford Victory Lap.
 I feel like there's something philosophically profound in there somehow.
 But I mean, ultimately, everybody is a winner in that kind of competition.
 And it led to sort of famously to the creation of Google Self-Driving Car Project and now Waymo.
 So can we give an overview of how is Waymo born?
 How is the Google Self-Driving Car Project born?
 What is the mission?
 What is the hope?
 What is it is the engineering kind of set of milestones that it seeks to accomplish?
 There's a lot of questions in there.
 Yeah.
 I don't know.
 Kind of the DARPA Urban Challenge and the previous DARPA Grand Challenges kind of led, I think, to a very large degree to that next step.
 And then Larry and Sergey, Larry Page and Sergey Bren, Google founders, I saw that competition and believed in the technology.
 The Google Self-Driving Car Project was born.
 You know, at that time, and we started in 2009, it was a pretty small group of us, about a dozen people who came together to work on this project at Google.
 At that time, we saw an incredible early result in the DARPA Urban Challenge.
 I think we're all incredibly excited about where we got to.
 And we believed in the future of the technology, but we still had a very rudimentary understanding of the problem space.
 So the first goal of this project in 2009 was to really better understand what we're up against.
 And with that goal in mind, when we started the project, we created a few milestones for ourselves that maximized learnings.
 Well, the two milestones were, one was to drive 100,000 miles in autonomous mode, which was at that time orders of magnitude that more than anybody has ever done.
 And the second milestone was to drive 10 routes.
 Each one was 100 miles long.
 They were specifically chosen to become extra spicy, extra complicated, and sample the full complexity of that domain.
 And you had to drive each one from beginning to end with no intervention, no human intervention.
 So you would get to the beginning of the course, you would press the button that engaged in autonomy, and you had to go for 100 miles, beginning to end with no interventions.
 And it sampled, again, the full complexity of driving conditions.
 Some were on freeways.
 We had one route that went all through all the freeways and all the bridges in the Bay Area.
 We had some that went around Lake Tahoe and kind of mountainous roads.
 We had some that drove through dense urban environments like in downtown Palo Alto and through San Francisco.
 So it was incredibly interesting to work on.
 It took us just under two years, about a year and a half or a bit more, to finish both of these milestones.
 And in that process, it was an incredible amount of fun, probably the most fun I had in my professional career.
 And you're just learning so much.
 The goal here is to learn a prototype.
 You're not yet starting to build a production system.
 So you're just working 24-7 and hacking things together.
 And you also don't know how hard this is.
 I mean, that's the point.
 So that's an ambitious, if I put myself in that mindset, even still, that's a really ambitious set of goals.
 Like just those two, picking 10 different difficult, spicy challenges and then having zero interventions.
 So like not saying gradually, we're going to like, you know, over a period of 10 years,
 we're going to have a bunch of roots and gradually reduce the number of interventions.
 That literally says, as soon as possible, we want to have zero and on hard roads.
 So to me, if I was facing that, it's unclear whether that takes two years or whether that takes 20 years.
 It took us under two.
 I guess that speaks to a really big difference between doing something once and having a prototype
 where you are going after learning about the problem versus how you go about engineering a product
 that where you look at, you properly do evaluation, you look at metrics, you drive down,
 and you're confident that you can do that.
 And I guess that's why it took a dozen people 16 months or a little bit more than that back in 2009 and 2010
 with the technology of more than a decade ago, that amount of time to achieve that milestone of 10 routes,
 100 miles each and no interventions.
 And it took us a little bit longer to get to a full driverless product that customers use.
 That's another really important moment.
 Is there some memories of technical lessons or just one?
 What did you learn about the problem of driving from that experience?
 I mean, we can now talk about what you learned from modern day Waymo,
 but I feel like you may have learned some profound things in those early days, even more so,
 because it feels like what Waymo is now is to trying to know how to do scale,
 how to make sure you create a product, how to make sure it's safe, those things, which is all fascinating challenges.
 But like you were facing the more fundamental philosophical problem of driving in those early days.
 Like, what the hell is driving as an autonomous?
 Or maybe I'm again romanticizing it, but is there some valuable lessons you picked up over there at those two years?
 A ton. The most important one is probably that we believe that it's doable.
 And we've gotten far enough into the problem that we had, I think, only a glimpse of the true complexity of the domain.
 It's a little bit like climbing a mountain where you kind of see the next peak and you think that's kind of the summit.
 But then you get to that and you kind of see that this is just the start of the journey.
 But we've tried, we've sampled enough of the problem space and we've made enough rapid success,
 even with technology of 2009, 2010, that it gave us confidence to then pursue this as a real product.
 So, OK, so the next step, you mentioned the milestones that you had in those two years.
 What are the next milestones that then led to the creation of Waymo and beyond?
 It was a really interesting journey.
 And Waymo came a little bit later than we completed those milestones in 2010.
 That was the pivot when we decided to focus on actually building a product using this technology.
 The initial couple of years after that, we were focused on a freeway,
 what you would call a driver assist, maybe an L3 driver assist program.
 Then around 2013, we've learned enough about the space and have thought more deeply about the product that we wanted to build that we pivoted.
 We pivoted towards this vision of building a driver and deploying it fully driverless vehicles without a person.
 And that's the path that we've been on since then.
 And it was exactly the right decision for us.
 So there was a moment where you also considered like, what is the right trajectory here?
 What is the right role of automation in the task of driving?
 There was still it wasn't from the early days, obviously, you want to go fully autonomous.
 From the early days, it was not.
 I think it was in around 2013, maybe that we've that became very clear and we made the pivot and also became very clear.
 And that it's either the way you go building a driver assist system is fundamentally different from how you go building a fully driverless vehicle.
 So we've pivoted towards the latter.
 And that's what we've been working on ever since.
 And so that was around 2013.
 Then there's a sequence of really meaningful for us, really important defining milestones since then.
 And 2015, we had our first actually the world's first fully driverless trade on public roads.
 It was in a custom built vehicle that we had.
 I must have seen those.
 We called them the Firefly, that funny looking marshmallow looking thing.
 And we put a passenger, his name was Steve Mann, a great friend of our project from the early days.
 The man happens to be blind.
 So we put him in that vehicle.
 The car had no steering wheel, no pedals.
 It was an uncontrolled environment.
 No lead or chase cars, no police escorts.
 And we did that trip a few times in Austin, Texas.
 So that was a really big milestone.
 But that was in Austin.
 Yeah.
 OK.
 And we only, but at that time, it took a tremendous amount of engineering.
 It took a tremendous amount of validation to get to that point.
 But we only did it a few times.
 We only did that.
 It was a fixed route.
 It was not kind of a controlled environment, but it was a fixed route.
 And we only did it a few times.
 Then in 2016, end of 2016, beginning of 2017, is when we founded Waymo, the company.
 That's when we kind of that was the next phase of the project, where I wanted we
 believed in the commercial vision of this technology.
 And it made sense to create an independent entity within that alphabet umbrella
 to pursue this product at scale.
 Beyond that, in 2017, later in 2017, was another really huge step for us,
 this really big milestone where we started, I think it was October of 2017,
 where when we started regular driverless operations on public roads,
 that first day of operations, we drove in one day,
 in that first day, 100 miles in driverless fashion.
 And then the most important thing about that milestone
 was not that 100 miles in one day, but that it
 was the start of kind of regular ongoing driverless operations.
 Can we say driverless?
 It means no driver.
 That's exactly right.
 So on that first day, we actually had a mix.
 And we didn't want to be on YouTube and Twitter that same day.
 So in many of the rides, we had somebody in the driver's seat,
 but they could not disengage.
 The car could not disengage.
 But actually, on that first day, some of the miles
 were driven in just completely empty driver's seat.
 This is the key distinction that I think people don't realize.
 Oftentimes, when you talk about autonomous vehicles,
 there's often a driver in the seat that's ready to take over,
 what's called a safety driver.
 And then Waymo is really one of the only companies, at least,
 that I'm aware of, or at least as boldly and carefully and all
 that actually has cases, and now we'll talk about more and more,
 where there is literally no driver.
 So that's another interesting case of where the driver is not
 supposed to disengage.
 That's like a nice middle ground.
 They're still there, but they're not supposed to disengage.
 But really, there's the case when there's no.
 OK, there's something magical about there being
 nobody in the driver's seat.
 Just like to me, you mentioned the first time
 you wrote some code for free space navigation of the parking lot.
 That was like a magical moment.
 To me, just as an observer of robots,
 the first magical moment is seeing an autonomous vehicle
 turn, like make a left turn, apply sufficient torque
 to the steering wheel to where there's a lot of rotation.
 And for some reason, and there's nobody in the driver's seat,
 for some reason that communicates that here's a being with power
 that makes a decision.
 There's something about the steering wheel,
 because we perhaps romanticize the notion of the steering wheel.
 It's so essential to our conception, our 20th century
 conception of a car.
 And it turning the steering wheel with nobody in the driver's
 seat that to me, I think maybe to others, it's really powerful.
 Like this thing is in control.
 And then there's this leap of trust that you give.
 Like I'm going to put my life in the hands of this thing that's
 in control.
 So in that sense, when there's no driver in the driver's seat,
 that's a magical moment for robots.
 So I got a chance last year to take a ride in a Waymo vehicle.
 And that was the magical moment.
 There's like nobody in the driver's seat.
 It's like the little details.
 You would think it doesn't matter whether there's a driver or not.
 But if there's no driver and the steering wheel is turning on its
 own, I don't know.
 It's magical.
 It's absolutely magical.
 I've taken many of these rides in a completely empty car.
 No human in the car pulls up.
 You call it on your cell phone, it pulls up.
 You get in, it takes you on its way.
 There's no driver in the car.
 There's nobody in the car but you.
 That's something called fully driverless, our rider only mode
 of operation.
 Yeah, it is magical.
 It is transformative.
 This is what we hear from our riders.
 It really changes your experience.
 And that really is what unlocks the real potential of this technology.
 But coming back to our journey, that was 2017 when we started truly
 driverless operations.
 Then in 2018, we've launched our public commercial service that we
 called Waymo One in Phoenix.
 In 2019, we started offering truly driverless rider only rides to our
 early rider population of users.
 And then 2020 has also been a pretty interesting year.
 One of the first ones, less about technology but more about the
 maturing and the growth of Waymo as a company.
 We raised our first round of external financing this year.
 We were part of Alphabet so obviously we have access to significant
 resources but on the journey of Waymo maturing as a company, it made
 sense for us to partially go externally in this round.
 So we raised about $3.2 billion from that round.
 We've also started putting our fifth generation of our driver, our
 hardware that is on the new vehicle but it's also a qualitatively
 different set of self-driving hardware that is now on the JLR
 PACE so that was a very important step for us.
 The hardware specs, fifth generation, I think it would be fun to maybe,
 I apologize if I'm interrupting but maybe talk about maybe the
 generations with a focus on what we're talking about in the fifth
 generation in terms of hardware specs, like what's on this car.
 Sure.
 So we separated out the actual car that we are driving from the
 self-driving hardware we put on it.
 Right now we have, so this is as I mentioned the fifth generation.
 We've gone through, we started building our own hardware many,
 many years ago and that Firefly vehicle also had the hardware suite
 that was mostly design, engineered and built in-house.
 Lighters are one of the more important components that we design
 and build from the ground up.
 So on the fifth generation of our drivers, of our self-driving
 hardware that we're switching to right now, we have as with previous
 generations in terms of sensing, we have lighters, cameras and radars
 and we have a pretty beefy computer that processes all that information
 and makes new decisions in real time on board the car.
 So in all of the, and it's really a qualitative jump forward in terms
 of the capabilities and the various parameters and specs of the hardware
 compared to what we had before and compared to what you can kind of
 get off the shelf in the market today.
 Meaning from fifth to fourth or from fifth to first?
 Definitely from first to fifth, but also from the fourth.
 That was the world's dumbest question.
 Definitely from fourth to fifth, as well as that last step is a big step forward.
 So everything's in-house, so like lighters built in-house
 and cameras are built in-house?
 It's different.
 We work with partners and there's some components that we get from our
 manufacturing and supply chain partners.
 What exactly is in-house is a bit different.
 We do a lot of custom design on all of our sensing models.
 There's lighters, radars, cameras.
 Exactly there's lighters are almost exclusively in-house
 and some of the technologies that we have, some of the fundamental
 technologies there are completely unique to Waymo.
 That is also largely true about radars and cameras.
 It's a little bit more of a mix in terms of what we do ourselves versus
 what we get from partners.
 Is there something super sexy about the computer that you can mention
 that's not top secret?
 Like for people who enjoy computers.
 There's a lot of machine learning involved, but there's a lot of just basic compute.
 You have to probably do a lot of signal processing on all the different sensors.
 You have to integrate everything.
 It has to be in real time.
 There's probably some kind of redundancy type of situation.
 Is there something interesting you can say about the computer for the people who
 love hardware?
 It does have all of the characteristics, all the properties that you just
 mentioned.
 Redundancy, very beefy compute for general processing as well as inference
 and ML models.
 It is some of the more sensitive stuff that I don't want to get into for IP
 reasons, but we've shared a little bit in terms of the specs of the sensors that
 we have on the car.
 We actually shared some videos of what our lighter see in the world.
 We have 29 cameras.
 We have five lighters.
 We have six radars on these vehicles.
 You can kind of get a feel for the amount of data that they're producing.
 That all has to be processed in real time to do perception, to do complex
 reasoning.
 That kind of gives you some idea of how beefy those computers are, but I don't
 want to get into specifics of exactly how we build them.
 Okay.
 Well, let me try some more questions that you can get into the specifics of,
 like GPU-wise.
 Is that something you can get into?
 I know that Google works with GPUs and so on.
 I mean, for machine learning folks, it's kind of interesting.
 Or is there no...
 How do I ask it?
 I've been talking to people in the government about UFOs and they don't
 answer any questions.
 So this is how I feel right now asking about GPUs.
 But is there something interesting that you could reveal or leave it up to our
 imagination, some of the compute?
 Is there any, I guess, is there any fun trickery?
 Like I talked to Chris Latner for a second time and he was a key person
 about GPUs.
 And there's a lot of fun stuff going on in Google in terms of hardware that
 optimizes for machine learning.
 Is there something you can reveal in terms of how much you mentioned
 customization, how much customization there is for hardware for machine
 learning purposes?
 I'm going to be like that government, you know, you've got guy versus body
 of foes.
 But I guess I will say that it's really compute.
 It's really important.
 We have very data hungry and compute hungry ML models of all over our stack.
 And this is where both being part of Alphabet as well as designing our own
 sensors and the entire hardware suite together, where on one hand, you get
 access to like really rich raw sensor data that you can pipe from your sensors
 into your compute platform and build like build the whole pipe from sensor,
 raw sensor data to the big compute as then have the massive compute to
 process all that data.
 And this is where we're finding that having a lot of control of that hardware
 part of the stack is really advantageous.
 One of the fascinating magical places to me, again, might not be able to speak
 to the details, but it is the other compute, which is like, you know, this
 we're just talking about a single car, but the, you know, the driving experience
 is a source of a lot of fascinating data and you have a huge amount of data
 coming in on the car and on the car and, you know, the infrastructure of storing
 some of that data to then train or to analyze or so on.
 That's a fascinating like piece of it that that I understand a single car.
 I don't understand how you pull it all together.
 In a nice way, is that something that you could speak to in terms of the
 challenges of, of seeing the network of cars and then bringing the data back
 and analyzing things that went like, like edge cases of driving, be able to
 learn on them, to improve the system, to, to see where things went wrong, where
 things went right and analyze all that kind of stuff.
 Is there something interesting there in the, from an engineering perspective?
 Oh, there's an incredible, incredible, incredible, incredible
 amount of really interesting work that's happening there, both in the, you know,
 the real time operation of the fleet of cars and the information that they
 exchange with each other in real time to make better decisions, as well as
 on the kind of the off board component where you have to deal with massive
 amounts of data for training your ML models, evaluating the ML models, for
 simulating the entire system and for evaluating your entire system.
 And this is where being part of Alphabet has once again been tremendously
 advantageous.
 I think we consume an incredible amount of compute for ML infrastructure.
 We build a lot of custom frameworks to get good at, on data mining, finding
 the interesting edge cases for training and for evaluation of the system, for
 both training and evaluating some components and your sub parts of the
 system in various ML models, as well as the evaluating the entire system and
 simulation.
 Okay, that first piece that you mentioned that cars communicating to
 each other, essentially, I mean, through perhaps through a centralized point,
 but what, that's fascinating too.
 How much does that help you?
 Like if you imagine, you know, right now the number of Waymo vehicles is
 whatever X, I don't know if you can talk to what that means.
 I don't know what that number is, but it's not in the hundreds of millions
 yet.
 And imagine if the whole world is Waymo vehicles, like that changes
 potentially the power of connectivity.
 Like the more cars you have, I guess, actually, if you look at Phoenix,
 because there's enough vehicles, there's enough, when there's like some level of
 density, you can start to probably do some really interesting stuff with the
 fact that cars can negotiate, can communicate with each other and thereby
 make decisions.
 Is there something interesting there that you can talk to about like, how does
 that help with the driving problem from as compared to just a single car solving
 the driving problem by itself?
 Yeah, it's a spectrum.
 I first say that, you know, it helps and it helps in various ways, but it's not
 required right now.
 The way we build our system, like each cars can operate independently, they can
 operate with no connectivity.
 So I think it is important that you have a fully autonomous, fully capable
 driver, computerized driver that each car has.
 Then, you know, they do share information and they share information in real time.
 It really, really helps.
 All right.
 So the way we do this today is, you know, whenever one car encounters something
 interesting in the world, whether it might be an accident or a new construction zone,
 that information immediately gets, you know, uploaded over the air and it's
 propagated to the rest of the fleet.
 So that's kind of how we think about maps as priors in terms of the knowledge of
 our drivers, of our fleet of drivers that is distributed across the fleet and it's
 updated in real time.
 So that's one use case you can imagine as the density of these vehicles go up that
 they can exchange more information in terms of what they're planning to do and
 start influencing how they interact with each other, as well as potentially sharing
 some observations to help with if you have enough density of these vehicles where one
 car might be seeing something that another is relevant to another car that is very
 dynamic.
 It's not part of kind of your updating your static prior of the map of the world, but
 it's more of a dynamic information that could be relevant to the decisions that
 another car is making in real time.
 So you can see them exchanging that information and you can build on that.
 But again, I see that as an advantage, but it's not a requirement.
 So what about the human in the loop?
 So when I got a chance to drive with a ride in a Waymo, you know, there's customer
 service, so like there is somebody that's able to dynamically like tune in and help
 you out.
 What role does the human play in that picture?
 That's a fascinating like, you know, the idea of teleoperation, be able to remotely
 control a vehicle.
 So here what we're talking about is like, like frictionless, like a human being able
 to in a frictionless way sort of help you out.
 I don't know if they're able to actually control the vehicle.
 Is that something you could talk to?
 Yes.
 Okay.
 To be clear, we don't do teleoperation.
 I'm going to believe in teleoperation for a reason.
 That's not what we have in our cars.
 We do, as you mentioned, have a version of customer support.
 We call it life health.
 In fact, we find it that it's very important for our rider experience, especially if
 it's your first trip.
 You've never been in a fully driverless rider only way more vehicle.
 You get in, there's nobody there.
 And so you can imagine having all kinds of questions in your head, like how this
 thing works.
 So we've put a lot of thought into kind of guiding our riders, our customers through
 that experience, especially for the first time, they get some information on the phone.
 If the fully driverless vehicle is used to service their trip, when you get into the
 car, we have an in-car screen and audio that kind of guides them and explains what to
 expect.
 They also have a button that they can push that will connect them to a real life human
 being that they can talk to about this whole process.
 So that's one aspect of it.
 I should mention that there is another function that humans provide to our cars, but it's
 not teleoperation.
 You can think of it a little bit more like fleet assistance, kind of like traffic control
 that you have where our cars, again, they're responsible on their own for making all of
 the driving decisions that don't require connectivity.
 Anything that is safety or latency critical is done purely autonomously by our onboard
 system.
 But there are situations where if connectivity is available and a car encounters a particularly
 challenging situation, you can imagine like a super hairy scene of an accident, the cars
 will do their best.
 They will recognize that it's an off-nominal situation.
 They will do their best to come up with the right interpretation and the best course of
 action in that scenario.
 But if connectivity is available, they can ask for confirmation from a human assistant
 to kind of confirm those actions and perhaps provide a little bit of kind of contextual
 information and guidance.
 So October 8th was when you're talking about the, was Waymo launched the public version
 of its fully driverless, that's the right term, I think, service in Phoenix.
 Is that October 8th?
 That's right.
 Yeah, it was the introduction of fully driverless, right, or only vehicles into our public Waymo
 One service.
 Okay.
 So that's amazing.
 So it's like anybody can get into a Waymo in Phoenix?
 That's right.
 So we previously had early people in our early rider program taking fully driverless rides
 in Phoenix.
 And just a little while ago, we opened on October 8th, we opened that mode of operation
 to the public.
 So I can download the app and go on a ride.
 There is a lot more demand right now for that service than we have capacity.
 So we're kind of managing that, but that's exactly the way you described it.
 Yeah, well, that's interesting.
 So there's more demand than you can handle.
 Like what has been the reception so far?
 Like what, I mean, okay, so, you know, this is a product, right?
 That's a whole nother discussion of like how compelling of a product it is.
 Great.
 But it's also like one of the most kind of transformational technologies of the 21st
 century.
 So it's also like a tourist attraction.
 Like it's fun to, you know, to be a part of it.
 So it'd be interesting to see like, what do people say?
 What do people, what have been the feedback so far?
 Still early days, but so far the feedback has been incredible, incredibly positive.
 They, you know, we asked them for feedback during the ride.
 We asked them for feedback after the ride as part of their trip.
 We asked them some questions.
 We asked them to rate the performance of our driver.
 Most by far, you know, most of our drivers give us five stars in our app.
 Which is absolutely great to see.
 And, you know, that's, and they're also giving us feedback on, you know, things we can improve.
 And, you know, that's, that's one of the main reasons we're doing this is Phoenix.
 And, you know, over the last couple of years and every day today, we are just learning a
 tremendous amount of new stuff from our users.
 There's no substitute for actually doing the real thing, actually having a fully driverless
 product out there in the field with, you know, users that are actually paying us money to
 get from point A to point B.
 So this is a legitimate, like there's a paid service.
 That's right.
 And the idea is you use the app to go from point A to point B.
 And then what, what are the A's, what are the, what's the freedom of the,
 of the starting and ending places?
 It's an area of geography where that service is enabled.
 It's a decent size of geography of territory.
 It's actually larger than, you know, the size of San Francisco.
 And, you know, within that you have full freedom of, you know, selecting where you want to go.
 You know, of course there's some, and you, on your app, you get a map, you tell the car
 where you want to be picked up, you know, and where you want the car to pull over and pick you up.
 And then you tell it where you want to be dropped off.
 All right.
 And of course there are some exclusions, right?
 You don't want to be, you know, you, where, in terms of where the car is allowed to pull over,
 right?
 So, you know, that you can't do, but besides that, it's amazing.
 It's not like a fixed, just would be very exciting.
 I guess, I don't know, maybe that's what's the question behind your question, but it's not a,
 you know, preset set of, uh,
 Yes, I guess.
 So within the geographic constraints with that, within that area, anywhere else, it can be,
 you can be picked up and dropped off anywhere.
 That's right.
 And, you know, people use them on like all kinds of trips they,
 we have, and we have an incredible spectrum of riders.
 I think the youngest actually have car seats, them and we have, you know,
 people taking their kids and rides.
 I think the youngest riders we had on cars are, you know, one or two years old,
 you know, and the full spectrum of use cases.
 People can take them to, you know, schools, uh, to, you know, go grocery stock shopping,
 to restaurants, to bars, you know, run errands, you know, go shopping, et cetera, et cetera.
 You can go to your office, right?
 Uh, like the full spectrum of use cases and, uh, people, you know, use them in their daily
 lives to get around.
 Uh, and we see all kinds of really interesting use cases and that, that, that's providing us
 incredibly valuable experience, uh, that we then, you know, use to improve our product.
 So as somebody who's been on, done a few long rants with Joe Rogan and others about
 the toxicity of the internet and the comments and the negativity in the comments,
 I'm fascinated by feedback.
 I, I believe that most people are good and kind and intelligent and can provide like,
 uh, even in disagreement, really fascinating ideas.
 So on a product side, it's fascinating to me, like, how do you get the richest possible
 user feedback?
 Like to improve what's, what are the channels that you use to measure?
 Cause like you're, you're no longer, that's one of the magical things about autonomous
 vehicles is it's not like it's frictionless interaction with the human.
 So like you don't get to, you know, it's just giving a ride.
 So like, how do you get feedback from people to, in order to improve?
 Uh, yeah, great question.
 Various mechanisms.
 So as part of the normal flow, we ask people for feedback.
 They, as the car is driving around, we have on the phone and in the car, and we have a
 touchscreen in the car, uh, you can actually click some buttons and provide, uh, real-time
 feedback on how the car is doing, uh, and how the car is handling a particular situation,
 you know, both positive and negative.
 Uh, so that's one channel.
 Uh, we have, as we discussed, customer support or life help where, you know, if a customer
 wants to, has a question, uh, or he has some sort of concern, they can talk to a person
 in real time.
 So that, that is another mechanism that gives us, uh, feedback.
 Uh, at the end of a trip, you know, we also ask them how things went.
 They give us, uh, comments and, you know, star rating and, you know, if it's, uh, we
 also, you know, ask them, uh, to, uh, uh, explain what, you know, went well and, you
 know, what could be improved.
 And, uh, we, we have, uh, our writers are providing, you know, very rich, uh, feedback
 there, uh, a lot, a large fraction is, uh, very passionate and very excited about this
 technology.
 So we get really good feedback.
 Uh, we also run, uh, UXR studies, right?
 You know, specific, uh, that are kind of more, you know, go more in depth and we will run
 both kind of, uh, lateral and longitudinal studies.
 Um, where we have deeper engagement, uh, with our customers, you know, we have our user
 experience research team tracking over time.
 That's things about legitude.
 No, that's cool.
 That's exactly right.
 And, you know, that's another really valuable, uh, feedback, uh, source of feedback.
 And we were just covering a tremendous amount, right?
 Uh, people go grocery shopping and they like want to load, you know, 20 bags of groceries
 in our cars and like that.
 That's one workflow that you maybe don't think about, uh, you know, getting just right when
 you're building the driverless product.
 I have people like, you know, who, uh, bike as part of their trip.
 So they, you know, bike somewhere, then they get on our cars, they take a path, part of
 their bike, they load into our vehicle.
 Then they go and that's, you know, how they, you know, where we want to pull over and how
 that, you know, uh, get in and get out, um, uh, process works, uh, provides us very, uh,
 useful feedback in terms of, you know, what makes a good, uh, pick up and drop off location.
 Uh, we get really valuable feedback.
 Yeah.
 And in fact, we had to, um, uh, do some really interesting work with, uh, high definition
 maps and, uh, thinking about walking directions.
 If you imagine you're in a store, right?
 In some giant space and then, you know, you want to be picked up somewhere.
 I could, if you just drop a pin at a current location, which is maybe in the middle of
 a shopping mall, like what's the best location for the car to come pick you up.
 And you can have simple heuristics where you're just going to take your, you know,
 you clean in distance, uh, and find the nearest, uh, spot where the car can pull over that's
 closest to you.
 But oftentimes that's not the most convenient one.
 You know, I have many anecdotes where that heuristic breaks in horrible ways.
 Uh, one example, uh, that I often mentioned is, uh, somebody wanted to be, you know, uh,
 dropped off, uh, and Phoenix, uh, and, you know, we, our car picked a location, uh, that
 was close, the closest to their, you know, where the pin was dropped on the map in terms
 of, you know, latitude and longitude.
 Uh, but it happened to be on the other side of a parking lot that had this row of cacti
 and the poor person had to like walk all around the parking lot to get to where they wanted
 to be in 110 degree heat.
 So that, you know, that was a buck.
 So then, you know, we took all, take all of these, um, all of that feedback from our users
 and, uh, incorporate it into our system and, uh, improve it.
 Yeah.
 I feel like that's like requires AGI to solve the problem of like when you're, which is a
 very common case when you're in a big space of some kind, like apartment building, it
 doesn't matter.
 It's not some large space.
 And then you call the, like the Waymo from there, right?
 It's like, so any, whatever it doesn't matter, ride share vehicle.
 And like, where's the pin supposed to drop?
 I feel like that's, I, you don't think, I think that requires AGI.
 I'm going to, okay, the alternative, which I think the Google search engine is taught
 is like, there's something really valuable about the perhaps slightly dumb answer, but
 a really powerful one, which is like what was done in the past by others, like what
 was the choice made by others?
 That seems to be like in terms of Google search, when you have like billions of searches
 that you could, you could see which, like when they recommend what you might possibly
 mean, they suggest based on not some machine learning thing, which they also do, but like
 on what was successful for others in the past and finding a thing that they were happy
 with, is that integrated at all the way more like what, what pickups worked for others?
 It is.
 I think you're exactly right.
 So there's a real, it's an interesting problem.
 Naive solutions have interesting failure modes.
 So there's definitely lots of things that can be done to improve and both learning
 from, you know, what works, what doesn't work in actual hail from, you know, getting
 richer data and getting more information about the environment and, you know, richer
 maps.
 But you're absolutely right that there's something, I think there's some properties of
 solutions that in terms of the effect that they have on users, so much, much, much better
 than others, right?
 And predictability and understandability is important.
 So you can have maybe something that is not quite as optimal, but is very natural and
 predictable to the user and kind of works the same way all the time.
 And that matters.
 That matters a lot for the user experience.
 But, you know, to get to the basics, the pretty fundamental property is that the car actually
 arrives where you told it to arrive.
 Like you can always, you know, change it, see it on the map and you can move it around
 if you don't like it.
 But like that property that the car actually shows up on pin is critical, which, you know,
 where compared to some of the human driven analogs, I think, you know, you can have more
 predictability.
 It's actually the fact, if I have a little bit of a detour here, I think the fact that
 it's your phone and the car, two computers talking to each other, can lead to some really
 interesting things we can do in terms of the user interfaces, both in terms of function,
 like the car actually shows up exactly where you told it you want it to be, but also some
 really interesting things on the user interface.
 Like as the car is driving, as you call it, and it's on the way to come and pick you up,
 and of course you get the position of the car and the route on the map, but and they
 actually follow that route, of course, but it can also share some really interesting
 information about what it's doing.
 So, you know, our cars, as they are coming to pick you up, if it's come, if a car is
 coming up to a stop sign, it will actually show you that like it's there sitting because
 it's at a stop sign or a traffic light will show you that it's got sitting at a red light.
 So, you know, like little things, right?
 But it's I find those little touch touches really interesting, really magical.
 And it's just, you know, little things like that that you can do to kind of delight your users.
 You know, this makes me think of there's some products that I just love.
 Like there's a there's a company called Rev Rev.com where I like for this podcast, for example,
 I can just drag and drop a video and then they do all the captioning.
 It's humans doing the captioning, but they connect.
 They automate everything of connecting you to the humans and they do the
 captioning and transcription.
 It's all effortless.
 And it like I remember when I first started using them, it was like,
 life's good.
 Like because it was so painful to figure that out earlier.
 The same thing with something called the iZotope RX.
 This company I use for cleaning up audio, like the sound cleanup they do.
 It's like drag and drop and it just cleans everything up very nicely.
 Another experience like that I had with Amazon one click purchase.
 First time.
 I mean, other places do that now, but just the effortlessness of purchasing,
 making it frictionless.
 It kind of communicates to me like I'm a fan of design.
 I'm a fan of products that you can just create a really pleasant experience.
 The simplicity of it, the elegance just makes you fall in love with it.
 So on the, do you think about this kind of stuff?
 I mean, it's exactly what we've been talking about.
 It's like the little details that somehow make you fall in love with the product.
 Is that we went from like urban challenge days where, where love was not part of
 the conversation probably.
 And to this point where there's a, where there's human beings and you want them
 to fall in love with the experience.
 Is that something you're trying to optimize for?
 Try to think about like, how do you, how do you create experience that people love?
 Absolutely.
 That's the vision is removing any friction or complexity from getting our users,
 our writers to where they want to go.
 And making that as simple as possible.
 And then, you know, beyond that, just transportation.
 Making, you know, things and goods get to their destination as seamlessly as possible.
 And you talked about a drag and drop experience where I kind of express your intent and then
 you know, it just magically happens.
 And for our writers, that's what we're trying to get to is you download an app and you click
 and car shows up.
 It's the same car.
 It's very predictable.
 It's a safe and high quality experience.
 And then it gets you in a very reliable, very convenient, frictionless way to where you
 want to be.
 And along the journey, I think we also want to do little things to delight our users.
 Like the ride sharing companies, because they don't control the experience, I think,
 they can't make people fall in love necessarily with the experience.
 Or maybe they haven't put in the effort, but I think if I were to speak to the ride sharing
 experience I currently have, it's just very convenient.
 But there's a lot of room for like falling in love with it.
 Like we can speak to sort of car companies.
 Car companies do this well.
 You can fall in love with a car, right?
 And be like a loyal car person, like whatever.
 Like I like badass hot rods.
 I guess 69 Corvette.
 And at this point, you know, you can't really...
 Cars are so...
 Owning a car is so 20th century, man.
 But is there something about the Waymo experience where you hope that people will fall in love
 with it?
 Because is that part of it?
 Or is it just about making a convenient ride, not ride sharing, I don't know what the right
 term is, but just a convenient A to B autonomous transport?
 Or like, do you want them to fall in love with Waymo?
 So maybe elaborate a little bit.
 I mean, almost like from a business perspective, I'm curious.
 Like how...
 Do you want to be in the background invisible or do you want to be like a source of joy
 that's in very much in the foreground?
 I want to provide the best, most enjoyable transportation solution.
 And that means building it, building our product and building our service in a way that people
 do use in a very seamless, frictionless way in their day-to-day lives.
 And I think that does mean, you know, in some way falling in love in that product, right?
 It just kind of becomes part of your routine.
 It comes down, in my mind, to safety, predictability of the experience, and privacy, I think,
 aspects of it, right?
 Our cars, you get the same car, you get very predictable behavior, and that is important,
 I think, if you're going to use it in your daily life.
 Privacy, and when you're in a car, you can do other things.
 You're spending a bunch, just another space where you're spending a significant part of
 your life, right?
 So not having to share it with other people who you don't want to share it with, I think,
 is a very nice property.
 Maybe you want to take a phone call or do something else in the vehicle.
 And, you know, safety on the quality of the driving, as well as the physical safety of
 not having to share that ride is important to a lot of people.
 What about the idea that when there's somebody, like a human driving, and they do a rolling
 stop on a stop sign, like sometimes, like, you know, you get a new bro lift or whatever,
 like human driver, and, you know, they can be a little bit aggressive as drivers.
 It feels like there's not all aggression is bad.
 Now, that may be a wrong, again, 20th century conception of driving.
 Maybe it's possible to create a driving experience.
 Like, if you're in the back busy doing something, maybe aggression is not a good thing.
 It's a very different kind of experience, perhaps.
 But it feels like in order to navigate this world, you need to, how do I phrase this?
 You need to kind of bend the rules a little bit, or at least like test the rules.
 I don't know what language politicians use to discuss this, but whatever language they use,
 you like flirt with the rules.
 I don't know.
 But like you sort of have a bit of an aggressive way of driving that asserts your presence in
 this world, thereby making other vehicles and people respect your presence, and thereby
 allowing you to sort of navigate through intersections in a timely fashion.
 I don't know if any of that made sense, but like how does that fit into the experience of
 driving autonomously?
 Is that-
 Makes a lot of sense.
 This is, you're hitting on a very important point of a number of behavioral components and,
 you know, parameters that make your driving feel assertive and natural, comfortable, predictable.
 Now, our cars will follow rules, right?
 They will do the safest thing possible in all situations, let me be clear on that.
 But if you think of really, really good drivers, just think about professional limo drivers,
 right?
 They will follow the rules.
 They're very, very smooth, and yet they're very efficient.
 But they're assertive.
 They're comfortable for the people in the vehicle.
 They're predictable for the other people outside the vehicle that they share the environment with.
 And that's the kind of driver that we want to build.
 And you just need to think if, you know, maybe there's a sport analogy there, right?
 You can do in very many sports, the true professionals are very efficient in their movements,
 right?
 They don't do like, you know, hectic flailing, right?
 They're, you know, smooth and precise, right?
 And they get the best results.
 So that's the kind of driver that we want to build.
 In terms of, you know, aggressiveness, yeah, you can like, you know, roll through the stop signs,
 you can do crazy lane changes.
 Typically, it doesn't get you to your destination faster.
 Typically not the safest or most predictable, most comfortable thing to do.
 And but there is a way to do both.
 And that's what we're doing.
 We're trying to build a driver that is safe, comfortable, smooth, and predictable.
 Yeah, that's a really interesting distinction.
 I think in the early days of autonomous vehicles, the vehicles felt cautious as opposed to efficient.
 And still probably.
 But when I rode in the Waymo, I mean, there was, it was quite assertive.
 It moved pretty quickly.
 Like, yeah, then he's one of the surprising feelings was that it actually, it went fast.
 And it didn't feel like awkwardly cautious than autonomous vehicles.
 Like, like, so I've also programmed autonomous vehicles and everything I've ever built was felt awkwardly either overly aggressive.
 Okay.
 Especially when it was my code or like awkwardly cautious is the way I would put it.
 And Waymo's vehicle felt like assertive and I think efficient is like the right terminology here.
 They wasn't.
 And I also like the professional limo driver because we often think like, you know, an Uber driver or a bus driver or taxi.
 This is the funny thing is people think they track taxi drivers are professionals.
 I mean, it's, it's like that.
 That's like saying I'm a professional walker just because I've been walking all my life.
 I think there's an art to it.
 Right.
 And if you take it seriously as an art form, then there's a certain way that mastery looks like.
 It's interesting to think about what is mastery look like in driving and perhaps what we associate with like aggressive driving.
 That's like a necessary fluff that efficiency you could.
 You can be, you can create a good driving experience within the rules.
 That's a year, the first person to tell me this, so it's kind of interesting.
 I need to think about this, but that's exactly what it felt like with Waymo.
 I kind of like the way it felt like.
 I think that's the way it felt like.
 I think it's kind of like an idea.
 I think it's like a, I'm not sure if I'm going to think about this, but that's exactly what it felt like with Waymo.
 I kind of had this intuition.
 Maybe it's the Russian thing.
 I don't know that you have to break the rules in life to get anywhere, but maybe, maybe it's possible that that's not the case in driving.
 I have to think about that, but it certainly felt that way on the streets of Phoenix when I was there in Waymo.
 It wasn't frustrating in that like, come on, move already kind of feeling. That wasn't there.
 Yeah, I mean, that's what we're going after.
 I don't think you have to pick one.
 I think truly good driving gives you both efficiency, assertiveness, but also comfort and predictability and safety.
 And that's what fundamental improvements in the core capabilities truly unlock.
 And you can kind of think of it as, you know, precision and recall trade off.
 You have certain capabilities of your model, and then it's very easy when you have some curve of precision and recall, you can move things around and can choose your operating point.
 And you're trading off precision versus recall, false positives versus false negatives.
 Right.
 But then, and you know, you can tune things on that curve and be kind of more cautious or more aggressive, but then aggressive is bad or, you know, cautious is bad.
 But true capabilities come from actually moving the whole curve up.
 And then you are kind of on a very different plane of those trade-offs.
 And that's what, you know, we're trying to do here is to move the whole curve up.
 Before I forget, let's talk about trucks a little bit.
 So I also got a chance to check out some of the Waymo trucks.
 I'm not sure if we want to go too much into that space, but it's a fascinating one.
 So maybe we can mention at least briefly, you know, Waymo is also now doing autonomous, autonomous, autonomous, autonomous, autonomous.
 Trucking and how different like philosophically and technically is that whole space of problems?
 It's one of our two big products and commercial applications of our driver, right?
 Right.
 Healing and deliveries.
 We have Waymo One and Waymo Via, moving people and moving goods.
 Trucking is an example of moving goods.
 We've been working on trucking since 2017.
 It is a very interesting space.
 And you're a question of how different is it?
 It has this really nice property that the first order challenges like the science, the hard engineering,
 whether it's hardware or onboard software or off-board software,
 all of the systems that you build for training your ML models for evaluating your entire system.
 Those fundamentals carry over the true challenges of driving, perception, semantic understanding,
 prediction, decision-making, planning, evaluation, the simulator, ML infrastructure.
 Those carry over the data and the application and the domains might be different,
 but the most difficult problems, all of that carries over between the domains.
 So that's very nice.
 So that's how we approach it.
 We're kind of building investing in the core, the technical core.
 And then there is specialization of that core technology to different product lines,
 to different commercial applications.
 So just to tease it apart a little bit, on trucks, starting with the hardware,
 the configuration of the sensors is different.
 They're different physically, geometrically, different vehicles.
 So for example, we have two of our main laser on the trucks on both sides so that we have
 the blind spots, whereas on the GLR I-PACE, we have one of it sitting at the very top,
 but the actual sensors are almost the same or largely the same.
 So all of the investment that over the years we've put into building our custom lighters,
 custom radars, putting the whole system together, that carries over very nicely.
 Then on the perception side, the fundamental challenges of seeing, understanding the world,
 whether it's object detection, classification, tracking, semantic understanding,
 all that carries over.
 Yes, there's some specialization when you're driving on freeways, range becomes more important.
 The domain is a little bit different, but again, the fundamentals carry over very, very nicely.
 Same, and I guess you get into prediction or decision making.
 The fundamentals of what it takes to predict what other people are going to do,
 to find the long tail, to improve your system in that long tail of behavior,
 prediction and response, that carries over, right?
 And so on and so on.
 So I mean, that's pretty exciting.
 By the way, does Waymovia include using the smaller vehicles for transportation of goods?
 That's an interesting distinction.
 So I would say there's three interesting modes of operation.
 So one is moving humans, one is moving goods, and one is like moving nothing, zero occupancy,
 meaning like you're going to the destination, your empty vehicle.
 The third is the less of it, that's the entirety of it, it's the less exciting from the commercial perspective.
 Well, I mean, in terms of like, if you think about what's inside a vehicle as it's moving,
 because it does, some significant fraction of the vehicle's movement has to be empty.
 I mean, it's kind of fascinating, maybe just on that small point, is there different
 control and like policies that are applied for zero occupancy vehicle?
 So a vehicle with nothing in it, or is it just move as if there is a person inside?
 What was with some subtle differences?
 As a first order approximation, there are no differences.
 And if you think about safety and comfort and quality of driving, only part of it has to do with
 the people or the goods inside of the vehicle.
 But you want to drive smoothly, as we discussed, not for the purely for the benefit of whatever
 you have inside the car, right?
 It's also for the benefit of the people outside kind of feeding, fitting naturally and predictably
 into that whole environment, right?
 So yes, there are some second order things you can do.
 It's going to change your route and optimize maybe kind of your fleet things at the fleet scale.
 And you would take into account whether some of your cars are actually serving a useful trip,
 whether with people or with goods, whereas other cars are driving completely empty to that next
 valuable trip that they're going to provide.
 But those are mostly second order effects.
 OK, cool.
 So Phoenix is an incredible place.
 And what you've announced in Phoenix is kind of amazing.
 But that's just like one city.
 How do you take over the world?
 I mean, I'm asking for a friend.
 One step at a time.
 Cartoon pinky in the brain.
 But gradually is a true answer.
 So I think the heart of your question is...
 Can you ask a better question than I asked?
 You're asking a great question.
 Answer that one.
 I'm just going to phrase it in the terms that I want to answer.
 That's exactly right.
 Brilliant.
 Where are we today and what happens next?
 And what does it take to go beyond Phoenix and what does it take to get this technology
 to more places and more people around the world?
 All right.
 So our next big area of focus is exactly that.
 Larger scale commercialization and scaling up.
 If I think about the main...
 And Phoenix gives us that platform and gives us that foundation upon which we can build.
 And there are few really challenging aspects of this whole problem that you have to pull
 together in order to build the technology that we need.
 To build the technology in order to deploy it into the field.
 To go from a driverless car to a fleet of cars that are providing a service.
 And then all the way to commercialization.
 And this is what we have in Phoenix.
 We've taken the technology from a proof point to an actual deployment.
 And we've taken our driver from one car to a fleet that can provide a service.
 Beyond that, if I think about what it will take to scale up and deploy in more places
 with more customers, I tend to think about three main dimensions, three main axes of scale.
 One is the core technology, the hardware and software, core capabilities of our driver.
 The second dimension is evaluation and deployment.
 And the third one is the product, commercial and operational excellence.
 So you can talk a bit about where we are along each one of those three dimensions,
 about where we are today and what will happen next.
 On the core technology, on the hardware and software together, comprised of driver,
 we obviously have that foundation that is providing fully driverless trips to our customers
 as we speak, in fact.
 And we've learned a tremendous amount from that.
 So now what we're doing is we are incorporating all those lessons into some pretty fundamental
 improvements in our core technology, both on the hardware side and on the software side,
 to build a more general, more robust solution that then will enable us to massively scale
 in the young Phoenix.
 So on the hardware side, all of those lessons are now incorporated into this fifth generation
 hardware platform that is being deployed right now.
 And that's the platform, the fourth generation, the thing that we have right now driving in Phoenix,
 it's good enough to operate fully driverless, night and day, at various speeds and various conditions.
 But the fifth generation is the platform upon which we want to go to massive scale.
 We've really made qualitative improvements in terms of the capability of the system,
 the simplicity of the architecture, the reliability of the redundancy.
 It is designed to be manufacturable at very large scale and provides the right unit economics.
 So that's the next big step for us on the hardware side.
 That's already there for scale, the version five.
 That's right.
 And is that coincidence or should we look into a conspiracy theory that it's the same version
 as the Pixel phone?
 Is that what's the hardware?
 We can neither confirm nor deny it looks.
 All right, cool.
 So sorry.
 So that's that axis.
 What else?
 So similarly, hardware is a very discreet jump, but similarly to how we're making that change
 from the fourth generation hardware to the fifth, we're making similar improvements on the software
 side to make it more robust and more general and allow us to quickly scale beyond Phoenix.
 So that's the first dimension of core technology.
 The second dimension is evaluation and deployment.
 Now, how do you measure your system?
 How do you evaluate it?
 How do you build the release and deployment process where, you know, with confidence,
 you can regularly release new versions of your driver into a fleet?
 How do you get good at it so that it is not, you know, a huge tax on your researchers and engineers?
 That, you know, so you can kind of how do you build all these processes, the frameworks,
 the simulation, the evaluation, the data science, the validation so that, you know,
 people can focus on improving the system and kind of the releases just go out the door and get
 deployed across the fleet.
 So we've gotten really good at that in Phoenix.
 That's been a tremendously difficult problem, but that's what we have in Phoenix right now.
 That gives us that foundation.
 And now we're working on kind of incorporating all the lessons that we've learned to make it
 more efficient to go to new places, you know, scale up and just kind of stamp things out.
 So that's that second dimension of evaluation and deployment.
 And the third dimension is product, commercial and operational excellence.
 And again, Phoenix there is providing an incredibly valuable platform.
 You know, that's why we're doing things end to end in Phoenix.
 We're learning, as we discussed a little earlier today, tremendous amount of really valuable
 lessons from our users getting really incredible feedback.
 And we'll continue to iterate on that and incorporate all those lessons into making
 our product, you know, even better and more convenient for our users.
 So you're converting this whole process of Phoenix in Phoenix into something that could
 be copied and pasted elsewhere.
 So like, perhaps you didn't think of it that way when you were doing the experimentation
 in Phoenix, but so how long did you basically, I mean, you can correct me, but you've been
 taking, I mean, it's still early days, but you've taken the full journey in Phoenix,
 right?
 As you were saying of like what it takes to basically automate.
 I mean, it's not the entirety of Phoenix, right?
 But I imagine it can encompass the entirety of Phoenix.
 That's some near term date, but that's not even perhaps important.
 Like as long as it's a large enough geographic area.
 So what, how copy pasteable is that process currently?
 And how, like, you know, like when you copy and paste in Google docs, I think, no, in
 or in word, you can like apply source formatting or apply destination formatting.
 So when you copy and paste the Phoenix into like, say Boston, how do you apply the
 destination formatting?
 Like how much of the core of the entire process of bringing an actual public
 transportation, autonomous transportation service to a city is there in Phoenix that
 you understand enough to copy and paste into Boston or wherever?
 So we're not quite there yet.
 We're not at a point where we're kind of massively copying pasting all over the place.
 But Phoenix, what we did in Phoenix, and we very intentionally have chosen Phoenix as
 our first full deployment area, you know, exactly for that reason to kind of tease the
 problem apart, look at each dimension, focus on the fundamentals of complexity and de-risking
 those dimensions, and then bringing the entire thing together to get all the way and force
 ourselves to learn all those hard lessons on this technology, hardware and software,
 on the evaluation deployment, on operating a service, operating a business using, actually
 serving our customers all the way so that we're fully informed about the most difficult,
 most important challenges to get us to that next step of massive copy and pasting, as
 you said.
 And that's what we're doing right now.
 We're incorporating all those things that we learned into that next system that then
 will allow us to kind of copy and paste all over the place and to massively scale to more
 users and more locations.
 And we just talked a little bit about what does that mean along those different dimensions.
 So on the hardware side, for example, again, it's that switch from the fourth to the fifth
 generation.
 And the fifth generation is designed to kind of have that property.
 Can you say what other cities you're thinking about?
 Like I'm thinking about, sorry, we're in San Francisco now.
 I thought I want to move to San Francisco, but I'm thinking about moving to Austin.
 I don't know why people are not being very nice about San Francisco currently.
 Maybe it's a small, maybe it's in vogue right now, but Austin seems, I visited there and
 it was, I was in a Walmart.
 It's funny these moments like turn your life.
 There's this very nice woman with kind eyes just like stopped and said, he looks so handsome
 in that tie honey to me.
 This has never happened to me in my life, but just the sweetness of this woman is something
 I've never experienced, certainly in the streets of Boston, but even in San Francisco where
 people wouldn't, that's just not how they spend their time.
 Or think, I don't know.
 There's a warmth to Austin that love.
 And since Waymo does have a little bit of a history there, is that a possibility?
 Is this your version of asking the question of like, you know, Demetri, I know you can't
 share your commercial and deployment roadmap, but I'm thinking about moving to San Francisco
 of Austin, like in a blink twice.
 If you think I should move to that's true.
 That's true.
 You got me.
 We've been testing in all over the place.
 I think we've been testing in more than 25 cities.
 We drive in San Francisco, we drive in Michigan for snow.
 We are doing significant amount of testing in the Bay Area, including San Francisco.
 Which is not like, because we're talking about the very different thing, which is like
 a full on large geographic area public service.
 You can't share.
 Okay.
 What about Moscow?
 When is that happening?
 Take on Yandex.
 I'm not paying attention to those folks.
 They're doing, you know, there's a lot of fun.
 I mean, maybe as a way of a question, you didn't speak to sort of like policy or like,
 is there tricky things with government and so on?
 Like, is there other friction that you've encountered except sort of technological friction
 of solving this very difficult problem?
 Is there other stuff that you have to overcome when deploying a public service in a city?
 That's interesting.
 It's very important.
 So we put significant effort in creating those partnerships and those relationships
 with governments at all levels, local governments, municipalities, state level, federal level.
 We've been engaged in very deep conversations from the earliest days of our projects.
 At all of these levels, whenever we go to test or operate in a new area,
 we always lead with a conversation with the local officials.
 But the result of that investment is that no, it's not challenges we have to overcome,
 but it is very important that we continue to have this conversation.
 Oh, yeah.
 I love politicians too.
 Okay.
 So Mr. Elon Musk said that LIDAR is a crutch.
 What are your thoughts?
 I wouldn't characterize it exactly that way.
 Okay. I know. I think LIDAR is very important.
 It is a key sensor that we use just like other modalities.
 As we discussed, our cars use cameras, LIDARs, and radars.
 They are all very important.
 They are at the kind of the physical level.
 They are very different.
 They have very different physical characteristics.
 Cameras are passive.
 LIDARs and radars are active.
 Use different wavelengths.
 That means they complement each other very nicely.
 And together, combined, they can be used to build a much safer and much more capable system.
 To me, it's more of a question, why the heck would you handicap yourself and not use one
 or more of those sensing modalities when they undoubtedly just make your system
 more capable and safer?
 Now, what might make sense for one product or one business might not make sense for another one.
 So if you're talking about driver assist technologies,
 you make certain design decisions and you make certain trade-offs and make different ones.
 If you are building a driver that you deploy in fully driverless vehicles.
 And LIDARs specifically, when this question comes up,
 typically the criticisms that I hear are the counterpoints that cost and aesthetics.
 And I don't find either of those, honestly, very compelling.
 So on the cost side, there's nothing fundamentally prohibitive about the cost of LIDARs.
 Radars used to be very expensive before people made certain advances in technology and started
 to manufacture them at massive scale and deploy them in vehicles.
 Similar with LIDARs.
 And this is where the LIDARs that we have in our cars, especially the fifth generation,
 we've been able to make some pretty qualitative discontinuous jumps in terms of the fundamental
 technology that allow us to manufacture those things at very significant scale and add a
 fraction of the cost of both our previous generation as well as a fraction of the cost of
 what might be available on the market off the shelf right now.
 And that improvement will continue.
 So I think cost is not a real issue.
 Second one is aesthetics.
 I don't think that's a real issue either.
 You can make LIDAR sexy again.
 I think you're exactly right.
 I think it is sexy.
 Honestly, I think form, whole function.
 I was actually, somebody brought this up to me.
 All forms of LIDAR, even the ones that are big, you can make look beautiful.
 There's no sense in which you can't integrate it into design.
 There's all kinds of awesome designs.
 I don't think small and humble is beautiful.
 It could be brutalism or it could be harsh corner.
 Like I said, hot rods.
 I don't necessarily like, oh man, I'm going to start so much controversy with this.
 I don't like Porsches.
 Okay.
 The Porsche 911, everyone says, oh, it's the most beautiful.
 No, it's like a baby car.
 It doesn't make any sense.
 But everyone, it's beauty is in the eye of the beholder.
 You're already looking at me like, what's this kid talking about?
 I'm happy to talk about-
 You're digging your own hole.
 The form and function and my take on the beauty of the hardware that we put on our vehicles,
 I will not comment on your Porsche monologue.
 Okay.
 All right.
 But aesthetics, fine.
 But there's an underlying philosophical question behind the LIDAR question.
 It's like how much of the problem can be solved with computer vision, with machine learning?
 So I think without sort of disagreements and so on, it's nice to put it on the spectrum
 because Waymo's doing a lot of machine learning as well.
 It's interesting to think how much of driving, if we look at five years, 10 years, 50 years
 down the road, can be learned in almost more and more and more end to end way.
 If we look at what Tesla is doing as a machine learning problem,
 they're doing a multitask learning thing where it's just they break up driving into a bunch of
 learning tasks and they have one single neural network and they're just collecting a huge
 amount of data that's training that.
 I've recently hung out with George Hotz.
 I don't know if you know George.
 I love him so much.
 He's just an entertaining human being.
 We were off mic talking about Hunter S. Thompson.
 He's the Hunter S. Thompson of what time I was driving.
 Okay.
 So I didn't realize this with Comma AI, but they're like really trying to do end to end.
 They're looking at the machine learning problem.
 They're really not doing multitask learning, but it's computing the drivable area as a
 machine learning task and hoping that down the line, this level two system,
 this driver assistance will eventually lead to allowing you to have a fully autonomous vehicle.
 Okay.
 There's an underlying deep philosophical question there, technical question
 of how much of driving can be learned.
 So LiDAR is an effective tool today for actually deploying a successful machine learning
 for actually deploying a successful service in Phoenix, right?
 That's safe, that's reliable, et cetera, et cetera.
 But the question, and I'm not saying you can't do machine learning on LiDAR,
 but the question is that like how much of driving can be learned eventually?
 Can we do fully autonomous that's learned?
 Yeah.
 Learning is all over the place and plays a key role in every part of our system.
 As you said, I would decouple the sensing modalities from the ML and the software parts of it.
 LiDAR, radar, cameras, it's all machine learning, all of the object detection classification,
 of course, that's what these modern deep nets and countenance are very good at.
 You feed them raw data, massive amounts of raw data, and that's actually what our custom
 build LiDARs and radars are really good at.
 And radars, they don't just give you point estimates of objects in space,
 they give you raw physical observations.
 And then you take all of that raw information, whether it's colors of the pixels,
 whether it's LiDARs returns and some auxiliary information, it's not just distance, right?
 And angle and distance is much richer information that you get from those returns,
 plus really rich information from the radars.
 You fuse it all together and you feed it into those massive ML models that then lead to the
 best results in terms of object detection, classification, state estimation.
 So there's a side to interrupt, but there is a fusion.
 I mean, that's something that people didn't do for a very long time,
 which is like at the sensor fusion level, I guess, like early on fusing the information together,
 whether so that the sensory information that the vehicle receives from the different modalities,
 or even from different cameras is combined before it is fed into the machine learning models.
 Yeah. So I think this is one of the trends. You're seeing more of that. You mentioned end-to-end.
 There's different interpretations of end-to-end. There is kind of the
 purest interpretation of I'm going to have one model that goes from raw sensor data to
 like steering torque and gas breaks. That's too much. I don't think that's the right way to do it.
 There's more smaller versions of end-to-end where you're kind of doing more end-to-end
 learning or core training or depropagation of kind of signals back and forth across the different
 stages of your system. There's really good ways it gets into some fairly complex design choices
 where on one hand you want modularity and decomposability, the composability of your system.
 But on the other hand, you don't want to create interfaces that are too narrow or too brittle,
 too engineered where you're giving up on the generality of the solution, or you're unable
 to properly propagate signal, reach signal forward and losses and back so you can optimize the whole
 system jointly. I would decouple, and I guess what you're seeing in terms of the fusion of
 the sensing data from different modalities as well as kind of fusion in the temporal level going more
 from frame by frame where you would have one net that would do frame by frame detection and camera,
 and then something that does frame by frame and lighter and then radar. Then you fuse it in a
 weaker engineered way later. The field over the last decade has been evolving in more kind of
 joint fusion, more end-to-end models that are solving some of these tasks jointly. There's
 tremendous power in that. That's the progression that our stack has been on as well. Now,
 so I would decouple the sensing and how that information is used from the role of ML and the
 entire stack. There's trade-offs in modularity and how do you inject inductive bias into your system.
 There's tremendous power in being able to do that. There's no part of our system that does not heavily
 leverage data-driven development or state of the art ML, but there's mapping, there's a simulator,
 or there's perception, object level perception, whether it's semantic understanding, prediction,
 decision-making, and so forth and so on. Of course, object detection and classification,
 like you're finding pedestrians and cars and cyclists and cones and signs and vegetation
 and being very good at estimating detection, classification, and state estimation, there's
 just stable stakes. That's step zero of this whole stack. You can be incredibly good at that,
 whether you use cameras or lighters or radar, but that's just stable stakes. That's just step zero.
 Beyond that, you get into the really interesting challenges of semantic understanding,
 the perception level. You get into scene-level reasoning. You get into very deep problems that
 have to do with prediction and joint prediction and interaction, so the interaction between all
 of the actors in the environment, pedestrians, cyclists, other cars, and you get into decision
 making. How do you build a lot of systems? We leverage ML very heavily in all of these components.
 I do believe that the best results you achieve by using a hybrid approach and having different
 types of ML, having different models with different degrees of inductive bias that you can have,
 and combining model-based approaches and some rule-based, physics-based systems.
 One example I can give you is traffic lights. There's a problem of the detection of traffic
 light state, and obviously, that's a great problem for computer vision. Confidants,
 that's their bread and butter. That's how you build that. But then the interpretation of a
 traffic light, you don't need to learn that. You don't need to build some complex ML model
 that infers with some precision and recall that red means stop. It's a very clear engineered
 signal with very clear semantics. You want to induce that bias, how you induce that bias and
 that whether it's a constraint or a cost function in your stack. But it is important to be able to
 inject that clear semantic signal into your stack. That's what we do. That's when you apply it to
 yourself when you are making decisions whether you want to stop for a red light or not.
 But if you think about how other people treat traffic lights, we're back to the ML version of
 that. You know they're supposed to stop for a red light, but that doesn't mean they will.
 Then you're back in the very heavy ML domain where you're picking up on very subtle cues
 that have to do with the behavior of objects, pedestrians, cyclists, cars, and the whole
 entire configuration of the scene that allow you to make accurate predictions on whether they will
 in fact stop or run a red light. So it sounds like already for Waymo, machine learning is a huge part
 of the stack. So it's a huge part of like not just so obviously the first level zero or whatever you
 said which is like just object detection of things that you know with know that machine learning can
 do but also starting to do prediction behavior and so on to model the what other or the other
 parties in the scene entities in the scene are going to do. So machine learning is more and more
 playing a role in that as well. Of course, absolutely. I think we've been going back to the
 earliest days like DARPA ERB, DARPA Grand Challenge. Our team was leveraging machine learning.
 I was like pre-Imagenet and it was a very different type of ML. And I think actually
 it was before my time but the Stanford team during the Grand Challenge had a very interesting
 machine learned system that would use lighter and camera when driving in the desert and it would
 kind of extend the range of free space reasoning. We get a clear signal from lighter and then it had
 a model that said, hey, like this stuff in camera kind of sort of looks like this stuff in lighter
 and I know this stuff that I've seen in lighter, I'm very confident this free space. So let me
 extend that free space zone into the camera range that would allow the vehicle to drive faster.
 And then we've been building on top of that and kind of staying and pushing the state of the art
 in ML in all kinds of different ML over the years. And in fact, we've been building on top of that
 in all kinds of different ML over the years. And in fact, from the earliest days, I think 2010
 is probably the year where Google, maybe 2011 probably, got pretty heavily involved
 in machine learning, kind of deep nuts. And at that time it was probably the only company that
 was very heavily investing in kind of state of the art ML and self-driving cars. And they go hand in
 hand. And we've been on that journey ever since. We're doing pushing a lot of these areas in terms
 of research at Waymo and we collaborate very heavily with the researchers in alphabet and
 all kinds of ML, supervised ML, unsupervised ML, published some interesting research papers in the
 space, especially recently. It's just super active learning as well. Yeah, super, super active.
 Of course, there's kind of the more mature stuff like, you know, convnets for object detection,
 but there's some really interesting, really active work that's happening in kind of more,
 you know, in bigger models and, you know, models that have more structure to them, you know, not
 just large bitmaps and reason about temporal sequences. And some of the interesting breakthroughs
 that we've seen in language models, right? You know, transformers, you know, GPT-3 inference.
 There's some really interesting applications of some of the core breakthroughs to those problems
 of behavior prediction as well as decision-making and planning, right? You can think about it,
 kind of the behavior, how, you know, the path, the trajectories, the how people drive. They have kind
 of a share a lot of the fundamental structure, you know, this problem. There's, you know, sequential
 nature. There's a lot of structure in this representation. There is a strong locality,
 kind of like in sentences, you know, words that follow each other. They're strongly connected,
 but there are also kind of larger context that doesn't have that locality. And you also see that
 in driving, right? What's happening in the scene as a whole has very strong implications on,
 you know, the kind of the next step in that sequence where whether you're predicting what
 other people are going to do, whether you're making your own decisions or whether in the simulator
 you're building generative models of, you know, humans walking, cyclists riding, the cars driving.
 That's all really fascinating. Like how it's fascinating to think that transforming models
 and all the breakthroughs in language and NLP that might be applicable to like driving at the
 higher level, at the behavioral level, that's kind of fascinating. Let me ask about pesky little
 creatures called pedestrians and cyclists. They seem, so humans are a problem. If we can get rid
 of them, I would. But unfortunately, they're also a source of joy and love and beauty. So
 let's keep them around. They're also our customers. Oh, for your perspective. Yes, yes, for sure.
 They're so some money. Very good. But I don't even know where I was going. Oh, yes, pedestrians
 and cyclists. I, you know, they're a fascinating injection into the system of uncertainty of,
 of like a game theoretic dance of what to do. And also they have perceptions of their own and they
 can tweet about your product. So you don't want to run them over from that perspective. I mean, I
 don't know. I'm joking a lot, but I think in seriousness, like, you know, pedestrians are
 complicated, computer vision problem, a complicated behavioral problem. Is there something
 interesting you could say about what you've learned from a machine learning perspective,
 from also an autonomous vehicle and a product perspective about just interacting with the
 humans in this world? Yeah, just, you know, just state on record, we care deeply about the safety
 of pedestrians, you know, even the ones that don't have Twitter accounts. Thank you. All right.
 But, you know, not me, but yes, I'm glad, I'm glad somebody does. Okay. But, you know, in all
 seriousness, safety of vulnerable road users, pedestrians or cyclists is one of our highest
 priorities. We do a tremendous amount of testing and validation and put a very significant emphasis
 on the capabilities of our systems that have to do with safety around those unprotected vulnerable
 road users. Cars, as we discussed earlier in Phoenix, we have completely empty cars,
 completely driverless cars driving in this very large area. And some people use them to go to
 school, so they will drive through school zones. Right. So kids are kind of the very special class
 of those vulnerable user road users, right? You want to be super, super safe and super, super
 cautious around those. We take it very, very, very seriously. And, you know, what does it take
 to be good at it? You know, an incredible amount of performance across your whole stack, you know,
 starts with hardware. And again, you want to use all sun signal modalities available to you.
 Imagine driving on a residential road at night and kind of making a turn and you don't have,
 you know, headlights covering some part of the space and like, you know, a kid might run out.
 And, you know, lighters are amazing at that. They see just as well in complete darkness as they do
 during the day. Right. So just again, it gives you that extra, you know, margin in terms of,
 you know, capability and performance and safety and quality. And in fact, we oftentimes in these
 kinds of situations, we have our system detect something in some cases even earlier than our
 trained operators in the car might do, especially in conditions like very dark nights. So starts
 with sensing, then, you know, perception has to be incredibly good. And you have to be very,
 very good at kind of detecting pedestrians in all kinds of situations and all kinds of environments,
 including, you know, people in weird poses, people kind of running around and, you know,
 being partially occluded. So, you know, that's stuff number one. Then you have to have in very
 high accuracy and very low latency in terms of your reactions to, you know, what these actors
 might do. Right. And we've put a tremendous amount of engineering and tremendous amount of validation
 into make sure our system performs properly. And oftentimes it does require a very strong reaction
 to do the safe thing. And we actually see a lot of cases like that. That's the long tail of really
 rare, you know, really crazy events that contribute to the safety around pedestrians. One example
 that comes to mind that we actually happened in Phoenix, where we were driving along and I think
 it was a 45 mile per hour road. So, you know, pretty high speed traffic and there was a sidewalk
 next to it. And there was a cyclist on the sidewalk. And as we were in the right lane,
 right next to the side, so it was a multi-lane road. So, as we got close to the cyclist on the
 sidewalk, it was a woman. She tripped and fell. Just, you know, fell right into the path of our
 vehicle. Right. And our, you know, car, you know, this was actually with a test driver, our test
 drivers did exactly the right thing. They kind of reacted and came to stop. It requires both very
 strong steering and, you know, strong application of the brake. And then we simulated what our
 system would have done in that situation. And it did exactly the same thing. And that speaks to
 all of those components of really good state estimation and tracking. And you can imagine,
 you know, a person on a bike and they're falling over and they're doing that right in front of you.
 Right. So you have to be really like, things are changing. The appearance of that whole thing is
 changing, right? And a person goes one way, they're falling on the road, they're, you know,
 being flat on the ground in front of you. You know, the bike goes flying the other direction.
 Like the two objects that used to be one are now, you know, are splitting apart and the car has to,
 like, detect all of that. Like milliseconds matter. And it doesn't, you know, it's not good enough to
 just brake. You have to like steer and brake and there's traffic around you. So, like, it all has
 to come together. And it was really great to see in this case and other cases like that, that we're
 actually seeing in the wild, that our system is, you know, performing exactly the way that we would
 have liked and is able to, you know, avoid collisions like this. It's such an exciting
 space for robotics. Like in that split second to make decisions of life and death. I don't know.
 The stakes are high in a sense, but it's also beautiful that for somebody who loves artificial
 intelligence, the possibility that an AI system might be able to save a human life,
 that's kind of exciting as a problem. Like to wake up. It's terrifying probably for an engineer
 to wake up and to think about, but it's also exciting because it's like it's in your hands.
 Let me try to ask a question that's often brought up about autonomous vehicles. And it might be fun
 to see if you have anything interesting to say, which is about the trolley problem.
 So the trolley problem is an interesting philosophical construct of that highlights,
 and there's many others like it, of the difficult ethical decisions that we humans
 have before us in this complicated world. So the specifically is the choice between
 if you are forced to choose to kill a group X of people versus a group Y of people,
 like one person, if you didn't, if you did nothing, you would kill one person, but if
 you would kill five people and if you decide to swerve out of the way, you would only kill one
 person. Do you do nothing or you choose to do something and you can construct all kinds of sort
 of ethical experiments of this kind that I think at least on a positive note, inspire you to think
 about like introspect, what are the, the physics of our morality? And there's usually not good
 answers there. I think people love it because it's just an exciting thing to think about.
 I think people who build autonomous vehicles usually roll their eyes because this is not
 this one as constructed, this like literally never comes up in reality. You never have to choose
 between killing one or like one of two groups of people, but I wonder if you can speak to,
 is there some, something interesting to you as an engineer of autonomous vehicles that's within the
 trolley problem or maybe more generally, are there, are there, are there, are there, are there, are
 there, are there, are there difficult ethical decisions that you find that algorithm must make?
 On the specific version of the trolley problem, which one would you do if you're driving?
 The question itself is a profound question because we humans ourselves cannot answer it.
 And that's the very point. I would kill both.
 Yeah. Humans, I think you're exactly right. And that, you know, humans are not particularly good.
 I think they kind of phrased it like what would a computer do, but humans, you know, are not very
 good. And actually oftentimes I think that, you know, freezing and kind of not doing anything
 because like you've taken a few extra milliseconds to just process and then you end up like doing the
 worst possible outcomes. Right. So I do think that as you've pointed out, it can be a bit of a
 distraction and it can be a bit of a kind of a red herring. I think it's an interesting discussion
 in the realm of philosophy, right. But in terms of what, you know, how that affects the actual
 engineering and deployment of self-driving vehicles, it's not how you go about building a system,
 right. We've talked about how you engineer a system, how you go about evaluating the different
 components and the, you know, the safety of the entire thing. How do you kind of inject the, you
 know, various model-based safety-based arguments and you're like, yes, you reason it parts of the system,
 you know, you reason about the probability of a collision, the severity of that collision, right.
 And that is incorporated and there's, you know, you have to properly reason about the uncertainty
 that flows through the system. Right. So, you know, those, you know, factors definitely play a role
 in how the cars then behave, but they tend to be more of like the immersion behavior. And what you
 see like you're absolutely right that these clear theoretical problems that they, you know, you don't
 occur to that in system and really kind of being back to our previous discussion of like what, you
 know, which one do you choose? Well, you know, oftentimes, like you made a mistake earlier,
 like you shouldn't be in that situation in the first place, right. And in reality, the system
 comes up. If you build a very good, safe and capable driver, you have enough, you know, clues
 in the environment that you drive defensively. So you don't put yourself in that situation,
 right. And again, you know, it has, you know, this, if you go back to that analogy of, you know,
 precision and recall, like, okay, you can make a very hard trade off of, but like neither answer
 is really good. But what instead you focus on is kind of moving the whole curve up. And then you
 focus on building the right capability and the right defensive driving so that, you know, you
 don't put yourself in a situation like this. I don't know if you have a good answer for this,
 but people love it when I asked this question about books. Are there books in your life that
 you've enjoyed? Philosophical, fiction, technical that had a big impact on you as an engineer or as
 a human being, you know, everything from science fiction to a favorite textbook. Is there three
 books that stand out that you can think of? Three books. So I would, you know, that impacted me,
 I would say, this one is, you probably know it well, but not generally well known. I think in the
 US or kind of internationally, The Master and Margarita. It's one of actually my favorite
 books. It is, you know, by Russian, it's a novel by Russian author, Mikhail Bulgakov. And it's just,
 it's a great book. It's one of those books that you can like reread your entire life and it's
 very accessible. You can read it as a kid. And like, you know, the plot is interesting. It's,
 you know, the devil, you know, visiting the Soviet Union. But like you read it, reread it
 at different stages of your life and you enjoy it for different, very different reasons. And you
 keep finding like deeper and deeper meaning and, you know, kind of affected, you know, definitely
 had an imprint on me, mostly from the probably kind of the cultural stylistic aspect. Like it
 makes you one of those books that, you know, is good and makes you think, but also has like this
 really, you know, silly, quirky, dark sense of humor. It captures the Russian soul more than
 perhaps many other books. On that like slight note, just out of curiosity, one of the saddest things
 is I've read that book in English. Did you by chance read it in English or in Russian?
 In Russian, only in Russian. And actually that is a question I had,
 kind of posed to myself every once in a while. I wonder how well it translates,
 if it translates at all. And there's the language aspect of it. And then there's the cultural
 aspect. So I, and actually I'm not sure if either of those would work well in English.
 Now I forget their names, but so when the COVID lifts a little bit, I'm traveling to Paris
 for several reasons. One is just, I've never been to Paris. I want to go to Paris, but there's a,
 the most famous translators of Dostoevsky, Tolstoy, of most of Russian literature,
 live there. There's a couple, they're famous, a man and a woman. And I'm going to sort of have
 a series of conversations with them. And in preparation for that, I'm starting to read
 Dostoevsky in Russian. So I'm really embarrassed to say that I've read, everything I've read in
 Russian literature of like serious depth has been in English. Even though I can also read, I mean,
 obviously in Russian, but for some reason it seemed in the optimization of life, it seemed the
 improper decision to do it, to read in Russian. Like, you know, like I don't need to opt, I need
 to think in English, not in Russian, but now I'm changing my mind on that. And so the question of
 how well it translates, it's a really fundamental one, like yet even with Dostoevsky. So from what
 I understand, Dostoevsky translates easier. Others don't as much. Obviously the poetry doesn't
 translate as well. I'm also the music, big fan of Vladimir Vysotsky. He doesn't obviously translate
 well. People have tried. But mastermind, I don't know, I don't know about that one. I just know
 it in English, you know, it's fun as hell in English. So, but it's a curious question and I
 want to study rigorously from both the machine learning aspect and also because I want to do a
 couple of interviews in Russia that I'm still unsure of how to properly conduct an interview
 across a language barrier. It's a fascinating question that ultimately communicates to an
 American audience. There's a few Russian people that I think are truly special human beings. And
 I feel like I sometimes encounter this with some incredible scientists and maybe you encounter this
 as well at some point in your life that it feels like because of the language barrier,
 their ideas are lost to history. It's a sad thing. I think about like Chinese scientists or even
 authors that like, that we don't in English speaking world, don't get to appreciate some like
 the depth of the culture because it's lost in translation. And I feel like I would love to show
 that to the world. Like I'm just some idiot, but because I have this like at least some semblance
 of skill in speaking Russian, I feel like, and I know how to record stuff on a video camera.
 I feel like I want to catch like Grigori Perlman, who's a mathematician. I'm not sure if you're
 familiar with him. I want to talk to him. Like he's a fascinating mind and to bring him to a
 wider audience in English speaking will be fascinating, but that requires to be rigorous
 about this question of how well Bulgakov translates. I mean, I know it's a silly concept,
 but it's a fundamental one because how do you translate? And that's the thing that Google
 Translate is also facing as a more machine learning problem, but I wonder as a more
 bigger problem for AI, how do we capture the magic that's there in the language?
 I think that's really interesting, really challenging problem. If you do read it,
 Master and Margarita in English, sorry, in Russian, I'd be curious to get your opinion.
 And I think part of it is language, but part of it is just centuries of culture that the cultures
 are different. So it's hard to connect that. Okay, so that was my first one. You had two more.
 The second one, I would probably pick the science fiction by the Strogatsky brothers.
 It's up there with Isaac Asimov and Ray Bradbury and company. The Strogatsky brothers kind of
 appealed more to me. I think it made more of an impression on me growing up. I, you know,
 I am a person. I apologize if I'm showing my complete ignorance. I'm so weak on sci-fi.
 What are they, right? Oh, roadside picnic, hard to be a god,
 beetle in an anthill, Monday starts on Saturday. Like it's not just science fiction. It's also
 like has very interesting, you know, interpersonal and societal questions. And some of the language
 is just completely hilarious. That's the one. That's interesting. Monday starts on Saturday.
 So I need to read. Okay. Oh, boy. You put that in the category of science fiction. That one is,
 I mean, this was more of a silly, you know, humorous work. I mean, there is kind of,
 but it's profound too, right? Science fiction, right? It's about, you know, this research
 institute and like it has deep parallels to like serious research, but the setting of course is
 that they're working on, you know, magic, right? And there's a lot of stuff. So I, and that's
 their style, right? And, you know, other books are very different, right? You know, hard to be a
 god, right? It's about kind of this higher society being injected into this primitive world and how
 they operate there. Some of the very deep ethical questions there, right? And like they've got this
 full spectrum, some is more about kind of more adventure style, but like I enjoy all of their
 books. There's probably a couple, actually one, I think that they consider it their most important
 work. I think it's the snail on a hill. I'm not exactly sure how it translates. I tried reading
 a couple of times. I still don't get it, but everything else I fully enjoyed. And like for
 one of my birthdays as a kid, I got like their entire collection, like occupied a giant shelf
 in my room. And then like over the holidays, I just like, you know, my parents couldn't drag
 me out of the room and I read the whole thing cover to cover and it, I really enjoyed it.
 And that's one more. For the third one, maybe a little bit darker, but you know, comes to mind is
 Orwell's 1984. And I, you know, I asked what made an impression on me and the books that people
 should read. That one, I think falls in the category of both. Definitely it's one of those
 books that you read and you just kind of put it down and you stare in space for a while.
 That kind of work. I think there's lessons there. People should not ignore. And you know,
 nowadays with like everything that's happening in the world, I can't help it, but you know,
 I have my mind jump to some parallels with what Orwell described. And like this whole
 concept of double think and ignoring logic and holding completely contradictory opinions
 in your mind and not have that not bother you and stick into the party line at all costs.
 Like, you know, there's something there. If anything, 2020 has taught me and I'm a
 huge fan of Animal Farm, which is a kind of friendly as a friend of 1984 by Orwell. It's kind of
 another thought experiment of how our society may go in directions that we wouldn't like it to go.
 But if, if anything that's been kind of heartbreaking to an optimist about 2020 is that
 that society is kind of fragile. Like we have this, this is a special little experiment we have going
 on and not, it's not unbreakable. Like we should be careful to like preserve whatever special thing
 we have going on. I mean, I think 1984 and these books, Brave New World, they, they're helpful in
 thinking like stuff can go wrong in non-obvious ways. And it's like, it's up to us to make sure
 it's up to us to preserve it. And it's like, it's a responsibility. It's been weighing heavy on me
 because like, for some reason, like more than my mom follows me on Twitter. And I feel like I have,
 I have like now somehow responsibility to, to this world. And it dawned on me that like me and
 millions of others are like the little ants that maintain this little colony. Right. So we have a
 responsibility not to be, I don't know what the right analogy is, but I'll put a flame thrower to
 the place. We want to not do that. And there's interesting, complicated ways of doing that. As
 1984 shows, it could be through bureaucracy, it could be through incompetence. It could be through
 misinformation. It could be through division and toxicity. I'm a huge believer in like that love
 will be the, somehow the solution. So love and robots, robots. Yeah. I think you're exactly
 right. Unfortunately, I think it's less of a flame thrower type of an answer. I think it's more of a,
 in many cases can be more of a slow boil and that that's the danger. Let me ask. It's a fun thing to
 make a world-class roboticist, engineer and leader uncomfortable with the ridiculous question about
 life. What is the meaning of life, Dmitri, from a robotics and a human perspective?
 You only have a couple of minutes or one minute to answer. So
 I don't know if that makes it more difficult or easier. Actually, yeah.
 You know, they're very tempted to quote one of the stories, stories by Isaac Asimov actually,
 actually titled appropriately titled the last question, short story where the plot is that
 humans build this supercomputer, this AI intelligence and once it gets powerful enough,
 they pose this question to it. How can the entropy in the universe be reduced? So the computer
 replies as of yet insufficient information to give a meaningful answer. And then thousands of years
 go by and they keep posing the same question and computer gets more and more powerful and keeps
 giving the same answer as of yet insufficient information to give a meaningful answer or
 something along those lines. And then it keeps happening and happening. You fast forward millions
 of years into the future and billions of years and at some point it's just the only entity in
 the universe. It's absorbed all humanity and all knowledge in the universe and keeps posing the
 same question to itself. And finally, it gets to the point where it is able to answer that question.
 But of course, at that point, yeah, there's, you know, the heat death of the universe has occurred
 and that's the only entity and there's nobody else to provide that answer to. So the only thing it
 can do is to answer it by demonstration. So it like, you know, recreates the big bang, right?
 And resets the clock, right? But I can try to give kind of a different version of the answer,
 you know, maybe not on the behalf of all humanity. I think that that might be a little presumptuous
 for me to speak about the meaning of life on the behalf of all humans, but at least, you know,
 personally, it changes, right? I think if you think about kind of what gives you and your life
 meaning and purpose and kind of what drives you, it seems to change over time, right? And
 the lifespan of your existence. When you just enter this world, right, it's all about kind of
 new experiences, right? You get like new smells, new sounds, new emotions, right? And like,
 that's what's driving you, right? You're experiencing new amazing things, right? And
 that's magical, right? That's pretty awesome, right? That gives you kind of a meaning.
 Then you get a little bit older, you start more intentionally learning about things.
 Right? I guess actually, before you start intentionally learning, probably fun. Fun is
 a thing that gives you kind of meaning and purpose and purpose and the thing you optimize for,
 right? And like fun is good. Then you get, you know, start learning. And I guess that this
 joy of comprehension and discovery is another thing that, you know, gives you meaning and
 purpose and drives you, right? Then, you know, you learn enough stuff and it, you want to give some
 of it back, right? And so impact and contributions back to technology or society, people, you know,
 local or more globally, becomes a new thing that drives a lot of kind of your behavior and
 something that gives you purpose and that you derive positive feedback from, right?
 Then you go and so on and so forth. You go through various stages of life. If you have kids,
 like that definitely changes your perspective on things. You know, I have three that definitely
 flips some bits in your head in terms of kind of what you care about and what you optimize for and
 you know, what matters, what doesn't matter, right? So, you know, and so on and so forth,
 right? And I, it seems to me that, you know, it's all of those things. And as you kind of go through
 life, you know, you want these to be additive, right? New experiences, fun, learning, information,
 learning impact, like you want, you want to, you know, be accumulating. I don't want to, you know,
 stop having fun or experiencing new things. And I think it's important that, you know,
 it just kind of becomes additive as opposed to a replacement or subtraction. But, you know,
 those few as far as I got, but, you know, ask me in a few years, I might have one or two more to add
 to the list. And before you know it, time is up just like it is for this conversation. But hopefully
 it was a fun ride. It was a huge honor to meet you. As you know, I've been a fan of yours and a fan of
 Google self-driving car and Waymo for a long time. I can't wait. I mean, it's one of the most exciting.
 If we look back in the 21st century, I truly believe it will be one of the most exciting things we
 descendants of apes have created on this earth. So I'm a huge fan and I can't wait to see what you do
 next. Thanks so much for talking to me. Thanks. Thanks for having me. And it's a
 also a huge fan. It doesn't work. Honestly, and I really enjoyed it. Thank you.
 Thank you for listening and hope to see you next time.