Self-driving from VISION ONLY - Tesla's self-driving progress by Andrej Karpathy (Talk Analysis)
#tesla #selfdriving #karpathy

Tesla is pushing the state-of-the-art in full self-driving, and interestingly, they explicitly switch from having multiple different sensors to a vision-only system. We discuss the highlights of Andrej Karpathy's talk about Tesla's FSD system, how to label petabytes of data, how to sample edge-cases, how to train a neural network that has to work in real-time, and why moving to having only cameras is superior to multi-sensor approaches.

OUTLINE:
0:00 - Intro & Overview
1:55 - Current Auto-Breaking system
3:20 - Full Self-Driving from vision only
4:55 - Auto-Labelling for collecting data
8:45 - How to get diverse data from edge-cases
12:15 - Neural network architecture
16:05 - Tesla's in-house supercomputer
17:00 - Owning the whole pipeline
18:20 - Example results from vision only
23:10 - Conclusion & Comments

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 All right, hello, everyone. Today we're going to look at Andrej Karpathy's CVPR talk about full self driving mode in Tesla and what Tesla has been doing to push that beyond its current state. So let's just say that autonomous driving is a hard problem. You have to control a car and pretty much anything could happen. However, we're able to teach it to pretty much any human on the planet. So the problem is definitely solvable. Now the current stack they have for full self driving or that they intended to use it seems like is what they call sensor fusion, which is where you take a bunch of different signals like camera signals, and radar signals and so on. And you try to fuse their signals together. This kind of works, it seems but it runs into problems such as what do you do when the different sensors disagree? And it turns out solving that problem is quite hard. And that's why Tesla apparently is transitioning to a fully only vision stack. Everything is going to be vision based in Tesla full self driving. Now today, we're going to look at the best and important bits of the talk right here. Now absolutely invite you to go watch the entire talk if you're interested. It is enjoyable in full length and it is on YouTube. Andrej gives a lot of good examples here and the amount of effort that went into engineering this into collecting the data, how this is deployed is astounding. Now keep in mind, this is the lead AI scientist for Tesla. So it is going to be a bit of an ad. However, it is pretty cool to see that we are actually making a real push towards full self driving. A lot of people have been super salty saying that Elon Musk has promised this like one or two years ago already. But come on. I mean, do you see anyone else doing fully self driving at this level? No, so shut up. So the first thing right here is a couple of scenarios of what Tesla is already doing, which is sort of a driver assistance. So if the person is driving, but the system is relatively sure that the person is making a mistake, the system kicks in mostly to do automatic braking for the user. So I just I want to show you this one example right here. You start slowly and probably you know, does not actually enter the intersection. These are examples from pedal misapplication mitigation. Here a person is parking from the driving spot and they are trying to turn and then they mess up and they accidentally floor it. So they floor it right there. So you see like the person wanted to break but stepped on the gas. There are people right in front of the car. So be salty all you want. This right here is already worth it. As a human, there is a lot of resistance against fully self driving feeling that you're no longer in control anymore. But the matter of the fact is that these systems already are and in the near future will be even much more better than humans at driving is going to be much cleaner, much safer, much faster, less traffic jams and so on to let the machines take over the driving pretty much in the same way as it's much safer to let the machines take over the braking in these scenarios. The only times you're actually going to drive by hand is when you do it for fun. Now I drive a motorbike. It's a lot of fun to drive but in a car especially with other people or if I do it for work if I may be a little bit tired machines all the way. So the full self driving beta is rolled out to a small handful of customers right now. And they do upload YouTube videos every now and then of what they're doing. And it seems to work fairly fairly well. Apparently they had had no crashes so far while driving about 1.7 million miles in full self driving. You can see on the screen in the middle right here that the predictions that the system gives is pretty good. Though we've also seen some other prediction that are not so good throughout YouTube. Like there's this one video where the truck in front of the car has street lights on its back and the car just keeps thinking it's kind of red lights. However we don't know if this is the legacy stack or not and if the car would actually break since the lights are not on red but it's been a scare going around YouTube for a little bit. So here Andre shows a video of Waymo already doing this much earlier than Tesla having sort of an automatic car drive around an intersection and so on. This works if you're in a really defined zone, let's say a city that you know that you have accurate maps for. This does not work if you want to do this anywhere in the world. To do this anywhere in the world, you need to rely on the car itself. That means you need a lot of data. So the data that this new system gets is just vision. It's eight cameras around the car. And that's it. And Andre makes a good case here that that is actually all you need. Humans are able to navigate from this and cars should be able to do the same. So an absolutely necessary ingredient to train such a system is a good clean label data set. If you just wanted to use humans to annotate every single frame of cars driving around that will probably be prohibitively expensive even for Tesla. So they came up with what I think is a pretty cool method called auto labeling. Now I'm sure they're not the inventors of the system. But to use it on this scale is very smart. And it works out pretty nicely. Of course, we need to collect training data. A typical approach might be to use humans to annotate cars around us in three dimensions. What we found actually works really well is an auto labeling approach. So it's not your humans just like annotating cars. It's an offline tracker as we call it. And it's an auto labeling process for collecting data at the scale that is necessary. So we need to get millions of hard examples. So this is where the scale comes from is that it's not labeled by humans. Although humans are involved, it's labeled automatically. So here's an example of some automatic labels we were able to derive for cars on the highway. And the way to do this is because you are offline and you are trying to just annotate a club, you have a large number of benefits that you don't typically have if you're at test time under strict latency requirements in the car. So you can take your time to fully figure out exactly all the objects in your data. You can use neural networks that are extremely heavy. They are not deployable for various reasons. You can use benefit of hindsight because you know the future of not just the past. You can use all kinds of expensive offline optimization and tracking techniques. You can use extra sensors. In this case, for example, actually radar was one of the sensors that we use for the auto labeling. But there's actually a massive difference between using radar at test time and using it in the offline track. So the point here is that if you record data, and you're trying to figure out at inference time, like while you're driving, what's happening, it's a lot harder than if you have the same data, but kind of at home in the lab. So what you want to do is you want to drive around and just record not even not predict or anything, just record data record from all your sensors, you can even stick expensive sensors on the cars where you collect the data. And then you take all that data, and you use the biggest heaviest processors, you have to figure out what actually happened during that time. What he mentions here is the benefit of hindsight, which means that if you're in a car and you're driving, and all of a sudden something obscures your vision, you will be sort of lost because all you have, okay, you can maybe guess that a car in front of you is still there. But who knows, they might turn or something. Now, if you record the whole video sequence, you're able to see what happens beyond the obstruction of vision. And if you see the car is still there, you can make a good inference that the car was actually there the whole time. And therefore you can annotate that data with a label saying, hey, that car was there the whole time, you can also do active learning and shell out to actual human annotators what you're not sure about. So this benefit of hindsight is really important here when you're under the time constraint of not being able to see into the future, as well as the latency constraint. And you have to have like an efficient neural network in the lab, you don't have any of this the method here, if you're developing something real time, I mean, this might seem obvious to you, I found it to be pretty cool. Yes, record, then figure out what happened, then use that as a labeled data set. So here's an example of how such a persistent track would look like after the neural network has been trained on data like this. Here's some examples of really tricky scenarios. I don't actually know exactly what this is. But basically, this car draws a bunch of debris on us, and we maintain a consistent track for the label. And of course, if you have millions of labels like this, the neural net, if it's a powerful enough neural net, will actually end up learning to persist these tracks in these kinds of scenarios. Here's another example. There's a car in front of us, I actually am not 100% sure what happens in this case. But as you'll see, there's some kind of a dust cloud that develops here and briefly occludes the car. But in the auto labeling tool, we are able to persist this track because we saw it before and we saw it after so we can actually stitch it up and use it as a training set for the neural. So that's how they get clean labels in an automatic or semi automatic way. But they still need to get a lot of data from kind of edge cases because most of driving is quite uneventful, straight driving and was done 40 years ago or something like this. I think Schmidhuber in GTC 21 talk talked about autonomous cars on highways on controlled stretches of highways super duper early already. So what we really need to collect is edge cases. And for collecting these edge cases, Tesla has developed these what they call triggers. So these are kind of hand programmed rules of what data should go into the annotation pipeline. So imagine if all these cars driving around not only the people with full self driving, but the detection, the actual recording of data is activated in all the Tesla cars driving around, they all send that data back to the server. Of course, that's way too much data. And also, it's very unbalanced in terms of how many critical situations are in there. Again, most of it will be sort of straight road, empty, just drive straight. So what they do is they filter this data for these trigger events. Now these trigger events can be as simple as whenever the radar and the vision mismatch. So whenever they disagree on something, that's an interesting example. But you know, it goes into very detailed such as we detect breaking lights, but the acceleration is positive. So with these triggers, they're able to source a diverse set of training samples and edge cases where the neural network can learn the tricky situations rather than just the long stretches of road. So I think it's safe to say that a good mark of quality on these systems is going to be how well these triggers are maintained, like how well do they represent the full driving experience of the end users of the cars. But so far from the results we got, it seems like they cover the road situations fairly well. And all of them are iteration and you're looking at what's coming back, you're tuning your trigger and you're sourcing data from all these scenarios. Basically over the last four months, we've done quite extensive data engine, we've ended up doing seven chatter modes and seven loops around this data engine here, where on the top right is where you begin, you have some seed data set, you train your neural network on your data set, and you deploy the neural network in the customer cars in shadow mode. And the network is silently making prediction. By the way, if you if you like squint really hard, I don't know if this is just a depiction of a neural network, or if this is the actual architecture they're using, I don't think so. But there is like a stride of six in there and max pooling, you know, just just noting that for no particular reason. And then you have to have some mechanisms for sourcing inaccuracies of the neural network, you're just looking at its predictions, and then you're using one of these triggers, you're getting these scenarios where the network is probably misbehaving, some of those clips end up going to unit tests to make sure that we even if we're failing right now, we make sure we pass later. And in addition, those examples are being auto labeled and incorporated into a training set. And then as a synchronous process, we're also always data cleaning the current training set. So we spend this loop over and over again, until the network basically becomes incredibly good. So in total, we've done seven rounds of shadow mode for this release. So shadow mode is what they call when they let the predictions run, but they don't hook them up to the control. So you're driving yourself, but the system predicts all the time. And whenever one of these trigger happens, that's an interesting data point that is going to send back to the server. Actually, let's be honest, it's probably going to send everything back to the server. So the data set they come up with is 1.5 petabytes of data. So that's crazy. So next is going to go into the architecture of the neural net. And this is also fairly interesting and not entirely standard on the top. All of them are processed by an image extractor, the layout of the synthetic visual cortex in order to efficiently process this information, our architecture roughly looks like this, we have these images coming from multiple cameras on the top, all of them are processed by an image extractor, like a backbone, like think resonant kind of style, then there's a multi cam fusion that uses the information from all the eight to use. And this is a kind of a transformer that we use to fuse this information. And then we fuse information first across all the cameras, and then across all of time. And that is also done either by transformer by recurrent neural network, or just by three dimensional convolutions. We've experimented with a lot of kind of fusion strategies here to get this to work really well. And then what we have afterwards, after the fusion is done is we have this branching structure that doesn't just consist of heads, but actually, we've expanded this over the last few last year or so where you now have heads that branch into trunks that branch into terminals. So there's a lot of branching structure. And the reason you want this branching structure is because there's a huge amount of outlets that you're interested in. And you can't afford to have a single neural network for every one of the individual outlets, you have to of course, amortize the forward pass. So this is pretty interesting. The top part here, what they call the backbone is pretty standard. If you have a video, especially with multiple cameras, you want to extract information from each frame of each camera sort of individually, then you want to fuse that information across all the cameras for a single time step. And then you want to fuse that information with the information of all the other time steps. So so far, so good. That sort of gives you a representation of what happens in these frames in these cameras during that stretch of time. However, after that, usually, even if you have multiple predictions, what you would do is you would sort of have like one prediction head on top of that backbone. However, since they are in a car and have to decide real fast, it's not really feasible to have sort of these different columns for each of the prediction tasks. Because as he says, they're interested in a lot of different signals, think depth prediction, which means that for every pixel, you have to provide a depth estimation, think tracks of other cars, think pedestrians think streetlights think okay, where are the lanes at or navigation in general. So all these signals are things to predict. And it's not good enough to have like a separate head for each of the predictions. So what they do is they have as you call these branching structures where there are multiple heads, yes. And within these multiple heads, there are what they call trunks. And within the trunks, there are the individual like little what they call terminals. Essentially, it's a hierarchical prediction, I'm going to guess that the tasks that go together, sort of are grouped together. So maybe one head is for all the pixel prediction tasks, and another head is more for the classification tasks. And then within one head, you have a trunk that deals more with like object classification and another trunk that deals more with like navigation classification. And the individual terminals then do the actual tasks. This is a pretty cool way of getting a highly performant many output network all together such that its size and computational speed are still maintained. The other nice benefit of the branching structure is that it decouples at the terminals, it decouples all these signals. So if I am someone working on velocity for a particular object type or something like that, I have a small piece of neural network that I can actually fine tune without touching any of the other signals. And so I can work in isolation to some extent and actually get something to work pretty well. And then once in a while, so basically, the iteration scheme is that a lot of people are fine tuning. And once you just got to imagine the ML ops behind this, it's like, Hey, where do you deploy your models? I do it on the Kubernetes, I have ML flow. Oh, no, I use the TensorFlow extended. Yeah, it's pretty cool. What do you do? car, I deploy on car. So next, he's going into this in house supercomputer that they built or are building. And this is a massive thing. Absolutely massive. He says that in terms of flops, it's something like the fifth biggest computer in the world. Its storage speed is incredible. So I'm pretty sure you could even actually render Far Cry two on this thing, maybe, but in total, it has 5760 GPU is not any GPUs, the most expensive a 180 gigabyte GPUs, it would be interesting to see what kind of algorithms they use on top of this to actually do the distributed training or whether it's all just kind of simple data parallelism, aggregating gradients and so on. Of course, they have super fast interconnect super fast storage, super fast everything and it looks sweet. Like is this a stock photo of a server room? Or is this the actual server room? This effort to basically is incredibly vertically integrated in the AI team. So as I showed you, we own the vehicle and the sensing and resource our own data and we annotate our own data and we train our on prem cluster and then we deploy all of the neural networks that we train on our in house developed chip. So we have the FSD computer here that has two SOCs as the chips here and they have our own custom and view neural processing unit here at roughly 36 times each. So these chips are specifically designed for the neural networks that we want to run for. Yeah, I mean, this is the dream, right? If you're an AI professional, only the whole pipeline is going to boost your productivity by so much. You're not bound by the constraint of anything other than the limits on the final system, which is a car so fairly difficult. But in between of that you have control over everything you have control over how the data is collected annotated, you have control over where it is deployed to on what architecture of chip because you make the chip. So I guess the lesson is if you're looking to change the world, you better own a good chunk of it. So that's going to show some examples of what this new vision only stack could do. Remember, they used to do fusion of sensors, which means they essentially have radar, they have vision, maybe some other sensors, and they try to integrate this information from all of the sensors, they compare this to the new vision based system. Now check out what happens in terms of the depth and velocity predictions that we're able to achieve by putting all these pieces together and training these networks at scale. So the first example here, I have a video where this is on track testing. So this is an engineering car, and we asked it to slam on the brakes as hard as it possibly can. So this is a very harsh braking here in front of us, even though it doesn't look like that in the videos is very harsh braking. So what you can see on the right here is you can see the outputs from the legacy stack, which had radar vision fusion and from the new stack, which is vision alone in blue. So in the orange legacy stack, you can actually see these track drops here when the car was breaking really harshly. And basically the issue is that the braking was so harsh that the radar stack that we have actually ended up not associating car and dropping the track and then reinitializing it all the time. And so it's as if the vehicle disappeared and reappeared like six times during the period of this braking. And so this created a bunch of artifacts here, but we see that the new stack in blue is actually not subject to this behavior at all. It just gives a clean signal. In fact, here, there's no smoothing, I believe, on the blue signal here. This is the raw depth and velocity that comes out from the neural net, the final neural net that we released with about three weeks ago. And you can see that it's fairly smooth here. And of course, you could go into the radar stack and you could, you know, adjust the hyper parameters of the tracker, like why is it dropping tracks and so on. But then you are spending engineering efforts and focus on a stack that is like not really barking up the right tree. And so it's better to again, focus on the vision and make it work really well. And we see that it is much more robust when you train it at scale. So there you have it proof by one example that the new thing works better. Isn't that every CVPR paper ever. But no, in any case, I can totally believe that the new stack even though it drops a bunch of the sensors is better. Because ultimately, if your one sensor if vision is so performant that in every single disagreement, you go with the vision thing, then why do you have the other sensors at all the thing in front of it is just kind of breaking too fast. So the radar kind of loses it and then regains it and loses it and regains it. Now, I have no idea how radar works. So I'm speaking from complete ignorance right here. But what I'm going to guess as far as I understand it, is that radar just kind of gives you the velocities of stuff in front of you. And then there is a tracking algorithm on top of radar that tries to figure out which stuff is the same stuff. And this is very much what they do in this auto labeling where they have sort of a track on something, right, and then they use hindsight, and then they have a tracking algorithm that decides which things are the same, even though we don't see them all the time. And here you can clearly see the benefit of shifting this from inference time, which is what you have to do with radar to the training time, which is what you can do with vision. So you can teach the vision system to sort of do this persistent tracking, whereas the radar system, you have to hand tune it to do this in real time. Now it makes the point that of course, you could go into the radar system, change the hyper parameters, but then he says why bark up the wrong tree? Why waste time on a stack that isn't functioning? It's a bit of a chicken and an egg problem, right? If you were to put as much effort into the radar stack as you were into the vision system, I'm going to guess that these results would go away, and that is able to keep up maybe, but the arguments for going vision only is a strong one. And I don't doubt that it is probably a good way forward. And basically what's happening here is that the radar is very trigger happy and it sees all these false stationary objects everywhere. Like everything that like sticks out is a stationary target and radar by itself doesn't know what actually is a stationary car and what isn't. So it's waiting for vision to associate with it and vision, if it's not held up to a high enough bar is noisy and contributes error and the sensor fusion stack just kind of like picks it up too late. And so again, you could fix all that, even though it's a very gross system with a lot of if statements and so on, because the sensor fusion is complicated because the error modes for vision and radar are slightly are quite different. But here, when we just work with vision alone and we take out the radar, vision recognizes this object very early gives the correct depth and velocity and there's no issues. So we actually get an initial slowdown much earlier and really like simplify the stack a lot. Yeah. So here you can see the same failure mode in vision that it kind of gets a track but doesn't but get a track but doesn't. The important part is that once you get closer to the object, it is fairly consistent, right? As you can see right here, the vision stack recognizes this truck on the side much earlier than the radar stack did. Now again, this might just be a function of the hyper parameters used, I'm sure you could just lower the threshold for the radar, but you'd run into different problems. During the Q&A, he makes a good point in that yes, other sensors would be nice to have, but just the pure economics speak in favor of vision to like we develop cameras with much more rigor as a society than we do radar systems. And therefore, the camera sensors are just so much better nowadays and cheaper. So you can afford to build many of them into all kinds of things and collect data and make your systems better through that then to put kind of a lidar on top of the car and having to sort of fuse those signals with the vision signals, especially when they're in conflict with one another. So if you ask me, I'm a fan. I like what I see here, even though I know it's kind of an ad. I don't own a Tesla, but I think it's still pretty cool. In the end, he talks a bit about what they do to validate this data, and how they roll it out and gives a bunch of more examples of tracking. And there's a Q&A at the end. So if you are interested in that, I absolutely welcome you to go watch the entire talk. It is on YouTube. And that was it for me. I hope you enjoyed this and I'll see you next time. Ciao!
[{"start": 0.0, "end": 6.48, "text": " All right, hello, everyone. Today we're going to look at Andrej Karpathy's CVPR talk about"}, {"start": 6.48, "end": 12.52, "text": " full self driving mode in Tesla and what Tesla has been doing to push that beyond its current"}, {"start": 12.52, "end": 17.76, "text": " state. So let's just say that autonomous driving is a hard problem. You have to control a car"}, {"start": 17.76, "end": 21.76, "text": " and pretty much anything could happen. However, we're able to teach it to pretty much any"}, {"start": 21.76, "end": 27.0, "text": " human on the planet. So the problem is definitely solvable. Now the current stack they have"}, {"start": 27.0, "end": 32.04, "text": " for full self driving or that they intended to use it seems like is what they call sensor"}, {"start": 32.04, "end": 37.16, "text": " fusion, which is where you take a bunch of different signals like camera signals, and"}, {"start": 37.16, "end": 42.72, "text": " radar signals and so on. And you try to fuse their signals together. This kind of works,"}, {"start": 42.72, "end": 47.84, "text": " it seems but it runs into problems such as what do you do when the different sensors"}, {"start": 47.84, "end": 53.72, "text": " disagree? And it turns out solving that problem is quite hard. And that's why Tesla apparently"}, {"start": 53.72, "end": 60.24, "text": " is transitioning to a fully only vision stack. Everything is going to be vision based in"}, {"start": 60.24, "end": 65.48, "text": " Tesla full self driving. Now today, we're going to look at the best and important bits"}, {"start": 65.48, "end": 69.88, "text": " of the talk right here. Now absolutely invite you to go watch the entire talk if you're"}, {"start": 69.88, "end": 74.72, "text": " interested. It is enjoyable in full length and it is on YouTube. Andrej gives a lot of"}, {"start": 74.72, "end": 80.86, "text": " good examples here and the amount of effort that went into engineering this into collecting"}, {"start": 80.86, "end": 87.72, "text": " the data, how this is deployed is astounding. Now keep in mind, this is the lead AI scientist"}, {"start": 87.72, "end": 92.92, "text": " for Tesla. So it is going to be a bit of an ad. However, it is pretty cool to see that"}, {"start": 92.92, "end": 98.28, "text": " we are actually making a real push towards full self driving. A lot of people have been"}, {"start": 98.28, "end": 104.28, "text": " super salty saying that Elon Musk has promised this like one or two years ago already. But"}, {"start": 104.28, "end": 110.2, "text": " come on. I mean, do you see anyone else doing fully self driving at this level? No, so shut"}, {"start": 110.2, "end": 115.72, "text": " up. So the first thing right here is a couple of scenarios of what Tesla is already doing,"}, {"start": 115.72, "end": 121.32000000000001, "text": " which is sort of a driver assistance. So if the person is driving, but the system is relatively"}, {"start": 121.32000000000001, "end": 126.4, "text": " sure that the person is making a mistake, the system kicks in mostly to do automatic"}, {"start": 126.4, "end": 131.32, "text": " braking for the user. So I just I want to show you this one example right here."}, {"start": 131.32, "end": 135.88, "text": " You start slowly and probably you know, does not actually enter the intersection. These"}, {"start": 135.88, "end": 141.44, "text": " are examples from pedal misapplication mitigation. Here a person is parking from the driving"}, {"start": 141.44, "end": 146.51999999999998, "text": " spot and they are trying to turn and then they mess up and they accidentally floor it."}, {"start": 146.51999999999998, "end": 151.2, "text": " So they floor it right there. So you see like the person wanted to break but stepped on"}, {"start": 151.2, "end": 156.0, "text": " the gas. There are people right in front of the car. So be salty all you want. This right"}, {"start": 156.0, "end": 160.64, "text": " here is already worth it. As a human, there is a lot of resistance against fully self"}, {"start": 160.64, "end": 165.48, "text": " driving feeling that you're no longer in control anymore. But the matter of the fact is that"}, {"start": 165.48, "end": 171.0, "text": " these systems already are and in the near future will be even much more better than"}, {"start": 171.0, "end": 177.39999999999998, "text": " humans at driving is going to be much cleaner, much safer, much faster, less traffic jams"}, {"start": 177.39999999999998, "end": 182.35999999999999, "text": " and so on to let the machines take over the driving pretty much in the same way as it's"}, {"start": 182.35999999999999, "end": 187.16, "text": " much safer to let the machines take over the braking in these scenarios. The only times"}, {"start": 187.16, "end": 193.12, "text": " you're actually going to drive by hand is when you do it for fun. Now I drive a motorbike."}, {"start": 193.12, "end": 199.5, "text": " It's a lot of fun to drive but in a car especially with other people or if I do it for work if"}, {"start": 199.5, "end": 205.88, "text": " I may be a little bit tired machines all the way. So the full self driving beta is rolled"}, {"start": 205.88, "end": 211.76, "text": " out to a small handful of customers right now. And they do upload YouTube videos every"}, {"start": 211.76, "end": 217.4, "text": " now and then of what they're doing. And it seems to work fairly fairly well. Apparently"}, {"start": 217.4, "end": 224.3, "text": " they had had no crashes so far while driving about 1.7 million miles in full self driving."}, {"start": 224.3, "end": 227.84, "text": " You can see on the screen in the middle right here that the predictions that the system"}, {"start": 227.84, "end": 233.72, "text": " gives is pretty good. Though we've also seen some other prediction that are not so good"}, {"start": 233.72, "end": 237.92000000000002, "text": " throughout YouTube. Like there's this one video where the truck in front of the car"}, {"start": 237.92000000000002, "end": 243.92000000000002, "text": " has street lights on its back and the car just keeps thinking it's kind of red lights."}, {"start": 243.92, "end": 248.6, "text": " However we don't know if this is the legacy stack or not and if the car would actually"}, {"start": 248.6, "end": 253.04, "text": " break since the lights are not on red but it's been a scare going around YouTube for"}, {"start": 253.04, "end": 258.44, "text": " a little bit. So here Andre shows a video of Waymo already doing this much earlier than"}, {"start": 258.44, "end": 263.65999999999997, "text": " Tesla having sort of an automatic car drive around an intersection and so on. This works"}, {"start": 263.65999999999997, "end": 269.24, "text": " if you're in a really defined zone, let's say a city that you know that you have accurate"}, {"start": 269.24, "end": 275.36, "text": " maps for. This does not work if you want to do this anywhere in the world. To do this"}, {"start": 275.36, "end": 281.16, "text": " anywhere in the world, you need to rely on the car itself. That means you need a lot"}, {"start": 281.16, "end": 286.76, "text": " of data. So the data that this new system gets is just vision. It's eight cameras around"}, {"start": 286.76, "end": 292.08, "text": " the car. And that's it. And Andre makes a good case here that that is actually all you"}, {"start": 292.08, "end": 297.12, "text": " need. Humans are able to navigate from this and cars should be able to do the same. So"}, {"start": 297.12, "end": 302.32, "text": " an absolutely necessary ingredient to train such a system is a good clean label data set."}, {"start": 302.32, "end": 307.84000000000003, "text": " If you just wanted to use humans to annotate every single frame of cars driving around"}, {"start": 307.84000000000003, "end": 313.76, "text": " that will probably be prohibitively expensive even for Tesla. So they came up with what"}, {"start": 313.76, "end": 319.76, "text": " I think is a pretty cool method called auto labeling. Now I'm sure they're not the inventors"}, {"start": 319.76, "end": 326.68, "text": " of the system. But to use it on this scale is very smart. And it works out pretty nicely."}, {"start": 326.68, "end": 330.40000000000003, "text": " Of course, we need to collect training data. A typical approach might be to use humans"}, {"start": 330.40000000000003, "end": 334.6, "text": " to annotate cars around us in three dimensions. What we found actually works really well is"}, {"start": 334.6, "end": 338.0, "text": " an auto labeling approach. So it's not your humans just like annotating cars. It's an"}, {"start": 338.0, "end": 341.92, "text": " offline tracker as we call it. And it's an auto labeling process for collecting data"}, {"start": 341.92, "end": 344.72, "text": " at the scale that is necessary. So we need to get millions of hard examples. So this"}, {"start": 344.72, "end": 347.64, "text": " is where the scale comes from is that it's not labeled by humans. Although humans are"}, {"start": 347.64, "end": 350.78000000000003, "text": " involved, it's labeled automatically. So here's an example of some automatic labels we were"}, {"start": 350.78000000000003, "end": 354.84000000000003, "text": " able to derive for cars on the highway. And the way to do this is because you are offline"}, {"start": 354.84, "end": 358.23999999999995, "text": " and you are trying to just annotate a club, you have a large number of benefits that you"}, {"start": 358.23999999999995, "end": 362.2, "text": " don't typically have if you're at test time under strict latency requirements in the car."}, {"start": 362.2, "end": 365.67999999999995, "text": " So you can take your time to fully figure out exactly all the objects in your data."}, {"start": 365.67999999999995, "end": 368.96, "text": " You can use neural networks that are extremely heavy. They are not deployable for various"}, {"start": 368.96, "end": 371.44, "text": " reasons. You can use benefit of hindsight because you know the future of not just the"}, {"start": 371.44, "end": 374.96, "text": " past. You can use all kinds of expensive offline optimization and tracking techniques. You can"}, {"start": 374.96, "end": 378.76, "text": " use extra sensors. In this case, for example, actually radar was one of the sensors that"}, {"start": 378.76, "end": 381.46, "text": " we use for the auto labeling. But there's actually a massive difference between using"}, {"start": 381.46, "end": 383.62, "text": " radar at test time and using it in the offline track."}, {"start": 383.62, "end": 388.24, "text": " So the point here is that if you record data, and you're trying to figure out at inference"}, {"start": 388.24, "end": 393.36, "text": " time, like while you're driving, what's happening, it's a lot harder than if you have the same"}, {"start": 393.36, "end": 398.04, "text": " data, but kind of at home in the lab. So what you want to do is you want to drive around"}, {"start": 398.04, "end": 404.32, "text": " and just record not even not predict or anything, just record data record from all your sensors,"}, {"start": 404.32, "end": 408.44, "text": " you can even stick expensive sensors on the cars where you collect the data. And then"}, {"start": 408.44, "end": 413.92, "text": " you take all that data, and you use the biggest heaviest processors, you have to figure out"}, {"start": 413.92, "end": 418.84, "text": " what actually happened during that time. What he mentions here is the benefit of hindsight,"}, {"start": 418.84, "end": 423.88, "text": " which means that if you're in a car and you're driving, and all of a sudden something obscures"}, {"start": 423.88, "end": 430.24, "text": " your vision, you will be sort of lost because all you have, okay, you can maybe guess that"}, {"start": 430.24, "end": 435.14, "text": " a car in front of you is still there. But who knows, they might turn or something. Now,"}, {"start": 435.14, "end": 440.2, "text": " if you record the whole video sequence, you're able to see what happens beyond the obstruction"}, {"start": 440.2, "end": 445.08, "text": " of vision. And if you see the car is still there, you can make a good inference that"}, {"start": 445.08, "end": 449.96, "text": " the car was actually there the whole time. And therefore you can annotate that data with"}, {"start": 449.96, "end": 454.91999999999996, "text": " a label saying, hey, that car was there the whole time, you can also do active learning"}, {"start": 454.91999999999996, "end": 460.26, "text": " and shell out to actual human annotators what you're not sure about. So this benefit of"}, {"start": 460.26, "end": 464.65999999999997, "text": " hindsight is really important here when you're under the time constraint of not being able"}, {"start": 464.66, "end": 469.52000000000004, "text": " to see into the future, as well as the latency constraint. And you have to have like an efficient"}, {"start": 469.52000000000004, "end": 474.44, "text": " neural network in the lab, you don't have any of this the method here, if you're developing"}, {"start": 474.44, "end": 479.24, "text": " something real time, I mean, this might seem obvious to you, I found it to be pretty cool."}, {"start": 479.24, "end": 485.12, "text": " Yes, record, then figure out what happened, then use that as a labeled data set. So here's"}, {"start": 485.12, "end": 490.8, "text": " an example of how such a persistent track would look like after the neural network has"}, {"start": 490.8, "end": 494.8, "text": " been trained on data like this. Here's some examples of really tricky scenarios. I don't"}, {"start": 494.8, "end": 498.40000000000003, "text": " actually know exactly what this is. But basically, this car draws a bunch of debris on us, and"}, {"start": 498.40000000000003, "end": 502.24, "text": " we maintain a consistent track for the label. And of course, if you have millions of labels"}, {"start": 502.24, "end": 506.04, "text": " like this, the neural net, if it's a powerful enough neural net, will actually end up learning"}, {"start": 506.04, "end": 509.52, "text": " to persist these tracks in these kinds of scenarios. Here's another example. There's"}, {"start": 509.52, "end": 513.08, "text": " a car in front of us, I actually am not 100% sure what happens in this case. But as you'll"}, {"start": 513.08, "end": 517.6800000000001, "text": " see, there's some kind of a dust cloud that develops here and briefly occludes the car."}, {"start": 517.68, "end": 521.8, "text": " But in the auto labeling tool, we are able to persist this track because we saw it before"}, {"start": 521.8, "end": 525.8, "text": " and we saw it after so we can actually stitch it up and use it as a training set for the"}, {"start": 525.8, "end": 532.16, "text": " neural. So that's how they get clean labels in an automatic or semi automatic way. But"}, {"start": 532.16, "end": 537.2399999999999, "text": " they still need to get a lot of data from kind of edge cases because most of driving"}, {"start": 537.2399999999999, "end": 543.3599999999999, "text": " is quite uneventful, straight driving and was done 40 years ago or something like this."}, {"start": 543.36, "end": 549.4, "text": " I think Schmidhuber in GTC 21 talk talked about autonomous cars on highways on controlled"}, {"start": 549.4, "end": 554.88, "text": " stretches of highways super duper early already. So what we really need to collect is edge"}, {"start": 554.88, "end": 560.6800000000001, "text": " cases. And for collecting these edge cases, Tesla has developed these what they call triggers."}, {"start": 560.6800000000001, "end": 566.36, "text": " So these are kind of hand programmed rules of what data should go into the annotation"}, {"start": 566.36, "end": 570.72, "text": " pipeline. So imagine if all these cars driving around not only the people with full self"}, {"start": 570.72, "end": 576.1600000000001, "text": " driving, but the detection, the actual recording of data is activated in all the Tesla cars"}, {"start": 576.1600000000001, "end": 580.78, "text": " driving around, they all send that data back to the server. Of course, that's way too much"}, {"start": 580.78, "end": 586.76, "text": " data. And also, it's very unbalanced in terms of how many critical situations are in there."}, {"start": 586.76, "end": 591.64, "text": " Again, most of it will be sort of straight road, empty, just drive straight. So what"}, {"start": 591.64, "end": 596.72, "text": " they do is they filter this data for these trigger events. Now these trigger events can"}, {"start": 596.72, "end": 601.9200000000001, "text": " be as simple as whenever the radar and the vision mismatch. So whenever they disagree"}, {"start": 601.9200000000001, "end": 606.62, "text": " on something, that's an interesting example. But you know, it goes into very detailed such"}, {"start": 606.62, "end": 612.62, "text": " as we detect breaking lights, but the acceleration is positive. So with these triggers, they're"}, {"start": 612.62, "end": 618.6800000000001, "text": " able to source a diverse set of training samples and edge cases where the neural network can"}, {"start": 618.6800000000001, "end": 623.5600000000001, "text": " learn the tricky situations rather than just the long stretches of road. So I think it's"}, {"start": 623.56, "end": 629.3599999999999, "text": " safe to say that a good mark of quality on these systems is going to be how well these"}, {"start": 629.3599999999999, "end": 635.06, "text": " triggers are maintained, like how well do they represent the full driving experience"}, {"start": 635.06, "end": 640.9, "text": " of the end users of the cars. But so far from the results we got, it seems like they cover"}, {"start": 640.9, "end": 645.16, "text": " the road situations fairly well. And all of them are iteration and you're looking at what's"}, {"start": 645.16, "end": 649.0799999999999, "text": " coming back, you're tuning your trigger and you're sourcing data from all these scenarios."}, {"start": 649.0799999999999, "end": 651.76, "text": " Basically over the last four months, we've done quite extensive data engine, we've ended"}, {"start": 651.76, "end": 655.52, "text": " up doing seven chatter modes and seven loops around this data engine here, where on the"}, {"start": 655.52, "end": 658.8, "text": " top right is where you begin, you have some seed data set, you train your neural network"}, {"start": 658.8, "end": 662.3199999999999, "text": " on your data set, and you deploy the neural network in the customer cars in shadow mode."}, {"start": 662.3199999999999, "end": 665.8, "text": " And the network is silently making prediction. By the way, if you if you like squint really"}, {"start": 665.8, "end": 671.56, "text": " hard, I don't know if this is just a depiction of a neural network, or if this is the actual"}, {"start": 671.56, "end": 676.98, "text": " architecture they're using, I don't think so. But there is like a stride of six in there"}, {"start": 676.98, "end": 682.5, "text": " and max pooling, you know, just just noting that for no particular reason. And then you"}, {"start": 682.5, "end": 685.52, "text": " have to have some mechanisms for sourcing inaccuracies of the neural network, you're"}, {"start": 685.52, "end": 688.24, "text": " just looking at its predictions, and then you're using one of these triggers, you're"}, {"start": 688.24, "end": 691.6, "text": " getting these scenarios where the network is probably misbehaving, some of those clips"}, {"start": 691.6, "end": 694.64, "text": " end up going to unit tests to make sure that we even if we're failing right now, we make"}, {"start": 694.64, "end": 698.04, "text": " sure we pass later. And in addition, those examples are being auto labeled and incorporated"}, {"start": 698.04, "end": 701.4, "text": " into a training set. And then as a synchronous process, we're also always data cleaning the"}, {"start": 701.4, "end": 704.9200000000001, "text": " current training set. So we spend this loop over and over again, until the network basically"}, {"start": 704.92, "end": 708.1999999999999, "text": " becomes incredibly good. So in total, we've done seven rounds of shadow mode for this"}, {"start": 708.1999999999999, "end": 714.16, "text": " release. So shadow mode is what they call when they let the predictions run, but they"}, {"start": 714.16, "end": 719.4, "text": " don't hook them up to the control. So you're driving yourself, but the system predicts"}, {"start": 719.4, "end": 724.28, "text": " all the time. And whenever one of these trigger happens, that's an interesting data point"}, {"start": 724.28, "end": 727.7199999999999, "text": " that is going to send back to the server. Actually, let's be honest, it's probably going"}, {"start": 727.7199999999999, "end": 733.1999999999999, "text": " to send everything back to the server. So the data set they come up with is 1.5 petabytes"}, {"start": 733.2, "end": 737.5200000000001, "text": " of data. So that's crazy. So next is going to go into the architecture of the neural"}, {"start": 737.5200000000001, "end": 744.1600000000001, "text": " net. And this is also fairly interesting and not entirely standard on the top. All of them"}, {"start": 744.1600000000001, "end": 748.48, "text": " are processed by an image extractor, the layout of the synthetic visual cortex in order to"}, {"start": 748.48, "end": 751.72, "text": " efficiently process this information, our architecture roughly looks like this, we have"}, {"start": 751.72, "end": 754.88, "text": " these images coming from multiple cameras on the top, all of them are processed by an"}, {"start": 754.88, "end": 758.12, "text": " image extractor, like a backbone, like think resonant kind of style, then there's a multi"}, {"start": 758.12, "end": 761.88, "text": " cam fusion that uses the information from all the eight to use. And this is a kind of"}, {"start": 761.88, "end": 765.64, "text": " a transformer that we use to fuse this information. And then we fuse information first across"}, {"start": 765.64, "end": 769.64, "text": " all the cameras, and then across all of time. And that is also done either by transformer"}, {"start": 769.64, "end": 772.88, "text": " by recurrent neural network, or just by three dimensional convolutions. We've experimented"}, {"start": 772.88, "end": 776.0, "text": " with a lot of kind of fusion strategies here to get this to work really well. And then"}, {"start": 776.0, "end": 779.34, "text": " what we have afterwards, after the fusion is done is we have this branching structure"}, {"start": 779.34, "end": 783.04, "text": " that doesn't just consist of heads, but actually, we've expanded this over the last few last"}, {"start": 783.04, "end": 786.68, "text": " year or so where you now have heads that branch into trunks that branch into terminals. So"}, {"start": 786.68, "end": 789.6, "text": " there's a lot of branching structure. And the reason you want this branching structure"}, {"start": 789.6, "end": 792.48, "text": " is because there's a huge amount of outlets that you're interested in. And you can't afford"}, {"start": 792.48, "end": 795.12, "text": " to have a single neural network for every one of the individual outlets, you have to"}, {"start": 795.12, "end": 797.2, "text": " of course, amortize the forward pass."}, {"start": 797.2, "end": 801.6800000000001, "text": " So this is pretty interesting. The top part here, what they call the backbone is pretty"}, {"start": 801.6800000000001, "end": 806.2, "text": " standard. If you have a video, especially with multiple cameras, you want to extract"}, {"start": 806.2, "end": 811.72, "text": " information from each frame of each camera sort of individually, then you want to fuse"}, {"start": 811.72, "end": 816.12, "text": " that information across all the cameras for a single time step. And then you want to fuse"}, {"start": 816.12, "end": 821.68, "text": " that information with the information of all the other time steps. So so far, so good."}, {"start": 821.68, "end": 826.1, "text": " That sort of gives you a representation of what happens in these frames in these cameras"}, {"start": 826.1, "end": 832.5600000000001, "text": " during that stretch of time. However, after that, usually, even if you have multiple predictions,"}, {"start": 832.5600000000001, "end": 836.2, "text": " what you would do is you would sort of have like one prediction head on top of that backbone."}, {"start": 836.2, "end": 843.08, "text": " However, since they are in a car and have to decide real fast, it's not really feasible"}, {"start": 843.08, "end": 848.2, "text": " to have sort of these different columns for each of the prediction tasks. Because as he"}, {"start": 848.2, "end": 853.12, "text": " says, they're interested in a lot of different signals, think depth prediction, which means"}, {"start": 853.12, "end": 858.34, "text": " that for every pixel, you have to provide a depth estimation, think tracks of other"}, {"start": 858.34, "end": 865.12, "text": " cars, think pedestrians think streetlights think okay, where are the lanes at or navigation"}, {"start": 865.12, "end": 870.72, "text": " in general. So all these signals are things to predict. And it's not good enough to have"}, {"start": 870.72, "end": 874.9200000000001, "text": " like a separate head for each of the predictions. So what they do is they have as you call these"}, {"start": 874.9200000000001, "end": 881.36, "text": " branching structures where there are multiple heads, yes. And within these multiple heads,"}, {"start": 881.36, "end": 885.28, "text": " there are what they call trunks. And within the trunks, there are the individual like"}, {"start": 885.28, "end": 889.5600000000001, "text": " little what they call terminals. Essentially, it's a hierarchical prediction, I'm going"}, {"start": 889.5600000000001, "end": 894.86, "text": " to guess that the tasks that go together, sort of are grouped together. So maybe one"}, {"start": 894.86, "end": 900.5600000000001, "text": " head is for all the pixel prediction tasks, and another head is more for the classification"}, {"start": 900.56, "end": 905.9799999999999, "text": " tasks. And then within one head, you have a trunk that deals more with like object classification"}, {"start": 905.9799999999999, "end": 910.8, "text": " and another trunk that deals more with like navigation classification. And the individual"}, {"start": 910.8, "end": 917.16, "text": " terminals then do the actual tasks. This is a pretty cool way of getting a highly performant"}, {"start": 917.16, "end": 922.76, "text": " many output network all together such that its size and computational speed are still"}, {"start": 922.76, "end": 926.8, "text": " maintained. The other nice benefit of the branching structure is that it decouples at"}, {"start": 926.8, "end": 930.8, "text": " the terminals, it decouples all these signals. So if I am someone working on velocity for"}, {"start": 930.8, "end": 933.9599999999999, "text": " a particular object type or something like that, I have a small piece of neural network"}, {"start": 933.9599999999999, "end": 937.3399999999999, "text": " that I can actually fine tune without touching any of the other signals. And so I can work"}, {"start": 937.3399999999999, "end": 940.52, "text": " in isolation to some extent and actually get something to work pretty well. And then once"}, {"start": 940.52, "end": 944.0, "text": " in a while, so basically, the iteration scheme is that a lot of people are fine tuning. And"}, {"start": 944.0, "end": 948.62, "text": " once you just got to imagine the ML ops behind this, it's like, Hey, where do you deploy"}, {"start": 948.62, "end": 955.0, "text": " your models? I do it on the Kubernetes, I have ML flow. Oh, no, I use the TensorFlow"}, {"start": 955.0, "end": 964.08, "text": " extended. Yeah, it's pretty cool. What do you do? car, I deploy on car. So next, he's"}, {"start": 964.08, "end": 969.8, "text": " going into this in house supercomputer that they built or are building. And this is a"}, {"start": 969.8, "end": 974.28, "text": " massive thing. Absolutely massive. He says that in terms of flops, it's something like"}, {"start": 974.28, "end": 980.52, "text": " the fifth biggest computer in the world. Its storage speed is incredible. So I'm pretty"}, {"start": 980.52, "end": 985.8, "text": " sure you could even actually render Far Cry two on this thing, maybe, but in total, it"}, {"start": 985.8, "end": 995.3199999999999, "text": " has 5760 GPU is not any GPUs, the most expensive a 180 gigabyte GPUs, it would be interesting"}, {"start": 995.3199999999999, "end": 1000.4, "text": " to see what kind of algorithms they use on top of this to actually do the distributed"}, {"start": 1000.4, "end": 1005.72, "text": " training or whether it's all just kind of simple data parallelism, aggregating gradients"}, {"start": 1005.72, "end": 1010.28, "text": " and so on. Of course, they have super fast interconnect super fast storage, super fast"}, {"start": 1010.28, "end": 1015.8399999999999, "text": " everything and it looks sweet. Like is this a stock photo of a server room? Or is this"}, {"start": 1015.8399999999999, "end": 1020.16, "text": " the actual server room? This effort to basically is incredibly vertically integrated in the"}, {"start": 1020.16, "end": 1024.36, "text": " AI team. So as I showed you, we own the vehicle and the sensing and resource our own data"}, {"start": 1024.36, "end": 1028.2, "text": " and we annotate our own data and we train our on prem cluster and then we deploy all"}, {"start": 1028.2, "end": 1031.84, "text": " of the neural networks that we train on our in house developed chip. So we have the FSD"}, {"start": 1031.84, "end": 1036.8799999999999, "text": " computer here that has two SOCs as the chips here and they have our own custom and view"}, {"start": 1036.88, "end": 1041.8000000000002, "text": " neural processing unit here at roughly 36 times each. So these chips are specifically"}, {"start": 1041.8000000000002, "end": 1044.8000000000002, "text": " designed for the neural networks that we want to run for."}, {"start": 1044.8000000000002, "end": 1049.48, "text": " Yeah, I mean, this is the dream, right? If you're an AI professional, only the whole"}, {"start": 1049.48, "end": 1055.68, "text": " pipeline is going to boost your productivity by so much. You're not bound by the constraint"}, {"start": 1055.68, "end": 1061.4, "text": " of anything other than the limits on the final system, which is a car so fairly difficult."}, {"start": 1061.4, "end": 1065.4, "text": " But in between of that you have control over everything you have control over how the data"}, {"start": 1065.4, "end": 1070.5600000000002, "text": " is collected annotated, you have control over where it is deployed to on what architecture"}, {"start": 1070.5600000000002, "end": 1074.44, "text": " of chip because you make the chip. So I guess the lesson is if you're looking to change"}, {"start": 1074.44, "end": 1079.88, "text": " the world, you better own a good chunk of it. So that's going to show some examples"}, {"start": 1079.88, "end": 1085.72, "text": " of what this new vision only stack could do. Remember, they used to do fusion of sensors,"}, {"start": 1085.72, "end": 1090.2, "text": " which means they essentially have radar, they have vision, maybe some other sensors, and"}, {"start": 1090.2, "end": 1095.2800000000002, "text": " they try to integrate this information from all of the sensors, they compare this to the"}, {"start": 1095.28, "end": 1100.04, "text": " new vision based system. Now check out what happens in terms of the depth and velocity"}, {"start": 1100.04, "end": 1103.24, "text": " predictions that we're able to achieve by putting all these pieces together and training"}, {"start": 1103.24, "end": 1106.68, "text": " these networks at scale. So the first example here, I have a video where this is on track"}, {"start": 1106.68, "end": 1110.3999999999999, "text": " testing. So this is an engineering car, and we asked it to slam on the brakes as hard"}, {"start": 1110.3999999999999, "end": 1113.58, "text": " as it possibly can. So this is a very harsh braking here in front of us, even though it"}, {"start": 1113.58, "end": 1116.6, "text": " doesn't look like that in the videos is very harsh braking. So what you can see on the"}, {"start": 1116.6, "end": 1120.44, "text": " right here is you can see the outputs from the legacy stack, which had radar vision fusion"}, {"start": 1120.44, "end": 1124.52, "text": " and from the new stack, which is vision alone in blue. So in the orange legacy stack, you"}, {"start": 1124.52, "end": 1128.68, "text": " can actually see these track drops here when the car was breaking really harshly. And basically"}, {"start": 1128.68, "end": 1132.28, "text": " the issue is that the braking was so harsh that the radar stack that we have actually"}, {"start": 1132.28, "end": 1135.84, "text": " ended up not associating car and dropping the track and then reinitializing it all the"}, {"start": 1135.84, "end": 1139.76, "text": " time. And so it's as if the vehicle disappeared and reappeared like six times during the period"}, {"start": 1139.76, "end": 1142.8799999999999, "text": " of this braking. And so this created a bunch of artifacts here, but we see that the new"}, {"start": 1142.8799999999999, "end": 1147.04, "text": " stack in blue is actually not subject to this behavior at all. It just gives a clean signal."}, {"start": 1147.04, "end": 1150.76, "text": " In fact, here, there's no smoothing, I believe, on the blue signal here. This is the raw depth"}, {"start": 1150.76, "end": 1153.96, "text": " and velocity that comes out from the neural net, the final neural net that we released"}, {"start": 1153.96, "end": 1157.0, "text": " with about three weeks ago. And you can see that it's fairly smooth here. And of course,"}, {"start": 1157.0, "end": 1160.28, "text": " you could go into the radar stack and you could, you know, adjust the hyper parameters"}, {"start": 1160.28, "end": 1163.76, "text": " of the tracker, like why is it dropping tracks and so on. But then you are spending engineering"}, {"start": 1163.76, "end": 1167.76, "text": " efforts and focus on a stack that is like not really barking up the right tree. And"}, {"start": 1167.76, "end": 1171.1200000000001, "text": " so it's better to again, focus on the vision and make it work really well. And we see that"}, {"start": 1171.1200000000001, "end": 1173.78, "text": " it is much more robust when you train it at scale."}, {"start": 1173.78, "end": 1178.72, "text": " So there you have it proof by one example that the new thing works better. Isn't that"}, {"start": 1178.72, "end": 1185.28, "text": " every CVPR paper ever. But no, in any case, I can totally believe that the new stack even"}, {"start": 1185.28, "end": 1191.08, "text": " though it drops a bunch of the sensors is better. Because ultimately, if your one sensor"}, {"start": 1191.08, "end": 1196.56, "text": " if vision is so performant that in every single disagreement, you go with the vision thing,"}, {"start": 1196.56, "end": 1200.76, "text": " then why do you have the other sensors at all the thing in front of it is just kind"}, {"start": 1200.76, "end": 1205.6000000000001, "text": " of breaking too fast. So the radar kind of loses it and then regains it and loses it"}, {"start": 1205.6, "end": 1211.1799999999998, "text": " and regains it. Now, I have no idea how radar works. So I'm speaking from complete ignorance"}, {"start": 1211.1799999999998, "end": 1215.8799999999999, "text": " right here. But what I'm going to guess as far as I understand it, is that radar just"}, {"start": 1215.8799999999999, "end": 1220.12, "text": " kind of gives you the velocities of stuff in front of you. And then there is a tracking"}, {"start": 1220.12, "end": 1225.24, "text": " algorithm on top of radar that tries to figure out which stuff is the same stuff. And this"}, {"start": 1225.24, "end": 1231.08, "text": " is very much what they do in this auto labeling where they have sort of a track on something,"}, {"start": 1231.08, "end": 1235.24, "text": " right, and then they use hindsight, and then they have a tracking algorithm that decides"}, {"start": 1235.24, "end": 1239.08, "text": " which things are the same, even though we don't see them all the time. And here you"}, {"start": 1239.08, "end": 1244.28, "text": " can clearly see the benefit of shifting this from inference time, which is what you have"}, {"start": 1244.28, "end": 1249.88, "text": " to do with radar to the training time, which is what you can do with vision. So you can"}, {"start": 1249.88, "end": 1255.36, "text": " teach the vision system to sort of do this persistent tracking, whereas the radar system,"}, {"start": 1255.36, "end": 1259.6200000000001, "text": " you have to hand tune it to do this in real time. Now it makes the point that of course,"}, {"start": 1259.6200000000001, "end": 1263.56, "text": " you could go into the radar system, change the hyper parameters, but then he says why"}, {"start": 1263.56, "end": 1268.26, "text": " bark up the wrong tree? Why waste time on a stack that isn't functioning? It's a bit"}, {"start": 1268.26, "end": 1272.8, "text": " of a chicken and an egg problem, right? If you were to put as much effort into the radar"}, {"start": 1272.8, "end": 1278.2, "text": " stack as you were into the vision system, I'm going to guess that these results would"}, {"start": 1278.2, "end": 1284.6, "text": " go away, and that is able to keep up maybe, but the arguments for going vision only is"}, {"start": 1284.6, "end": 1290.0, "text": " a strong one. And I don't doubt that it is probably a good way forward. And basically"}, {"start": 1290.0, "end": 1293.3999999999999, "text": " what's happening here is that the radar is very trigger happy and it sees all these false"}, {"start": 1293.4, "end": 1296.52, "text": " stationary objects everywhere. Like everything that like sticks out is a stationary target"}, {"start": 1296.52, "end": 1299.76, "text": " and radar by itself doesn't know what actually is a stationary car and what isn't. So it's"}, {"start": 1299.76, "end": 1303.0400000000002, "text": " waiting for vision to associate with it and vision, if it's not held up to a high enough"}, {"start": 1303.0400000000002, "end": 1306.68, "text": " bar is noisy and contributes error and the sensor fusion stack just kind of like picks"}, {"start": 1306.68, "end": 1310.0400000000002, "text": " it up too late. And so again, you could fix all that, even though it's a very gross system"}, {"start": 1310.0400000000002, "end": 1313.44, "text": " with a lot of if statements and so on, because the sensor fusion is complicated because the"}, {"start": 1313.44, "end": 1317.1200000000001, "text": " error modes for vision and radar are slightly are quite different. But here, when we just"}, {"start": 1317.1200000000001, "end": 1320.48, "text": " work with vision alone and we take out the radar, vision recognizes this object very"}, {"start": 1320.48, "end": 1323.4, "text": " early gives the correct depth and velocity and there's no issues. So we actually get"}, {"start": 1323.4, "end": 1327.16, "text": " an initial slowdown much earlier and really like simplify the stack a lot."}, {"start": 1327.16, "end": 1331.6, "text": " Yeah. So here you can see the same failure mode in vision that it kind of gets a track"}, {"start": 1331.6, "end": 1335.64, "text": " but doesn't but get a track but doesn't. The important part is that once you get closer"}, {"start": 1335.64, "end": 1340.34, "text": " to the object, it is fairly consistent, right? As you can see right here, the vision stack"}, {"start": 1340.34, "end": 1346.44, "text": " recognizes this truck on the side much earlier than the radar stack did. Now again, this"}, {"start": 1346.44, "end": 1350.44, "text": " might just be a function of the hyper parameters used, I'm sure you could just lower the threshold"}, {"start": 1350.44, "end": 1355.64, "text": " for the radar, but you'd run into different problems. During the Q&A, he makes a good"}, {"start": 1355.64, "end": 1360.92, "text": " point in that yes, other sensors would be nice to have, but just the pure economics"}, {"start": 1360.92, "end": 1367.9, "text": " speak in favor of vision to like we develop cameras with much more rigor as a society"}, {"start": 1367.9, "end": 1373.48, "text": " than we do radar systems. And therefore, the camera sensors are just so much better nowadays"}, {"start": 1373.48, "end": 1378.24, "text": " and cheaper. So you can afford to build many of them into all kinds of things and collect"}, {"start": 1378.24, "end": 1383.48, "text": " data and make your systems better through that then to put kind of a lidar on top of"}, {"start": 1383.48, "end": 1389.4, "text": " the car and having to sort of fuse those signals with the vision signals, especially when they're"}, {"start": 1389.4, "end": 1394.36, "text": " in conflict with one another. So if you ask me, I'm a fan. I like what I see here, even"}, {"start": 1394.36, "end": 1397.92, "text": " though I know it's kind of an ad. I don't own a Tesla, but I think it's still pretty"}, {"start": 1397.92, "end": 1402.8, "text": " cool. In the end, he talks a bit about what they do to validate this data, and how they"}, {"start": 1402.8, "end": 1409.1599999999999, "text": " roll it out and gives a bunch of more examples of tracking. And there's a Q&A at the end."}, {"start": 1409.1599999999999, "end": 1414.6399999999999, "text": " So if you are interested in that, I absolutely welcome you to go watch the entire talk. It"}, {"start": 1414.6399999999999, "end": 1419.6, "text": " is on YouTube. And that was it for me. I hope you enjoyed this and I'll see you next time."}, {"start": 1419.6, "end": 1437.6, "text": " Ciao!"}]