<story><title>Uber’s self-driving car could not detect pedestrians outside of a crosswalk</title><url>https://www.theregister.co.uk/2019/11/06/uber_self_driving_car_death/</url></story><parent_chain><item><author>Strilanc</author><text>&amp;gt; &lt;i&gt;5.6 seconds before impact, it classified her as a vehicle. Then it changed its mind to “other,” then to vehicle again, back to “other,” then to bicycle, then to “other” again, and finally back to bicycle.&lt;/i&gt;&lt;p&gt;System can&amp;#x27;t decide what&amp;#x27;s happening.&lt;p&gt;&amp;gt; &lt;i&gt;It wasn’t until 1.2 seconds before the impact that the system recognized that the SUV was going to hit Herzberg&lt;/i&gt;&lt;p&gt;System is too slow to realize something serious is happening.&lt;p&gt;&amp;gt; &lt;i&gt;That triggered what Uber called “action suppression,” in which the system held off braking for one second&lt;/i&gt;&lt;p&gt;A hardcoded 1 second delay during a potential emergency situation. Horrifying.&lt;p&gt;I bet they added it because the system kept randomly thinking something serious was going to happen for a few milliseconds when everything was going fine. If you ever find yourself doing that for a safety critical piece of software, you should stop and reconsider what you are doing. This is a hacky patch over a serious underlying classification issue. You need to fix the underlying problem, not hackily patch over it.&lt;p&gt;How is this not the title of the story? This is so much worse than the &amp;quot;it couldn&amp;#x27;t see her as a person, only as a bicycle&amp;quot;. At least the car would still try to avoid a bicycle, in principle, instead of blindly gliding into it while hoping for the best.&lt;p&gt;&amp;gt; &lt;i&gt;with 0.2 seconds left before impact, the car sounded an audio alarm, and Vasquez took the steering wheel, disengaging the autonomous system. Nearly a full second after striking Herzberg, Vasquez hit the brakes.&lt;/i&gt;&lt;p&gt;And then top it off with systemic issues around the backup driver not actually being ready to react.</text></item></parent_chain><comment><author>chooseaname</author><text>&amp;gt;&amp;gt; 5.6 seconds before impact, it classified her as a vehicle. Then it changed its mind to “other,” then to vehicle again, back to “other,” then to bicycle, then to “other” again, and finally back to bicycle.&lt;p&gt;The system should have started applying brake at this point. If a 3500lb vehicle can&amp;#x27;t decide what it is about to impact, it needs to slow down (to a stop if necessary).&lt;p&gt;&amp;gt; That triggered what Uber called “action suppression,” in which the system held off braking for one second&lt;p&gt;This is borderline criminal negligence.&lt;p&gt;&amp;gt; with 0.2 seconds left before impact, the car sounded an audio alarm, and Vasquez took the steering wheel, disengaging the autonomous system. Nearly a full second after striking Herzberg, Vasquez hit the brakes.&lt;p&gt;Why were there no alarms going off at 5.6 seconds when the vehicle was confused!!!??&lt;p&gt;SMH.  This is just ... I&amp;#x27;m flabbergasted.</text></comment>
<story><title>Uber’s self-driving car could not detect pedestrians outside of a crosswalk</title><url>https://www.theregister.co.uk/2019/11/06/uber_self_driving_car_death/</url></story><parent_chain><item><author>Strilanc</author><text>&amp;gt; &lt;i&gt;5.6 seconds before impact, it classified her as a vehicle. Then it changed its mind to “other,” then to vehicle again, back to “other,” then to bicycle, then to “other” again, and finally back to bicycle.&lt;/i&gt;&lt;p&gt;System can&amp;#x27;t decide what&amp;#x27;s happening.&lt;p&gt;&amp;gt; &lt;i&gt;It wasn’t until 1.2 seconds before the impact that the system recognized that the SUV was going to hit Herzberg&lt;/i&gt;&lt;p&gt;System is too slow to realize something serious is happening.&lt;p&gt;&amp;gt; &lt;i&gt;That triggered what Uber called “action suppression,” in which the system held off braking for one second&lt;/i&gt;&lt;p&gt;A hardcoded 1 second delay during a potential emergency situation. Horrifying.&lt;p&gt;I bet they added it because the system kept randomly thinking something serious was going to happen for a few milliseconds when everything was going fine. If you ever find yourself doing that for a safety critical piece of software, you should stop and reconsider what you are doing. This is a hacky patch over a serious underlying classification issue. You need to fix the underlying problem, not hackily patch over it.&lt;p&gt;How is this not the title of the story? This is so much worse than the &amp;quot;it couldn&amp;#x27;t see her as a person, only as a bicycle&amp;quot;. At least the car would still try to avoid a bicycle, in principle, instead of blindly gliding into it while hoping for the best.&lt;p&gt;&amp;gt; &lt;i&gt;with 0.2 seconds left before impact, the car sounded an audio alarm, and Vasquez took the steering wheel, disengaging the autonomous system. Nearly a full second after striking Herzberg, Vasquez hit the brakes.&lt;/i&gt;&lt;p&gt;And then top it off with systemic issues around the backup driver not actually being ready to react.</text></item></parent_chain><comment><author>Strilanc</author><text>To rant a bit more about this one second delay thing.&lt;p&gt;This reeks of a type of thinking where you are relying on other parts of the system to compensate. You might expect to hear things like &amp;quot;it&amp;#x27;s okay, the safety driver will catch it&amp;quot;. Speaking for myself personally, this type of thinking comes very naturally. I like to come up with little proofs that a problem is handled by one part of a program, and then forget about that problem. But in my experience (which does not involve writing anything safety critical) this strategy kinda sucks at getting things right. Dependencies change, assumptions become invalid, holes in your intuitive proof become apparent, etc, etc, etc, and the thing falls over.&lt;p&gt;If you are designing a safety critical system, something you &lt;i&gt;really&lt;/i&gt; want to work, I don&amp;#x27;t think you should be thinking in the mode where each problem is handled by one part of the system. You need to be thinking in terms of defense in depth. When something goes wrong, &lt;i&gt;many&lt;/i&gt; parts of the system should &lt;i&gt;all&lt;/i&gt; be able to detect and correct the problem. And then when something bad does come up, and 9 out of 10 of those defensive layers each individually were sufficient to save the day so there was no disaster, you should go figure out what the hell went wrong in the tenth.</text></comment>