<story><title>Uber Self-Driving Car That Struck Pedestrian Wasn’t Set to Stop in an Emergency</title><url>https://www.ntsb.gov/investigations/AccidentReports/Reports/HWY18MH010-prelim.pdf</url></story><parent_chain><item><author>JorgeGT</author><text>Direct link to NTSB preliminary report: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.ntsb.gov&amp;#x2F;investigations&amp;#x2F;AccidentReports&amp;#x2F;Reports&amp;#x2F;HWY18MH010-prelim.pdf&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.ntsb.gov&amp;#x2F;investigations&amp;#x2F;AccidentReports&amp;#x2F;Reports&amp;#x2F;...&lt;/a&gt;&lt;p&gt;&lt;i&gt;According to data obtained from the self-driving system, the system first registered radar and LIDAR
observations of the pedestrian about 6 seconds before impact, when the vehicle was traveling at 43 mph.
As the vehicle and pedestrian paths converged, the self-driving system software classified the pedestrian
as an unknown object, as a vehicle, and then as a bicycle with varying expectations of future travel path.
At 1.3 seconds before impact, the self-driving system determined that an emergency braking maneuver
was needed to mitigate a collision (see figure 2).
2 According to Uber, emergency braking maneuvers are
not enabled while the vehicle is under computer control, to reduce the potential for erratic vehicle
behavior. The vehicle operator is relied on to intervene and take action. The system is not designed to
alert the operator.&lt;/i&gt;</text></item></parent_chain><comment><author>nkoren</author><text>I worked on the autonomous pod system at Heathrow airport[1]. We used a very conservative control methodology; essentially the vehicle would remain stopped unless it received a positive &amp;quot;GO&amp;quot; signal from multiple independent sensor and control systems. The loss of &lt;i&gt;any&lt;/i&gt; &amp;quot;GO&amp;quot; signal would result in an emergency stop. It was very challenging to get those all of those &amp;quot;GO&amp;quot; indicators reliable enough to prevent false positives and constant emergency-braking.&lt;p&gt;The reason we were ultimately able to do this is because we were operating in a fully-segregated environment of our own design. We could be certain that every other vehicle in the system was something that should be fully under our control, so anything even slightly anomalous should be treated as a hazard situation.&lt;p&gt;There are a lot of limitations to this approach, but I&amp;#x27;m confident that it could carry literally billions of passengers without a fatality. It is &lt;i&gt;overwhelmingly&lt;/i&gt; safe.&lt;p&gt;Operating in a mixed environment is profoundly different. The control system logic is fully reversed: you must presume that it is safe to proceed unless a &amp;quot;STOP&amp;quot; signal received. And because the interpretation of image &amp;amp; LIDAR data is a rather... fuzzy... process, that &amp;quot;STOP&amp;quot; signal needs to have fairly liberal thresholds, otherwise your vehicle will not move.&lt;p&gt;Uber made a critical mistake in counting on a human-in-the-loop to suddenly take control of the vehicle (note: this is why Type 3 automation is something I&amp;#x27;m very dubious about), but it&amp;#x27;s important to understand that if you want autonomous vehicles to move through mixed-mode environments at the speeds which humans drive, then it is absolutely necessary for them to take a fuzzy, probabilistic approach to safety. This will inevitably result in fatalities -- almost certainly fewer than when humans drive, but plenty of fatalities nonetheless. The design of the overall system is system is inherently unsafe.&lt;p&gt;Do you find this unacceptable? If so, then then ultimately the only way to address this is through changing the design of the streets and&amp;#x2F;or our rules about how they are be used. These are fundamentally infrastructural issues. Merely swapping out vehicle control systems -- robot vs. human -- will be less revolutionary than many expect.&lt;p&gt;1: &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;www.ultraglobalprt.com&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;www.ultraglobalprt.com&amp;#x2F;&lt;/a&gt;</text></comment>
<story><title>Uber Self-Driving Car That Struck Pedestrian Wasn’t Set to Stop in an Emergency</title><url>https://www.ntsb.gov/investigations/AccidentReports/Reports/HWY18MH010-prelim.pdf</url></story><parent_chain><item><author>JorgeGT</author><text>Direct link to NTSB preliminary report: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.ntsb.gov&amp;#x2F;investigations&amp;#x2F;AccidentReports&amp;#x2F;Reports&amp;#x2F;HWY18MH010-prelim.pdf&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.ntsb.gov&amp;#x2F;investigations&amp;#x2F;AccidentReports&amp;#x2F;Reports&amp;#x2F;...&lt;/a&gt;&lt;p&gt;&lt;i&gt;According to data obtained from the self-driving system, the system first registered radar and LIDAR
observations of the pedestrian about 6 seconds before impact, when the vehicle was traveling at 43 mph.
As the vehicle and pedestrian paths converged, the self-driving system software classified the pedestrian
as an unknown object, as a vehicle, and then as a bicycle with varying expectations of future travel path.
At 1.3 seconds before impact, the self-driving system determined that an emergency braking maneuver
was needed to mitigate a collision (see figure 2).
2 According to Uber, emergency braking maneuvers are
not enabled while the vehicle is under computer control, to reduce the potential for erratic vehicle
behavior. The vehicle operator is relied on to intervene and take action. The system is not designed to
alert the operator.&lt;/i&gt;</text></item></parent_chain><comment><author>jacquesm</author><text>That&amp;#x27;s not just one error but a whole book of errors, and that last bit combined with the reliance on the operator to take action is criminal. (And if it isn&amp;#x27;t it should be.)&lt;p&gt;I hope that whoever was responsible for this piece of crap software loses a lot of sleep over it, and that Uber will admit that they have no business building safety critical software. Idiots.&lt;p&gt;For &lt;i&gt;6&lt;/i&gt; seconds the system had crucial information and failed to relay it, for 1.3 seconds the system knew an accident was going to happen and failed to act on that knowledge.&lt;p&gt;Drunk drivers suck, but this is much worse. This is the equivalent of plowing into a pedestrian with a vehicle while you&amp;#x27;re in full control of it because you are afraid that your perception of the world is so crappy that you will over-react to such situations often enough that the risk of killing someone you know is there is perceived as the lower one.&lt;p&gt;Not to mention all the errors in terms of process and oversight that allowed this p.o.s. software to be deployed in traffic.</text></comment>