<story><title>Delays in Boeing Max Return Began with Near-Crash in Simulator</title><url>https://www.bloomberg.com/news/articles/2019-11-08/delays-in-boeing-max-return-began-with-near-crash-in-simulator</url></story><parent_chain><item><author>zaroth</author><text>The can of worms has been officially opened, and this may turn out to be a case study in how bad software and outsourcing your centers of excellence can bankrupt a company.&lt;p&gt;If TFA is to believed, Boeing has embarked on a project to take what is currently a master-slave failover design and jury-rig it into a master-master real-time system. This is based on bad performance in the simulator—not because of MCAS—but while testing errant memory errors from a simulated gamma ray strike!&lt;p&gt;The obvious question of how can dual-master ever work when they disagree if there isn’t a 3rd source of truth to vote out the failure? This means the software running in both computers has to somehow agree which one is wrong, while one of them is in a potential failure state. This... isn’t how flight control systems are designed from first principals.&lt;p&gt;I’m not a flight system designer, but there must be hundreds of physical, electrical, and architectural considerations taken at every point in the design process which enable multi-master controllers with the ability to vote out a failure. Everything from the particular sensor suite, the number of sensors, the way they are wired, the way data is acquired and bused through the system, the timing and synchronization of the system clocks, the way that control outputs are calculated, queued, and ultimately issued to downstream controllers... none of the necessary pieces will be in place in a system which up until now makes you select a single master controller before embarking on your flight.&lt;p&gt;The only conclusion I can reach is that Boeing has lost its mind, and this project is absolutely doomed to fail. The architecture that Boeing has apparently committed itself to now is extremely difficult to design from first principals and a blank slate. I just don’t see how it’s something that can realistically be papered onto a legacy dual-computer system after the fact.&lt;p&gt;The part in TFA about “adding a wire” practically made me spit out my coffee. The sheer level of arrogance that Boeing management must have to think this would be possible - is just a classic example of an elitist MBA management group totally disconnected from any technical domain expertise. Boeing said they would have this ready for certification &lt;i&gt;by the end of this year&lt;/i&gt;?!</text></item></parent_chain><comment><author>drawkbox</author><text>Classic case of hitting the date but crunched out with technical debt and buggy software, a tell that management&amp;#x2F;finance&amp;#x2F;marketing is driving the decision over the engineer&amp;#x2F;designer&amp;#x2F;product people.&lt;p&gt;People forget a project was late if it succeeds, they never forget a project on time that failed because it was rushed, buggy and not polished production.&lt;p&gt;MBA&amp;#x2F;business and marketing need to go back to letting the product people and engineers create value, &lt;i&gt;then&lt;/i&gt; they can extract the value and sell it.  It is much easier to sell value of a good product, it basically sells itself. For some reason American business has completely forgotten the rule this millennia, product then marketing and selling.&lt;p&gt;You see this same thing happen at game companies when they put a date out there and it is driven by management and marketing over reality of engineering&amp;#x2F;product&amp;#x2F;design that makes a fun and solid game.&lt;p&gt;Boeing was greatly diminished in engineering&amp;#x2F;product excellence by the McDonnell Douglas management control takeover, everything newly produced since has the tell of this.</text></comment>
<story><title>Delays in Boeing Max Return Began with Near-Crash in Simulator</title><url>https://www.bloomberg.com/news/articles/2019-11-08/delays-in-boeing-max-return-began-with-near-crash-in-simulator</url></story><parent_chain><item><author>zaroth</author><text>The can of worms has been officially opened, and this may turn out to be a case study in how bad software and outsourcing your centers of excellence can bankrupt a company.&lt;p&gt;If TFA is to believed, Boeing has embarked on a project to take what is currently a master-slave failover design and jury-rig it into a master-master real-time system. This is based on bad performance in the simulator—not because of MCAS—but while testing errant memory errors from a simulated gamma ray strike!&lt;p&gt;The obvious question of how can dual-master ever work when they disagree if there isn’t a 3rd source of truth to vote out the failure? This means the software running in both computers has to somehow agree which one is wrong, while one of them is in a potential failure state. This... isn’t how flight control systems are designed from first principals.&lt;p&gt;I’m not a flight system designer, but there must be hundreds of physical, electrical, and architectural considerations taken at every point in the design process which enable multi-master controllers with the ability to vote out a failure. Everything from the particular sensor suite, the number of sensors, the way they are wired, the way data is acquired and bused through the system, the timing and synchronization of the system clocks, the way that control outputs are calculated, queued, and ultimately issued to downstream controllers... none of the necessary pieces will be in place in a system which up until now makes you select a single master controller before embarking on your flight.&lt;p&gt;The only conclusion I can reach is that Boeing has lost its mind, and this project is absolutely doomed to fail. The architecture that Boeing has apparently committed itself to now is extremely difficult to design from first principals and a blank slate. I just don’t see how it’s something that can realistically be papered onto a legacy dual-computer system after the fact.&lt;p&gt;The part in TFA about “adding a wire” practically made me spit out my coffee. The sheer level of arrogance that Boeing management must have to think this would be possible - is just a classic example of an elitist MBA management group totally disconnected from any technical domain expertise. Boeing said they would have this ready for certification &lt;i&gt;by the end of this year&lt;/i&gt;?!</text></item></parent_chain><comment><author>zbrozek</author><text>&amp;quot;Dual-dual&amp;quot; is a thing in this space. There are safety-intent microprocessors which have a couple of cores running a couple of cycles out of sync with a delay and comparison on registers. If they disagree after that delay, the processor concludes it&amp;#x27;s insane and you can take action based on that hardware-level signal.&lt;p&gt;It&amp;#x27;s not my favorite solution for every application, but it&amp;#x27;s a good tool for a handful of situations. And it&amp;#x27;s a good way to avoid having to think about picking who to trust with only two sockets. There are other approaches, of course. For example, if your system can afford more than a couple clock cycles of latency (e.g., if you don&amp;#x27;t believe you can crash a plane with a couple milliseconds of control error) you can do the same thing with well-designed software and a bit of extra hardware. It gives you more freedom since there&amp;#x27;s more hardware to buy, at the expense of engineering and test effort to prove that your solution does what you want.&lt;p&gt;Doing this over large physical distances also adds some challenges. There are debates around dedicated signal wires direct between control centers vs data networks. Every situation is a little different, so there aren&amp;#x27;t really easy answers that you can swoop in and prescribe as best-practices. In general I find that people are more-skeptical of packet-switched solutions (e.g., Ethernet) than I would be. But most folks are also better at reasoning about the failure modes there than they are at thinking about the lower-level issues that Ethernet solves for you. And of course there are tons of different signaling systems - not just Ethernet - to consider.&lt;p&gt;Consider just a few different possible applications and the constraints that they impose. A car has less severe failure modes (many crashes are nonfatal) but less freedom in time. Your trajectory is simply more cluttered. A passenger plane has dramatically more energy, so failures are typically more severe, but you have a lot more time to deal with them because your trajectory is empty (much of the time, anyway). And if you&amp;#x27;re sane, you&amp;#x27;ve designed it to be passively aerodynamically stable because twitchy maneuverability isn&amp;#x27;t critical when you&amp;#x27;re not a fighter jet. A rocket is a different animal altogether. Those are often rigged with self-destruct explosives, and you don&amp;#x27;t really get much time to change your mind if you erroneously choose to trigger them. Plus they&amp;#x27;re typically less stable (performance is more dear, gravity isn&amp;#x27;t helping you, and purely aerodynamic solutions don&amp;#x27;t take you outside the atmosphere) and are thus more dependent on their control systems.</text></comment>