<story><title>One Fastly customer triggered internet meltdown</title><url>https://www.bbc.co.uk/news/technology-57413224</url></story><parent_chain><item><author>oneeyedpigeon</author><text>&amp;gt; Early June 8, a customer pushed a valid configuration change that included the specific circumstances that triggered the bug, which caused 85% of our network to return errors.&lt;p&gt;Is it necessary to refer to &amp;quot;a customer&amp;quot; at all in this statement? What would be problematic if the above were rewritten as something like:&lt;p&gt;&amp;gt; Early June 8, a configuration change triggered a bug in our software, which caused 85% of our network to return errors.&lt;p&gt;The advantage is that you wouldn&amp;#x27;t get ignorant reporting that &amp;quot;one customer took down the internet&amp;quot;. I&amp;#x27;m not sure there are disadvantages that net outweigh that.</text></item><item><author>Jenk</author><text>How on earth do you figure that is anyone&amp;#x27;s fault but the BBC?&lt;p&gt;Read Fastly&amp;#x27;s statement. There is nothing about it blaming the customer(s) at all. There is nothing trying to save face.&lt;p&gt;What is your point here?</text></item><item><author>oneeyedpigeon</author><text>I know — as I&amp;#x27;ve said, the main blame lies with the bbc. However, as it&amp;#x27;s reported, it comes across very much as Fastly trying to save face. Maybe the blame is entirely on the bbc, maybe Fastly were naive in thinking that giving them this information wouldn&amp;#x27;t result in irresponsible headlines.</text></item><item><author>mytailorisrich</author><text>&amp;quot;&lt;i&gt;But a customer quite legitimately changing their settings had exposed a bug in a software update issued to customers in mid-May, causing &amp;quot;85% of our network to return errors&lt;/i&gt;&amp;quot;&lt;p&gt;They are careful to make clear that the customer did nothing wrong and that the problem was a bug in their software.</text></item><item><author>oneeyedpigeon</author><text>I&amp;#x27;m sorry, but I disagree. They gave the BBC enough detail that a very misleading headline was produced as a result. True, the main blame lies with the BBC, but it also comes across — to me, anyway, maybe I&amp;#x27;m being too cynical — as a bit of an excuse from Fastly.</text></item><item><author>iainmerrick</author><text>Throwing in my positive hot take among all the negative ones here: the immediate response and blog post from Fastly here is really good.&lt;p&gt;A quick fix, a clear apology, enough detail to give an idea of what happened, but not so much detail that there might be a mistake they’ll have to clarify or retract. What more are you looking for?&lt;p&gt;Apart from “not have the bug in the first place” -- and I hope and expect they’ll go into more detail later when they’ve had time for a proper post mortem -- I’d be interested to hear what anyone thinks they could have done better in terms of their immediate firefighting.</text></item></parent_chain><comment><author>Jenk</author><text>Yes, because it is explaining that it was a valid *customer* configuration, which is a separate set of concerns from, say, infrastructure config.&lt;p&gt;The important adjective &amp;quot;valid&amp;quot; means it was completely normal&amp;#x2F;expected input and thus not the fault of the customer.&lt;p&gt;It&amp;#x27;s perfectly clear you&amp;#x27;ve come at this with a pre-determined agenda of &amp;quot;I bet fastly, like most other public statements after corporate booboos I&amp;#x27;ve seen, will try to shrug this one off as someone else&amp;#x27;s fault&amp;quot; after reading the BBCs title and haven&amp;#x27;t bothered to read it at all until now.</text></comment>
<story><title>One Fastly customer triggered internet meltdown</title><url>https://www.bbc.co.uk/news/technology-57413224</url></story><parent_chain><item><author>oneeyedpigeon</author><text>&amp;gt; Early June 8, a customer pushed a valid configuration change that included the specific circumstances that triggered the bug, which caused 85% of our network to return errors.&lt;p&gt;Is it necessary to refer to &amp;quot;a customer&amp;quot; at all in this statement? What would be problematic if the above were rewritten as something like:&lt;p&gt;&amp;gt; Early June 8, a configuration change triggered a bug in our software, which caused 85% of our network to return errors.&lt;p&gt;The advantage is that you wouldn&amp;#x27;t get ignorant reporting that &amp;quot;one customer took down the internet&amp;quot;. I&amp;#x27;m not sure there are disadvantages that net outweigh that.</text></item><item><author>Jenk</author><text>How on earth do you figure that is anyone&amp;#x27;s fault but the BBC?&lt;p&gt;Read Fastly&amp;#x27;s statement. There is nothing about it blaming the customer(s) at all. There is nothing trying to save face.&lt;p&gt;What is your point here?</text></item><item><author>oneeyedpigeon</author><text>I know — as I&amp;#x27;ve said, the main blame lies with the bbc. However, as it&amp;#x27;s reported, it comes across very much as Fastly trying to save face. Maybe the blame is entirely on the bbc, maybe Fastly were naive in thinking that giving them this information wouldn&amp;#x27;t result in irresponsible headlines.</text></item><item><author>mytailorisrich</author><text>&amp;quot;&lt;i&gt;But a customer quite legitimately changing their settings had exposed a bug in a software update issued to customers in mid-May, causing &amp;quot;85% of our network to return errors&lt;/i&gt;&amp;quot;&lt;p&gt;They are careful to make clear that the customer did nothing wrong and that the problem was a bug in their software.</text></item><item><author>oneeyedpigeon</author><text>I&amp;#x27;m sorry, but I disagree. They gave the BBC enough detail that a very misleading headline was produced as a result. True, the main blame lies with the BBC, but it also comes across — to me, anyway, maybe I&amp;#x27;m being too cynical — as a bit of an excuse from Fastly.</text></item><item><author>iainmerrick</author><text>Throwing in my positive hot take among all the negative ones here: the immediate response and blog post from Fastly here is really good.&lt;p&gt;A quick fix, a clear apology, enough detail to give an idea of what happened, but not so much detail that there might be a mistake they’ll have to clarify or retract. What more are you looking for?&lt;p&gt;Apart from “not have the bug in the first place” -- and I hope and expect they’ll go into more detail later when they’ve had time for a proper post mortem -- I’d be interested to hear what anyone thinks they could have done better in terms of their immediate firefighting.</text></item></parent_chain><comment><author>davisoneee</author><text>In what way is the BBC at fault for this? Their title is objectively true. A _valid_ configuration setting that was used by a customer _did_ cause fastly to have an outage.&lt;p&gt;It&amp;#x27;s not limited to one specific customer (i.e this customer isn&amp;#x27;t the only customer who could have caused the issue, presumably), but it _was_ something the customer (legitimately) did. It wasn&amp;#x27;t a server outage. It wasn&amp;#x27;t a fire. It wasn&amp;#x27;t a cut cable.&lt;p&gt;&amp;quot;a customer quite legitimately changing their settings (BBC: one fastly customer) had exposed a bug (BBC: triggered internet meltdown) in a software update issued to customers (fastly admitting, when combined with &amp;#x27;legitimately&amp;#x27;, that fastly are at fault) in mid-May&amp;quot;.</text></comment>