<story><title>Retrospective and technical details on the recent Firefox outage</title><url>https://hacks.mozilla.org/2022/02/retrospective-and-technical-details-on-the-recent-firefox-outage/</url></story><parent_chain><item><author>deng</author><text>I&amp;#x27;d claim that no other company is criticized as harshly as Mozilla around here. The amount of blame that is assigned to the Firefox team is staggering.&lt;p&gt;To me, this is a perfectly valid write-up with a good lessons learned. They have written it in a very diplomatic way, but to me, it is absolutely clear that Google screwed up here. How can you make such a change to a default behavior of critical infrastructure unannounced? That&amp;#x27;s just reckless towards your customers, and solidifies my belief to stay away from GCP.&lt;p&gt;If they had properly announced the change, even if the Firefox team hadn&amp;#x27;t then tested beforehand, at least the DevOps team would have put one and one together and just changed back to HTTP&amp;#x2F;2 and the outage would have lasted maybe 10 minutes. Instead, they frantically went through their git log to see what in the code base might have triggered this bug. Everyone who has been in such a position knows how incredibly stressful this is. I&amp;#x27;d be absolutely livid at Google in their position. That it took two hours to fix this is clearly their fault.</text></item></parent_chain><comment><author>yjftsjthsd-h</author><text>&amp;gt; I&amp;#x27;d claim that no other company is criticized as harshly as Mozilla around here. The amount of blame that is assigned to the Firefox team is staggering.&lt;p&gt;They set themselves a higher standard by marketing as the good guys who fight for the user, and then made any number of moves that said users viewed as &lt;i&gt;not&lt;/i&gt; being in their interests. Of course they get more blame. Like, Chrome has issues, but they&amp;#x27;re issues in line with being made by an adtech company; we might be unhappy at Google breaking adblockers (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.eff.org&amp;#x2F;deeplinks&amp;#x2F;2021&amp;#x2F;12&amp;#x2F;chrome-users-beware-manifest-v3-deceitful-and-threatening&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.eff.org&amp;#x2F;deeplinks&amp;#x2F;2021&amp;#x2F;12&amp;#x2F;chrome-users-beware-ma...&lt;/a&gt;), but it&amp;#x27;s not out of character. Mozilla can say &amp;quot;More power to you. Mozilla puts people before profit, creating products, technologies and programs that make the internet healthier for everyone.&amp;quot; (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.mozilla.org&amp;#x2F;en-US&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.mozilla.org&amp;#x2F;en-US&amp;#x2F;&lt;/a&gt;) or they can, say, make Google the default engine ($), bake in a proprietary service (Pocket), rip out features (RIP compact theme), overrule user autonomy (Want to install an extension? Better upload it to Mozilla to get signed so they permit you to run it on your own computer!), ship a marketing extension through the &amp;quot;experiments&amp;quot; feature (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;blog.mozilla.org&amp;#x2F;en&amp;#x2F;products&amp;#x2F;firefox&amp;#x2F;update-looking-glass-add&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;blog.mozilla.org&amp;#x2F;en&amp;#x2F;products&amp;#x2F;firefox&amp;#x2F;update-looking-...&lt;/a&gt;).... but not both. Either empower the user, or don&amp;#x27;t, but don&amp;#x27;t pretend to empower the user &lt;i&gt;while&lt;/i&gt; ripping away their control.</text></comment>
<story><title>Retrospective and technical details on the recent Firefox outage</title><url>https://hacks.mozilla.org/2022/02/retrospective-and-technical-details-on-the-recent-firefox-outage/</url></story><parent_chain><item><author>deng</author><text>I&amp;#x27;d claim that no other company is criticized as harshly as Mozilla around here. The amount of blame that is assigned to the Firefox team is staggering.&lt;p&gt;To me, this is a perfectly valid write-up with a good lessons learned. They have written it in a very diplomatic way, but to me, it is absolutely clear that Google screwed up here. How can you make such a change to a default behavior of critical infrastructure unannounced? That&amp;#x27;s just reckless towards your customers, and solidifies my belief to stay away from GCP.&lt;p&gt;If they had properly announced the change, even if the Firefox team hadn&amp;#x27;t then tested beforehand, at least the DevOps team would have put one and one together and just changed back to HTTP&amp;#x2F;2 and the outage would have lasted maybe 10 minutes. Instead, they frantically went through their git log to see what in the code base might have triggered this bug. Everyone who has been in such a position knows how incredibly stressful this is. I&amp;#x27;d be absolutely livid at Google in their position. That it took two hours to fix this is clearly their fault.</text></item></parent_chain><comment><author>moeris</author><text>I agree that Google is at fault here for failing Firefox.  But Firefox is guilty of failing its users.  Why should the functioning of a browser be dependent on telemetry working?  It sounds like if there is high enough latency in their telemetry, or if request for telemetry start failing, it&amp;#x27;s possible for that to disrupt using the network stack at all.  They have a massive design flaw, and they didn&amp;#x27;t even mention that in the article.  Maybe they have good reasons for designing a single point of failure that relies on a cloud provider, but it&amp;#x27;s not clear what those might be since they don&amp;#x27;t address it.</text></comment>