<story><title>Worst Case</title><url>https://www.tbray.org/ongoing/When/202x/2021/10/08/The-WOrst-Case</url></story><parent_chain><item><author>michaelt</author><text>I assume the author isn&amp;#x27;t asking &amp;quot;What if there was a 30 minute outage of us-east-1&amp;quot; (just wait for it to come back) or &amp;quot;What if there was an outage of a single AZ in us-east-1&amp;quot; (just spin things up in a different AZ)&lt;p&gt;Rather, they&amp;#x27;re asking &amp;quot;What if there was a 30 day outage of us-east-1&amp;quot; - so anyone who isn&amp;#x27;t multi-region or multi-cloud loses everything, including backups, AMIs, and control plane access.&lt;p&gt;(FWIW I agree with people disagreeing with the worry levels in the article - a solar storm last seen in 1859 is more likely than a software bug? Ha!)</text></item><item><author>dijit</author><text>&amp;gt; Amazon itself would leverage every tool at it&amp;#x27;s disposal to protect its reputation for reliability.&lt;p&gt;This is a joke, right? The _real_ degradation map of us-east-1 of the last 5 years looks significantly worse than my non-UPS backed Home PC in Sweden.&lt;p&gt;Personally I&amp;#x27;m not looking at us-east-1 as reliable at all; they even suffered a &amp;quot;harddrive crash&amp;quot; &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.bleepingcomputer.com&amp;#x2F;news&amp;#x2F;technology&amp;#x2F;amazon-aws-outage-shows-data-in-the-cloud-is-not-always-safe&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.bleepingcomputer.com&amp;#x2F;news&amp;#x2F;technology&amp;#x2F;amazon-aws-...&lt;/a&gt;</text></item><item><author>implying</author><text>This is a bizarre analysis. Public legal risk is absolutely the last imaginable threat to us-east-1, short of aliens abducting it. The U.S. security apparatus depends on AWS and would never allow it, Wall Street would never allow it, never mind the fact that Amazon itself would leverage every tool at it&amp;#x27;s disposal to protect its reputation for reliability. The politicians involved in this scenario might seek to remove Amazon&amp;#x27;s competitive advantages, or fine them, but the people who understand what AWS even is would never consider a move to shut down a datacenter.&lt;p&gt;Both the &amp;quot;enemy action&amp;quot; and &amp;quot;operation failure&amp;quot; scenarios are much bigger risks than this article makes out to be. Every non-aligned nation-state offensive cyber team has a knockout of us-east-1 at the top of their desired capabilities. I&amp;#x27;m sure efforts range from recruiting Amazon employees to preparing physical sabotage to hoarding 0days in the infrastructure. There&amp;#x27;s no reason to think one of them wouldnt rock the boat if geopolitics dictated.&lt;p&gt;Operational failure is probably the most likely. AWS might have a decade of experience building resilience, but some events happen on longer timescales. A bug that silently corrupts data before checksums and duplication and doesn&amp;#x27;t get noticed until almost every customer is borked, a vendor gives bad ECC ram that fails after 6 months in the field and is already deployed to 10,000 servers, etc. Networking is hard and an extended outage on the order of a week isn&amp;#x27;t completely impossible. How many customer systems can survive a week of downtime? How many customer businesses can?</text></item></parent_chain><comment><author>jdub</author><text>Please don&amp;#x27;t mischaracterise the point: The solar storm is not more likely than a software bug. The &lt;i&gt;effect&lt;/i&gt; of the solar storm is more likely than a similar effect by a software bug (particularly in the context of AWS).</text></comment>
<story><title>Worst Case</title><url>https://www.tbray.org/ongoing/When/202x/2021/10/08/The-WOrst-Case</url></story><parent_chain><item><author>michaelt</author><text>I assume the author isn&amp;#x27;t asking &amp;quot;What if there was a 30 minute outage of us-east-1&amp;quot; (just wait for it to come back) or &amp;quot;What if there was an outage of a single AZ in us-east-1&amp;quot; (just spin things up in a different AZ)&lt;p&gt;Rather, they&amp;#x27;re asking &amp;quot;What if there was a 30 day outage of us-east-1&amp;quot; - so anyone who isn&amp;#x27;t multi-region or multi-cloud loses everything, including backups, AMIs, and control plane access.&lt;p&gt;(FWIW I agree with people disagreeing with the worry levels in the article - a solar storm last seen in 1859 is more likely than a software bug? Ha!)</text></item><item><author>dijit</author><text>&amp;gt; Amazon itself would leverage every tool at it&amp;#x27;s disposal to protect its reputation for reliability.&lt;p&gt;This is a joke, right? The _real_ degradation map of us-east-1 of the last 5 years looks significantly worse than my non-UPS backed Home PC in Sweden.&lt;p&gt;Personally I&amp;#x27;m not looking at us-east-1 as reliable at all; they even suffered a &amp;quot;harddrive crash&amp;quot; &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.bleepingcomputer.com&amp;#x2F;news&amp;#x2F;technology&amp;#x2F;amazon-aws-outage-shows-data-in-the-cloud-is-not-always-safe&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.bleepingcomputer.com&amp;#x2F;news&amp;#x2F;technology&amp;#x2F;amazon-aws-...&lt;/a&gt;</text></item><item><author>implying</author><text>This is a bizarre analysis. Public legal risk is absolutely the last imaginable threat to us-east-1, short of aliens abducting it. The U.S. security apparatus depends on AWS and would never allow it, Wall Street would never allow it, never mind the fact that Amazon itself would leverage every tool at it&amp;#x27;s disposal to protect its reputation for reliability. The politicians involved in this scenario might seek to remove Amazon&amp;#x27;s competitive advantages, or fine them, but the people who understand what AWS even is would never consider a move to shut down a datacenter.&lt;p&gt;Both the &amp;quot;enemy action&amp;quot; and &amp;quot;operation failure&amp;quot; scenarios are much bigger risks than this article makes out to be. Every non-aligned nation-state offensive cyber team has a knockout of us-east-1 at the top of their desired capabilities. I&amp;#x27;m sure efforts range from recruiting Amazon employees to preparing physical sabotage to hoarding 0days in the infrastructure. There&amp;#x27;s no reason to think one of them wouldnt rock the boat if geopolitics dictated.&lt;p&gt;Operational failure is probably the most likely. AWS might have a decade of experience building resilience, but some events happen on longer timescales. A bug that silently corrupts data before checksums and duplication and doesn&amp;#x27;t get noticed until almost every customer is borked, a vendor gives bad ECC ram that fails after 6 months in the field and is already deployed to 10,000 servers, etc. Networking is hard and an extended outage on the order of a week isn&amp;#x27;t completely impossible. How many customer systems can survive a week of downtime? How many customer businesses can?</text></item></parent_chain><comment><author>dijit</author><text>realistically if AWS is down for 1+ day in that region you will be wondering very much if they&amp;#x27;re coming back in just 1hr more, 1 more day, 1 month or if at all.&lt;p&gt;Yeah, it&amp;#x27;s kind of &amp;quot;incredible&amp;quot; that they&amp;#x27;d be gone for so long, but I&amp;#x27;ve been on AWS during long outages before and it&amp;#x27;s never certain how long it will go on for and if data will be kept.&lt;p&gt;We assume too much I think of our providers, and I believe that&amp;#x27;s what the post is about. If you already _know_ AWS will be down for 30 days then you can make an informed decision based on that.&lt;p&gt;For me, there&amp;#x27;s no difference between 8 hours or 80, if you&amp;#x27;re down more than 8hrs I&amp;#x27;m going to activate my redundancies. But: I have redundancies. Many people don&amp;#x27;t.</text></comment>