<story><title>Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region</title><url>https://aws.amazon.com/message/12721/</url></story><parent_chain><item><author>amzn-throw</author><text>&amp;gt; a VP must sign off on changing status pages, which is... backwards to say the least.&lt;p&gt;I think most people&amp;#x27;s experience with &amp;quot;VP&amp;#x27;s&amp;quot; makes them not realize what AWS VP&amp;#x27;s do.&lt;p&gt;VP&amp;#x27;s here are not sitting in an executive lounge wining and dining customers, chomping on cigars and telling minions to &amp;quot;Call me when the data center is back up and running again!&amp;quot;&lt;p&gt;They are on the tech call, working with the engineers, evaluating the problem, gathering the customer impact, and attempting to balance communicating too early with being precise.&lt;p&gt;Is there room for improvement? Yes. I wish we would just throw up a generic &amp;quot;Shit&amp;#x27;s Fucked Up. We Don&amp;#x27;t Know Why Yet, But We&amp;#x27;re Working On It&amp;quot; message.&lt;p&gt;But the reason why we don&amp;#x27;t, doesn&amp;#x27;t have anything to do with having to get VP approval to put that message up. The VP&amp;#x27;s are there in the trenches most of the time.</text></item><item><author>fastball</author><text>I think most of the outrage is not because &amp;quot;it happened&amp;quot; but because AWS is saying things like &amp;quot;S3 was unaffected&amp;quot; when the anecdotal experience of many in this thread suggests the opposite.&lt;p&gt;That and the apparent policy that a VP must sign off on changing status pages, which is... backwards to say the least.</text></item><item><author>jetru</author><text>Complex systems are really really hard. I&amp;#x27;m not a big fan of seeing all these folks bash AWS for this, and not really understanding the complexity or nastiness of situations like this. Running the kind of services they do for the kind of customers, this is a VERY hard problem.&lt;p&gt;We ran into a very similar issue, but at the database layer in our company literally 2 weeks ago, where connections to our MySQL exploded and completely took down our data tier and caused a multi-hour outage, compounded by retries and thundering herds. Understanding this problem under the stressful scenario is extremely difficult and a harrowing experience. Anticipating this kind of issue is very very tricky.&lt;p&gt;Naive responses to this include &amp;quot;better testing&amp;quot;, &amp;quot;we should be able to do this&amp;quot;, &amp;quot;why is there no observability&amp;quot; etc. The problem isn&amp;#x27;t testing. Complex systems behave in complex ways, and its difficult to model and predict, especially when the inputs to the system aren&amp;#x27;t entirely under your control. Individual components are easy to understand, but when integrating, things get out of whack. I can&amp;#x27;t stress how difficult it is to model or even think about these systems, they&amp;#x27;re very very hard. Combined with this knowledge being distributed among many people, you&amp;#x27;re dealing with not only distributed systems, but also distributed people, which adds more difficulty in wrapping this around your head.&lt;p&gt;Outrage is the easy response. Empathy and learning is the valuable one. Hugs to the AWS team, and good learnings for everyone.</text></item></parent_chain><comment><author>jrochkind1</author><text>&amp;gt; I wish we would just throw up a generic &amp;quot;Shit&amp;#x27;s Fucked Up. We Don&amp;#x27;t Know Why Yet, But We&amp;#x27;re Working On It&amp;quot; message.&lt;p&gt;I gotta say, the implication that you can&amp;#x27;t register an outage until you know why it happened is pretty damning. The status page is where we look to see if services are effected, if that information can&amp;#x27;t be shared there until you understand the cause, that&amp;#x27;s very broken.&lt;p&gt;The AWS status page has become kind of a joke to customers.&lt;p&gt;I was encouraged to see the announcement in OP say that there is &amp;quot;a new version of our Service Health Dashboard&amp;quot; coming. I hope it can provide actual capabilities to display, well, service health.&lt;p&gt;From how people talk about it, it kind of sounds like updates to the Service Health Dashboard are currently purely a manual process. Rather than automated monitoring automatically updating the Service Health Dashboard in any way at all. I find that a surprising implementation for an organization of Amazon&amp;#x27;s competence and power. That alarms me more than &lt;i&gt;who&lt;/i&gt; it is that has the power to manually update it; I agree that I don&amp;#x27;t have enough knowledge of AWS internal org structures to have an opinion on if it&amp;#x27;s the &amp;quot;right&amp;quot; people or not.&lt;p&gt;I suspect AWS must have internal service health pages that are actually automatically updated in some way by monitoring, that is, that actually work to display service health.  It &lt;i&gt;seems&lt;/i&gt; like a business decision rather than a technical challenge if the public facing system has no inputs but manual human entry, but that&amp;#x27;s just how it seems from the outside, I may not have full information. We only have what Amazon shares with us of course.</text></comment>
<story><title>Summary of the AWS Service Event in the Northern Virginia (US-East-1) Region</title><url>https://aws.amazon.com/message/12721/</url></story><parent_chain><item><author>amzn-throw</author><text>&amp;gt; a VP must sign off on changing status pages, which is... backwards to say the least.&lt;p&gt;I think most people&amp;#x27;s experience with &amp;quot;VP&amp;#x27;s&amp;quot; makes them not realize what AWS VP&amp;#x27;s do.&lt;p&gt;VP&amp;#x27;s here are not sitting in an executive lounge wining and dining customers, chomping on cigars and telling minions to &amp;quot;Call me when the data center is back up and running again!&amp;quot;&lt;p&gt;They are on the tech call, working with the engineers, evaluating the problem, gathering the customer impact, and attempting to balance communicating too early with being precise.&lt;p&gt;Is there room for improvement? Yes. I wish we would just throw up a generic &amp;quot;Shit&amp;#x27;s Fucked Up. We Don&amp;#x27;t Know Why Yet, But We&amp;#x27;re Working On It&amp;quot; message.&lt;p&gt;But the reason why we don&amp;#x27;t, doesn&amp;#x27;t have anything to do with having to get VP approval to put that message up. The VP&amp;#x27;s are there in the trenches most of the time.</text></item><item><author>fastball</author><text>I think most of the outrage is not because &amp;quot;it happened&amp;quot; but because AWS is saying things like &amp;quot;S3 was unaffected&amp;quot; when the anecdotal experience of many in this thread suggests the opposite.&lt;p&gt;That and the apparent policy that a VP must sign off on changing status pages, which is... backwards to say the least.</text></item><item><author>jetru</author><text>Complex systems are really really hard. I&amp;#x27;m not a big fan of seeing all these folks bash AWS for this, and not really understanding the complexity or nastiness of situations like this. Running the kind of services they do for the kind of customers, this is a VERY hard problem.&lt;p&gt;We ran into a very similar issue, but at the database layer in our company literally 2 weeks ago, where connections to our MySQL exploded and completely took down our data tier and caused a multi-hour outage, compounded by retries and thundering herds. Understanding this problem under the stressful scenario is extremely difficult and a harrowing experience. Anticipating this kind of issue is very very tricky.&lt;p&gt;Naive responses to this include &amp;quot;better testing&amp;quot;, &amp;quot;we should be able to do this&amp;quot;, &amp;quot;why is there no observability&amp;quot; etc. The problem isn&amp;#x27;t testing. Complex systems behave in complex ways, and its difficult to model and predict, especially when the inputs to the system aren&amp;#x27;t entirely under your control. Individual components are easy to understand, but when integrating, things get out of whack. I can&amp;#x27;t stress how difficult it is to model or even think about these systems, they&amp;#x27;re very very hard. Combined with this knowledge being distributed among many people, you&amp;#x27;re dealing with not only distributed systems, but also distributed people, which adds more difficulty in wrapping this around your head.&lt;p&gt;Outrage is the easy response. Empathy and learning is the valuable one. Hugs to the AWS team, and good learnings for everyone.</text></item></parent_chain><comment><author>gingerlime</author><text>&amp;gt; I wish we would just throw up a generic &amp;quot;Shit&amp;#x27;s Fucked Up. We Don&amp;#x27;t Know Why Yet, But We&amp;#x27;re Working On It&amp;quot; message.&lt;p&gt;I think that&amp;#x27;s the crux of the matter? AWS seems to now have a reputation for ignoring issues that are easily observable by customers, and by the time any update shows up, it&amp;#x27;s way too late. Whether VPs make this decision or not is irrelevant. If this becomes a known pattern (and I think it has), then the system is broken.&lt;p&gt;disclaimer: I have very little skin in this game. We use S3 for some static assets, and with layers of caching on top, I think we are rarely affected by outages. I&amp;#x27;m still curious to observe major cloud outages and how they are handled, and the HN reaction from people on both side of the fence.</text></comment>