<story><title>AWS outage summary</title><url>https://aws.amazon.com/message/680342/</url><text></text></story><parent_chain></parent_chain><comment><author>seldo</author><text>I dunno about you, but I could use a TL;DR for this:&lt;p&gt;1. They fucked up an internal DNS change and didn&apos;t notice&lt;p&gt;2. Internal systems on EBS hosts piled up with messages trying to get to the non-existent domain&lt;p&gt;3. Eventually the messages used up all the memory on the EBS hosts, and thousands of EBS hosts began to die simultaneously&lt;p&gt;4. Meanwhile, panicked operators trying to slow down this tidal wave hit the Throttle Everything button&lt;p&gt;5. The throttling was so aggressive the even normal levels of operation became impossible&lt;p&gt;6. The incident was a single AZ, but the throttling was across the whole region, which spread the pain further&lt;p&gt;[Everybody who got throttled gets a 3-hour refund]&lt;p&gt;7. Any single-AZ RDS instance on a dead EBS host was fucked&lt;p&gt;8. &lt;i&gt;Multi&lt;/i&gt;-AZ RDS instances ran into two separate bugs, and either became stuck or hit a replication race condition and shut down&lt;p&gt;[Everybody whose multi-AZ RDS didn&apos;t fail over gets 10 days free credit]&lt;p&gt;9. Single-AZ ELB instances in the broken AZ failed because they use EBS too&lt;p&gt;10. Because everybody was freaking out and trying to fix their ELBs, the ELB service ran out of IP addresses and locked up&lt;p&gt;11. Multi-AZ ELB instances took too long to notice EBS was broken and then hit a bug and didn&apos;t fail over properly anyway&lt;p&gt;[ELB users get no refund, which seems harsh]&lt;p&gt;For those keeping score, that&apos;s 1 human error, 2 dependency chains, 3 design flaws, 3 instances of inadequate monitoring, and 5 brand-new internal bugs. From the length and groveling tone of the report, I can only assume that a big chunk of customers are very, VERY angry at them.</text></comment>
<story><title>AWS outage summary</title><url>https://aws.amazon.com/message/680342/</url><text></text></story><parent_chain></parent_chain><comment><author>Trufa</author><text>I really love when the companies take time to explain their customers what happened specially in such detail.&lt;p&gt;It&apos;s clearly a very complicated setting, and this type of posts make me trust them more, don&apos;t get me wrong, and outage is an outage, but knowing that they are in control and take time to explain shows respect and the correct attitude towards a mistake.&lt;p&gt;Good for them!</text></comment>