<story><title>Coinbase Incident Post Mortem: June 25–26, 2019</title><url>https://blog.coinbase.com/incident-post-mortem-june-25-26-2019-1d08c1657cbc</url></story><parent_chain></parent_chain><comment><author>cle</author><text>Lots of architectural and cultural problems IMO.  Mixing different kinds of queries on the same cluster, no auto scaling on neither cluster nor web server, they seem to be okay with customer-impacting maintenance events (seriously?!), and their &amp;quot;fix&amp;quot; for an event caused by cache misses is to add more caching, which will make their system even harder to understand and predict, increasing the likelihood of more severe and byzantine failures.&lt;p&gt;It&amp;#x27;s often an unpopular opinion around here, but this is why I prefer simple hosted databases with limited query flexibility for high volume and high availability services (Firestore, DynamoDB, etc.).  It&amp;#x27;s harder to be surprised by expensive queries, and you won&amp;#x27;t have to fiddle with failovers, auto scaling, caching, etc.  Design your system around their constraints and it will have predictable performance and can more easily scale under unexpected load.</text></comment>
<story><title>Coinbase Incident Post Mortem: June 25–26, 2019</title><url>https://blog.coinbase.com/incident-post-mortem-june-25-26-2019-1d08c1657cbc</url></story><parent_chain></parent_chain><comment><author>viraptor</author><text>&amp;gt; We have ensured that failovers for this cluster may only be initiated during rare, scheduled downtime, when there will be no impact on customers.&lt;p&gt;I hope all their hardware crashes are also scheduled when there will be no impact... This seems a bit backwards - unless you constantly exercise the instantaneous failover, how do you know it works?&lt;p&gt;Edit: Actually it&amp;#x27;s worse - if you don&amp;#x27;t test the instant failover under a full load, how do you know it&amp;#x27;s still instant then?</text></comment>