<story><title>What we learned after I deleted the main production database by mistake</title><url>https://medium.com/@hugo.oliveira.rocha/what-we-learned-after-i-deleted-the-main-production-database-by-mistake-13c0a5815de5</url></story><parent_chain></parent_chain><comment><author>gerdesj</author><text>&amp;quot;Processes are to blame, not people&amp;quot;&lt;p&gt;It&amp;#x27;s called Disaster Recovery and it has a best mate called Business Continuity.  If you don&amp;#x27;t actually have a process for something then it will fail unconditionally, without you even knowing it is going to happen, until it does.&lt;p&gt;OK, it&amp;#x27;s easy for me to snipe but nowhere did I see terms like those.  Backups are mentioned almost casually and there is this: &amp;quot;but no process was implemented for ElasticSearch databases&amp;quot;.&lt;p&gt;So, not only were critical parts of the system not actually backed up but there seems to have been no attempt to even discuss how to put Humpty Dumpty (1) back together again if the silly sod falls off the wall.&lt;p&gt;Then we get the meat of the &amp;quot;Processes are to blame, not people&amp;quot; section.  It discusses avoiding fucking up by making some small changes to working practices and so on but completely misses the real point.  The blogger lacks a process for recovery.  It&amp;#x27;s all very well worrying about avoiding a fuck up involving a DB deletion but how do you recover?  That should be only one entry in your DR plan.  BC also needs some work ...&lt;p&gt;I own a small company and I spend quite a lot of time worrying about an awful lot of things.   This &amp;quot;contrite&amp;quot; article that implies that the writer has actually learned a lesson worth divulging is extremely concerning to me.  I suppose it&amp;#x27;s a good start but I feel it falls rather short of identifying the real problem, learning from it and implementing a proper ... process.&lt;p&gt;(1) Humpty Dumpty: UK nursery rhyme nominally about an egg. Bit more to it than than but good enough for this discussion.</text></comment>
<story><title>What we learned after I deleted the main production database by mistake</title><url>https://medium.com/@hugo.oliveira.rocha/what-we-learned-after-i-deleted-the-main-production-database-by-mistake-13c0a5815de5</url></story><parent_chain></parent_chain><comment><author>PeterisP</author><text>For me, an interesting statement was &amp;quot;However, it took 6 days to fetch all data for all 17 million products.&amp;quot; - in my experience of DB systems, 17 million entries is significant but not particularly much, it&amp;#x27;s something that fits in the RAM of a laptop and can be imported&amp;#x2F;exported&amp;#x2F;indexed&amp;#x2F;processed in minutes (if you do batch processing, not a separate transaction per entry), perhaps hours if the architecture is lousy but certainly not days.</text></comment>