<story><title>An administrator accidentally deleted the production database</title><url>http://support.gliffy.com/entries/98911057--Gliffy-Online-System-Outage</url></story><parent_chain><item><author>arethuza</author><text>My very first job - ~25 years ago.&lt;p&gt;Destroyed the production payroll database for a customer with a bug in a shell script.&lt;p&gt;No problem - they had 3 backup tapes.&lt;p&gt;First tape - read fails.&lt;p&gt;Second tape - read fails.&lt;p&gt;Third tape - worked.... (very nervous at this point).&lt;p&gt;I think most people have an equivalent educational experience at some point in their careers.&lt;p&gt;Edit: Had a project cancelled for one customer because they lost the database of test results..... 4 months work! Their COO (quite a large company) actually apologised to me in person!&lt;p&gt;Edit: Also had someone from Oracle break a financial consolidation system for a billion dollar company - his last words were &amp;quot;you need to restore from tape&amp;quot; and then he disappeared. I was &lt;i&gt;not&lt;/i&gt; happy as it was his attempts at &amp;quot;improving&amp;quot; things were the cause of the incident! Wouldn&amp;#x27;t have been angry if he had admitted he had made a mistake and worked with us to fix it - simply saying &amp;quot;restore from tape&amp;quot; and running away was not a good approach.</text></item></parent_chain><comment><author>roflc0ptic</author><text>A coworker of mine used to say &amp;quot;It&amp;#x27;s not the backup, it&amp;#x27;s the restore.&amp;quot; Meaning your backup process isn&amp;#x27;t meaningful unless you have a tested and effective means for recreating your system from that backup. It has stuck with me.</text></comment>
<story><title>An administrator accidentally deleted the production database</title><url>http://support.gliffy.com/entries/98911057--Gliffy-Online-System-Outage</url></story><parent_chain><item><author>arethuza</author><text>My very first job - ~25 years ago.&lt;p&gt;Destroyed the production payroll database for a customer with a bug in a shell script.&lt;p&gt;No problem - they had 3 backup tapes.&lt;p&gt;First tape - read fails.&lt;p&gt;Second tape - read fails.&lt;p&gt;Third tape - worked.... (very nervous at this point).&lt;p&gt;I think most people have an equivalent educational experience at some point in their careers.&lt;p&gt;Edit: Had a project cancelled for one customer because they lost the database of test results..... 4 months work! Their COO (quite a large company) actually apologised to me in person!&lt;p&gt;Edit: Also had someone from Oracle break a financial consolidation system for a billion dollar company - his last words were &amp;quot;you need to restore from tape&amp;quot; and then he disappeared. I was &lt;i&gt;not&lt;/i&gt; happy as it was his attempts at &amp;quot;improving&amp;quot; things were the cause of the incident! Wouldn&amp;#x27;t have been angry if he had admitted he had made a mistake and worked with us to fix it - simply saying &amp;quot;restore from tape&amp;quot; and running away was not a good approach.</text></item></parent_chain><comment><author>oolongCat</author><text>Now that&amp;#x27;s terrifying. Glad it worked out for you.&lt;p&gt;My tifu is when I first started working for an actual client, I thought I was a genius running my own git server on my own computer. Also I had the ssh key for my server on the same computer.&lt;p&gt;When my hdd failed, I lost everything.&lt;p&gt;Now the funny thing was, while my server was running I had no way to gain access to the server, the customer had already started using it, and they had inserted around 400-500 records already.&lt;p&gt;Now there I was, locked out of my own server, no source code I was working on for around 3-4 months.&lt;p&gt;Lucky for me, someone online had mentioned about a way to gain access to an aws server if you lost the ssh key.&lt;p&gt;I created a new instance, and mapped the old storage to the new instance. Lucky for me this worked.&lt;p&gt;Also, 2 weeks(ish) before this blunder I had sent a copy of my code to a friend who was helping me with some issues I had.&lt;p&gt;So my email saved me.&lt;p&gt;Now I backup regularly, use github(recent projects gitlab) and I have my ssh keys on 3 separate pen-drives and this &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;www.oakalleyit.com&amp;#x2F;node&amp;#x2F;4&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;www.oakalleyit.com&amp;#x2F;node&amp;#x2F;4&lt;/a&gt;
----------------</text></comment>